cilium: require k8s-connectivity in liveness probe - #18237

This is a regression that blocks refresh of control-plane address for the agents. This was introduced during the last major update and is present in kOps v1.34 and v1.35.

The Q would, do we want to remove this "feature" or we may want to ignore cilium agents during cluster validation?

https://docs.cilium.io/en/stable/api/

GET /healthz returns health and status information of the Cilium daemon and related components such as the local container runtime, connected datastore, Kubernetes integration and Hubble. It accepts the request header "brief" which returns a brief representation of the Cilium status, and "require-k8s-connectivity" which, if set to true, causes failure of the agent to connect to the Kubernetes control plane to also fail the agent's health status.

cilium/cilium#32724

The community discussed the issue in the APAC community meeting and concluded that killing the whole Cilium agent on k8s disconnection is too aggressive — if the kube-apiserver goes down, that would force every cilium-agent instance in the cluster to also go down. Similar to how the dataplane should continue without the local control plane, it makes sense for the local control plane to continue without the central control plane.

/cc @rifelpet @ameukam