Make cloud-controller-manager pods tolerate all taints - #18510

What

Set tolerations: [{operator: Exists}] on every cloud-controller-manager pod that runs with hostNetwork: true (AWS, Azure, DigitalOcean, GCP, Scaleway), so the CCM can always be scheduled onto a node regardless of which taints are currently present. The Scaleway CCM additionally gains a node-role.kubernetes.io/control-plane nodeAffinity, since broadening its tolerations means tolerations can no longer keep it off worker nodes.

The Azure cloud-node-manager DaemonSet keeps nodeSelector: kubernetes.io/os: linux (it is a per-node agent and must run on every node); only its tolerations are broadened.

This matches the pattern already used by node-level DaemonSets that must run everywhere — the Cilium agent, CNI plugins, and CSI node drivers all use - operator: Exists.

Why

e2e-kops-do-gossip is intermittently failing cluster validation, e.g.:
https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/e2e-kops-do-gossip/2068280267818143744

The deadlock

With --cloud-provider=external, a node has no InternalIP until the cloud-controller-manager initializes it. The Cilium agent needs that node IP — without it the agent dies with unable to determine direct routing device (k8sNodeIP=""), so it never becomes ready.

Meanwhile the cilium-operator (set-cilium-node-taints: "true") adds node.cilium.io/agent-not-ready:NoSchedule to any node whose Cilium agent isn't ready yet. The DO CCM DaemonSet did not tolerate that taint. So once the operator taints the control-plane node before the CCM pod is bound there:

1. CCM can no longer schedule onto the control-plane node → node never initialized → no InternalIP
2. Cilium agent can't start without the node IP → never becomes ready
3. operator never removes the agent-not-ready taint → back to (1)

A permanent deadlock. The CCM pod never appears at all and the cluster fails to validate.

Why it's a race, and why only some clouds lose it

The agent-not-ready taint only blocks new scheduling — it never evicts an already-bound pod. So the outcome hinges entirely on whether the CCM pod is bound to the control-plane node before the cilium-operator taints it.

This is decided during control-plane bootstrap by which static-pod component wins a restart-timing race:

• The kube-scheduler and kube-controller-manager both crash-loop on startup until the kube-apiserver has published the extension-apiserver-authentication configmap (identical behavior on every cloud).
• On a healthy/fast control plane, the scheduler is up and caches-synced before the daemonset-controller creates the CCM pod, so the CCM pod binds immediately — well before the operator's taint controller (which has a built-in ~11s leader-election/startup delay) ever runs.
• On a slower/less-stable control plane, the apiserver restarts an extra time, the scheduler crash-loops one or two extra cycles, and CrashLoopBackOff (10s → 20s → 40s → 80s) turns that into a large gap. The CCM/cilium pods sit Pending with no scheduler; when the scheduler finally comes up it binds the all-tolerating Cilium pod to the control-plane node first, the operator taints the node, and the CCM pod is locked out.

Observed on the failing DO run vs. a passing AWS run:

Because the margin is environmental (control-plane bring-up speed, not config), the same DO job passes on runs where the control plane comes up quickly — which is exactly why it's flaky rather than always-failing. The placement constraints, node-IP handling, Cilium routing mode (tunnel/ipam: kubernetes), and addon apply order are all identical between the passing and failing clouds, so none of those explain the difference — it is purely the bootstrap restart-timing race.

Every external CCM in kops shares this latent exposure; they just run on clouds whose control planes usually win the race. Making the CCM tolerate all taints removes the dependency on winning that window entirely.

Testing

• Regenerated integration golden files (hack/update-expected.sh); manifestHashes update so existing clusters re-apply the new manifests automatically.
• go test ./cmd/kops/ and the bootstrapchannelbuilder tests pass.

🤖 Generated with Claude Code