Fix node bootstrap in IPv6-only clusters with --dns=none - #18684
Merged
kubernetes-prow[bot] merged 2 commits intoAug 13, 2026
Merged
Conversation
With --dns=none, nodeupconfigbuilder puts the API load balancer's addresses into BootConfig.APIServerIPs, which become the kops-controller bootstrap endpoints and the /etc/hosts entries for api.internal and kops-controller.internal. On AWS an address qualifies if it is inside the network CIDR or if it is IPv6, so a dualstack NLB contributes both its private IPv4 address and its IPv6 address. Nodes in an IPv6-only cluster sit in subnets that have no IPv4 CIDR, so the IPv4 address is unroutable from them: Post "https://172.20.6.26:3988/bootstrap": dial tcp 172.20.6.26:3988: connect: network is unreachable FindAddresses sorts the addresses as strings, which puts the IPv4 one first, so that is the endpoint nodeup spends its retry budget on and no node ever joins. Drop IPv4 addresses for IPv6-only clusters. This combination has no coverage today: every IPv6 job passes --dns=public, and no dns-none job is IPv6. e2e-kops-aws-ipv6-karpenter reached it first because it does not pass --dns=public and so picked up the dns=none default. Add an ipv6 + dns=none integration test, and pull the address selection out into selectControlPlaneIPs so it can be unit tested -- the integration test mock does not populate load balancer addresses, so it cannot exercise this.
BootConfig.ConfigServer.Servers can hold more than one endpoint, and getNodeConfigFromServers walks the list -- but Client.Query retries a single server 100 times with a capped exponential backoff, roughly 50 minutes. Any server that is permanently unreachable, such as an IPv4 address in an IPv6-only cluster, therefore consumes the whole budget and the remaining entries are never tried. Give the client a configurable Backoff, and have getNodeConfigFromServers set a short one so each server gets about a minute before we move on. Keep asking until the overall bootstrap timeout, cycling the list rather than giving up after one pass, so a control plane that is slow to come up is still waited out. BootstrapClientTask keeps the previous behaviour: it has a single server and does not set Backoff.
Contributor
|
Skipping CI for Draft Pull Request. |
Member
Author
|
/test pull-kops-e2e-aws-ipv6-karpenter |
rifelpet
marked this pull request as ready for review
August 13, 2026 13:42
Member
Author
|
/cc @hakman |
Member
Author
|
confirmed the karpenter job passed and the EC2NodeClasses have the expected nodeup config: cat > conf/kube_env.yaml << '__EOF_KUBE_ENV'
APIServerIPs:
- 2600:1f16:284:9002:71e4:b233:3211:5cee
CloudProvider: aws
ClusterName: pr18684-kops-e2e-aws-ipv6-karpenter.tests-kops-aws.k8s.io
ConfigServer:
servers:
- https://[2600:1f16:284:9002:71e4:b233:3211:5cee]:3988/
tlsServerName: kops-controller.internal.pr18684-kops-e2e-aws-ipv6-karpenter.tests-kops-aws.k8s.io
InstanceGroupName: nodes
InstanceGroupRole: Node |
Member
Nice! |
hakman
approved these changes
Aug 13, 2026
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hakman The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Nodes never join an IPv6-only cluster that uses
--dns=none. Two independent bugs; one commit each.Why
e2e-kops-aws-ipv6-karpenterstarted failing as soon as it began actually creating an IPv6 cluster (#18681 plus kubernetes/test-infra#37675 — before that it was silently building an IPv4 cluster). Every node fails to bootstrap:The generated
EC2NodeClassuserData shows why:The IPv4 address should not be there. With
--dns=none,nodeupconfigbuilderfillsAPIServerIPsfrom the API NLB's ENI addresses, keeping anything inside the network CIDR or any IPv6 address — so a dualstack NLB contributes both. Nodes in an IPv6-only cluster are in subnets with no IPv4 CIDR and cannot reach172.20.6.26at all. These addresses also become the/etc/hostsentries forapi.internalandkops-controller.internal.nodeup never reaches the second server.
getNodeConfigFromServerswalks the server list, butClient.Queryretries one server 100 times with a capped backoff — roughly 50 minutes.FindAddressessorts the addresses as strings, so the IPv4 one sorts first and absorbs the entire budget. The multi-server list is effectively single-server whenever the first entry is down.Either fix alone unblocks the job; both are worth having.
Coverage gap
IPv6 +
dns=noneis untested. Every IPv6 job passes--dns=public, and no job inkops-periodics-dns-none.yaml/kops-presubmits-dns-none.yamlis IPv6.e2e-kops-aws-ipv6-karpenterhit it only because it omits--dns=publicand so picked up thedns=nonedefault fromsetupDNSTopology.This PR adds
TestMinimalIPv6DNSNone, the first ipv6 + dns=none integration test. It does not exercise the fix directly — the integration test mock never populates load balancer addresses, soAPIServerIPscomes out empty and the config falls back to the DNS name (minimal-dns-nonehas the same gap). The address selection is therefore pulled out intoselectControlPlaneIPsand unit tested.Testing
go test ./cmd/kops/... ./pkg/nodemodel/... ./upup/pkg/fi/nodeup/... ./util/pkg/vfs/...passes./test pull-kops-e2e-aws-ipv6-karpenter— the only presubmit that covers this combination. Note it isalways_run: falsewith itsrun_if_changedcommented out, so it has to be requested explicitly.