Sitelet https://github.com/kubernetes/kops/pull/18581
Skip to content

gce: register SSH keys under the "ubuntu" user for Ubuntu images - #18581

Merged
kubernetes-prow[bot] merged 2 commits into
kubernetes:masterfrom
hakman:gce-ubuntu-ssh-user
Jul 13, 2026
Merged

gce: register SSH keys under the "ubuntu" user for Ubuntu images#18581
kubernetes-prow[bot] merged 2 commits into
kubernetes:masterfrom
hakman:gce-ubuntu-ssh-user

Conversation

@hakman

@hakman hakman commented Jul 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

kOps registers SSH public keys in GCE instance metadata under the "admin" username, but on Ubuntu images the GCE guest agent fails to create that user because the images ship with an "admin" group, so the key is never installed and SSH to instances does not work.

Derive the username from the instance group image instead: Ubuntu images register the key under the built-in "ubuntu" user, while other images keep "admin" so SSH access to existing non-Ubuntu clusters is unchanged.

Refs #16642

@hakman
gce: register SSH keys under the "ubuntu" user for Ubuntu images
7c9f29e 
kOps registers SSH public keys in GCE instance metadata under the
"admin" username, but on Ubuntu images the GCE guest agent fails to
create that user because the images ship with an "admin" group, so the
key is never installed and SSH to instances does not work.

Derive the username from the instance group image instead: Ubuntu
images register the key under the built-in "ubuntu" user, while other
images keep "admin" so SSH access to existing non-Ubuntu clusters is
unchanged.
@kubernetes-prow
kubernetes-prow Bot requested a review from olemarkus July 12, 2026 05:55
@kubernetes-prow kubernetes-prow Bot added the area/provider/gcp Issues or PRs related to gcp provider label Jul 12, 2026
@kubernetes-prow
kubernetes-prow Bot requested a review from zetaab July 12, 2026 05:55
@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 12, 2026
@rifelpet

rifelpet commented Jul 12, 2026

Copy link
Copy Markdown
Member

I see the prow jobs are able to SSH:

https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/e2e-kops-grid-gce-calico-u2404arm64-k34/2074571622748000256

 I0707 19:42:31.375447   21346 toolbox_dump.go:208] will SSH using username "prow"
I0707 19:42:31.375459   21346 toolbox_dump.go:209] ssh auth methods [0x5e20f80]
I0707 19:42:31.375480   21346 dumper.go:58] detected a bastion instance, with the address: 34.63.82.64
I0707 19:42:31.375492   21346 dumper.go:166] starting to dump 1 control plane nodes fetched through the Kubernetes APIs
I0707 19:42:31.375496   21346 dumper.go:270] Dumping node control-plane-us-central1-a-0jgp
W0707 19:42:35.493987   21346 dumper.go:286] error dumping node control-plane-us-central1-a-0jgp: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=external-dns\"; fi": Process exited with status 1
W0707 19:42:35.494003   21346 dumper.go:286] error dumping node control-plane-us-central1-a-0jgp: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=dns-controller\"; fi": Process exited with status 1
W0707 19:42:35.494005   21346 dumper.go:286] error dumping node control-plane-us-central1-a-0jgp: error executing command "if command -v kubectl &> /dev/null; then kubectl logs -n kube-system --all-containers -l \"k8s-app=kops-channels\"; fi": Process exited with status 1
I0707 19:42:35.494063   21346 dumper.go:270] Dumping node nodes-us-central1-a-50mr
I0707 19:42:38.881031   21346 dumper.go:270] Dumping node nodes-us-central1-a-71p5
I0707 19:42:42.047720   21346 dumper.go:270] Dumping node nodes-us-central1-a-hqxw
I0707 19:42:45.188249   21346 dumper.go:270] Dumping node nodes-us-central1-a-pb2b

We set KUBE_SSH_USER=prow

https://github.com/kubernetes/test-infra/blob/9b391474f607e28a3dc789b32841a3cd776b4c04/config/jobs/kubernetes/kops/build_jobs.py#L152

https://github.com/kubernetes/test-infra/blob/9b391474f607e28a3dc789b32841a3cd776b4c04/config/jobs/kubernetes/kops/kops-periodics-gce.yaml#L40-L44

but thats only used by kops for SSHing in to dump logs, we dont create that user:

https://github.com/search?q=repo%3Akubernetes%2Fkops%20SSHUser&type=code

Any idea why this is working with prow ? Ideally we'd remove all of the SSH configuration in build_jobs.py

@hakman

hakman commented Jul 13, 2026

Copy link
Copy Markdown
Member Author

toolbox dump works, but it never uses the key kOps registers. In CI, kubetest2 runs gcloud compute config-ssh, which adds a separate key to project-level metadata under the CI username and passes that user via --ssh-user. The key kOps writes to instance metadata as admin: <key> is never installed on Ubuntu, because the guest agent fails on useradd admin (the images ship with an admin group, see #16175). So dump bypasses the bug; plain ssh with the kOps key has no working path on Ubuntu.

I verified this on real VMs: only Ubuntu has the conflicting group, while Debian, RHEL, Rocky and COS create the admin user fine. Hence the first commit only special-cases Ubuntu and keeps admin elsewhere. The second commit fixes the related dump gap: the GCE dumper never set SSHUser per instance (AWS does), so kubetest2's SSH fallback had an empty user; it now derives it from the boot disk image with the same helper.

@kubernetes-prow

kubernetes-prow Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jul 13, 2026
@hakman

hakman commented Jul 13, 2026

Copy link
Copy Markdown
Member Author

/override pull-kops-e2e-k8s-aws-calico

@kubernetes-prow

kubernetes-prow Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

@hakman: Overrode contexts on behalf of hakman: pull-kops-e2e-k8s-aws-calico

Details

In response to this:

/override pull-kops-e2e-k8s-aws-calico

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow
kubernetes-prow Bot merged commit 0c45a7f into kubernetes:master Jul 13, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rifelpet rifelpet rifelpet approved these changes
@olemarkus olemarkus Awaiting requested review from olemarkus
@zetaab zetaab Awaiting requested review from zetaab

Assignees

@rifelpet rifelpet

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/provider/gcp Issues or PRs related to gcp provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hakman @rifelpet