tests/e2e: resolve SSH keys the same way on every cloud - #18686

First step toward dropping the SSH key env vars and presets from the kops prow jobs entirely. This PR is a no-op for every job today — it unifies the code path and adds the missing fallbacks, but keeps honoring every variable the current presets set.

The problem

deployer/common.go resolved the keypair in four unrelated per-cloud branches:

So two clouds had no fallback, Azure silently discarded --ssh-private-key, and GCE behaved differently depending on whether --gcp-project was supplied.

The change

One resolveSSHKeys() for all clouds, in precedence order:

1. --ssh-private-key / --ssh-public-key
2. KUBE_SSH_KEY_PATH / KUBE_SSH_PUBLIC_KEY_PATH (cloud-agnostic)
3. the legacy per-cloud variables
4. an ephemeral keypair generated for this cluster

Generating is safe everywhere because kops registers whatever public key it is given with the cloud provider itself — an EC2 key pair (awstasks.SSHKey), GCE ssh-keys instance metadata (gcemodel/autoscalinggroup.go), the Azure VMSS osProfile (azuretasks/vmscaleset.go), or a droplet key (dotasks/droplet.go). up.go already passes --ssh-public-key on every cloud.

env() now exports both KUBE_SSH_KEY_PATH and KUBE_SSH_USER for all providers, instead of exporting only the key on AWS, so ginkgo and clusterloader2 use the credentials the deployer resolved.

Both are required together, which the first revision of this PR got wrong and pull-kops-azure-dns-none caught. The framework's SkipUnlessSSHKeyPresent() skips its SSH tests when GetSigner() finds no key, so exporting the key alone un-skips [sig-node] SSH should SSH to all nodes on azure — which then failed with an empty username (error getting SSH client to @20.39.226.86:22), because azure and digitalocean assign the SSH user internally (kops / root) rather than from the job config, and the framework otherwise falls back to $USER. Note this cannot be fixed with an e2e flag: test/e2e/framework/ssh/ssh.go reads KUBE_SSH_USER from the environment and test_context.go has no --ssh-user flag.

That azure run also confirms the fix works: toolbox-dump.yaml reported sshUser: kops on all five instances, and the deployer had already SSHed into that same cluster as --ssh-user kops --private-key /tmp/kops/pr18686-kops-azure-dns-none/id_ed25519 to collect a 1.6 MB journal.log. The credentials were proven good; the framework simply was not told the user.

Two dead things removed

gce.SetupSSH ran gcloud compute config-ssh only when both GCE_SSH_* were empty. preset-k8s-ssh always sets them, so it never ran — config-ssh appears zero times in GCE job build logs. Its remaining effect was registering a key in GCE project metadata; the instance-metadata key kops installs makes that unnecessary.

lib/common.sh forced AWS_SSH_PRIVATE_KEY_FILE=${HOME}/.ssh/id_rsa when unset, which defeated the generation fallback by pointing at a file that doesn't exist in the prow pod. It had to go before generation could ever engage.

Why this is a no-op today

Every job still sets the legacy variables via its preset, and they resolve to the same paths as before:

• preset-aws-ssh → /etc/aws-ssh/aws-ssh-{private,public}
• preset-k8s-ssh → /etc/ssh-key-secret/ssh-{private,public}
• preset-do-ssh → /etc/do-ssh/{private,public}-ssh-key
• Azure sets none, so it still generates

Behavior changes only where there was previously no working path at all (DigitalOcean and GCE-with-project without their variables), or where a flag was being ignored (Azure).

Testing

• New TestEnvExportsSSHKeyAndUser: asserts env() exports both variables on azure, digitalocean and aws.
• New TestResolveSSHKeys: 11 cases covering flag precedence, cloud-agnostic vars winning over legacy ones, each preset's variables resolving unchanged, one cloud's variables not leaking into another, the .pub sibling lookup, and generation.
• go build ./..., go vet ./..., and go test ./... in tests/e2e. Two pre-existing failures are unrelated and reproduce identically on master: kubetest2-kops/aws TestRandomZones (needs AWS credentials) and scenarios/bare-metal (needs a live cluster at 127.0.0.1:6443).
• Checked the two scenarios that invoke kubetest2 with --test but no --up (aws-boskos/run-test.sh, lib/upgrade.sh): both pass --cluster-name explicitly, and CreateSSHKeyPair is deterministic per cluster name and skips regeneration when the key already exists, so repeated invocations in one pod share a key.

Follow-ups

• Derive the SSH user from kops toolbox dump (instance.SSHUser), which is populated on all four clouds, so that it no longer has to come from the job config at all.
• Then drop KUBE_SSH_KEY_PATH, KUBE_SSH_USER, preset-aws-ssh, preset-k8s-ssh, and preset-do-ssh from the job configs.

Note for the GCE part of that follow-up: GCE jobs currently SSH as prow, whose key comes from GCP project metadata, not from kops. An ephemeral key lands in instance metadata under the image-derived user (ubuntu on Ubuntu, admin elsewhere), so the GCE KUBE_SSH_USER has to change in the same step. I confirmed on u2404, cos125, deb12 and rocky10 job artifacts that the guest agent already creates that user and installs kops's key for it, and that the user has sudo.

🤖 Generated with Claude Code