gce: download nodeup from private GCS buckets with curl by hakman · Pull Request #18623 · kubernetes/kops

hakman · 2026-07-25T11:31:39Z

What this PR does / why we need it:

#18466 added support for fetching nodeup from a private GCS bucket by prepending gcloud storage cp to the bootstrap script's download list, which requires the gcloud CLI on the node image. This uses plain curl instead, authenticated with the instance service account token from the metadata server. The token is piped in via -H @-, so it never reaches the disk, the process arguments, or the serial console.

Since the download URLs are known when the script is generated, the method is chosen there rather than by testing the URL scheme at boot. Scripts using the default https sources are unchanged on every provider, and the GCE golden files return to their pre-#18466 form.

Two follow-ons make it usable and tested: spec.assets.fileRepository now accepts a gs:// URL on GCE, so node assets can live in the same private bucket (validation still rejects it elsewhere, as only GCE instances can authenticate to GCS), and GCE e2e jobs now pass kops the gs:// form of the staged artifacts so the path is covered by CI.

/cc @cheftako @justinsb @rifelpet

kubernetes-prow · 2026-07-25T11:31:42Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

hakman · 2026-07-25T13:05:41Z

/test pull-kops-e2e-k8s-gce-ipalias

hakman · 2026-07-25T15:40:00Z

/test pull-kops-gce-master-scale-performance-100

hakman · 2026-07-25T17:49:02Z

/test pull-kops-e2e-aws-load-balancer-controller

cheftako · 2026-07-28T20:08:07Z

+	case "gs":
+		// Only GCE instances can authenticate to GCS with their service account.
+		if cloudProvider != kops.CloudProviderGCE {
+			allErrs = append(allErrs, field.Invalid(fieldPath, s, "gs:// fileRepository is only supported on GCE"))


I think it would be nice to include the current cloud provider in the error message. If they think they are using GCE (likely with a gs:// protocol) then knowing what its actually set to may help debugging.

Done. The message now includes the configured cloud provider.

cheftako · 2026-07-28T20:18:26Z

 func buildVFSPath(target string) (string, error) {
-	if !strings.Contains(target, "://") || strings.HasPrefix(target, "memfs://") || strings.HasPrefix(target, "file://") {
+	if !strings.Contains(target, "://") || strings.HasPrefix(target, "memfs://") || strings.HasPrefix(target, "file://") ||
+		strings.HasPrefix(target, "gs://") {


If we now exits early on "gs://" which is the expected form, then comment and code from lines 196-206 maybe should reflect they are dealing with the "http(s)://storage.googleapis.com/" edge case?

Good point. Updated the comment to make it clear that this branch now only handles the https://storage.googleapis.com/ form, and the error hint mentions both forms.

cheftako · 2026-07-28T20:23:23Z

+      local token
+      echo "== Downloading ${url} =="
+      # Pipe the service account token to curl, so that it stays out of the logs.
+      if ! token=$(curl -s -f --noproxy '*' --connect-timeout 2 --max-time 5 -H 'Metadata-Flavor: Google' 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token' | grep -o '"access_token" *: *"[^"]*"' | cut -d '"' -f 4); then


It seems like creating an entry for 169.254.169.254 to a hostname like LOCAL_METADATA would make this easier to understand. Would also allow us to change the IP to something like its IPv6 equivalent more easily.

Done, factored into a metadata_server variable. Kept the IP rather than metadata.google.internal so that early boot does not depend on DNS, which matches what the Go metadata client does. GCE does have an IPv6 endpoint these days, http://[fd20:ce::254], though only for IPv6-only VMs, which kOps does not create on GCE yet, as nodes are always dual-stack.

cheftako · 2026-07-28T21:45:11Z

+      # Pipe the service account token to curl, so that it stays out of the logs.
+      if ! token=$(curl -s -f --noproxy '*' --connect-timeout 2 --max-time 5 -H 'Metadata-Flavor: Google' 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token' | grep -o '"access_token" *: *"[^"]*"' | cut -d '"' -f 4); then
+        echo "== Failed to get a service account token =="
+      elif ! echo "Authorization: Bearer ${token}" | curl -f -Lo "${file}" --connect-timeout 20 --retry 6 --retry-delay 10 -H @- "https://storage.googleapis.com/${url#gs://}"; then


I'm used to using a few extra headers from https://docs.cloud.google.com/apis/docs/system-parameters but if this works for you, lets keep it simple.

Tested the plain Authorization: Bearer header against a real private bucket and it works, so keeping it simple.

cheftako · 2026-07-28T21:48:22Z

+			continue
+		}
+		for _, location := range asset.Locations {
+			if strings.HasPrefix(location, "gs://") {


Is there any chance this could be in the unsanitized "http(s)://storage.googleapis.com/" form?

It can, and that form takes the standard unauthenticated download path, same as before this PR: it works for public buckets and gets 403 for private ones. gs:// is the explicit opt-in for the authenticated download.

cheftako · 2026-07-28T21:51:11Z

+		if asset == nil {
+			continue
+		}
+		for _, location := range asset.Locations {


Is there any chance that we get a mix of gs:// and not gs:// ? Seems like this should be decided per location?

In practice no, mirrors are only added for the default artifacts.k8s.io base. A custom base, which is the only way to get gs://, always yields a single location.

PR kubernetes#18466 added support for fetching nodeup from a private GCS bucket by calling "gcloud storage cp", which requires the gcloud CLI to be installed on the node image. Use curl with the instance service account token instead, so that no extra tooling is needed on the node. The source URLs are known when the bootstrap script is generated, so the download method is chosen there rather than by inspecting the URL at boot. For the default https sources, the script goes back to the standard download commands, identical to the other providers. On GCE, spec.assets.fileRepository can now be a gs:// URL, so that the node assets can be hosted in the same private bucket. Nodes elsewhere cannot authenticate to GCS, so validation keeps rejecting it there. "kops get assets --copy" can now also write to a gs:// repository. GCE e2e jobs and dev-build-gce.sh now use the gs:// form of the staged artifacts, so that the download is covered by CI and by dev clusters. Only kops is given this form; the tooling that downloads the kops binary keeps using https.

hakman · 2026-07-29T05:29:37Z

/test pull-kops-verify-terraform

hakman · 2026-07-29T08:13:51Z

/retest

cheftako · 2026-07-29T20:46:12Z

+// nodes download the staged artifacts with their instance service-account credentials. Any other
+// URL, and any other cloud provider, passes through unchanged. Only kops invocations get the gs://
+// form; the scripts that download the kops binary run outside the deployer and keep using https.
+func (d *deployer) maybeGSurl(/sitelet?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fkops%2Fpull%2FbaseURL%2520string) string {


Nice to have. I don't find the outcome of a method called "maybeGSURL" intuitive. I would suggest something like "canonicalizeGSURL" or "standardizeGSURL" as better indicating what the method does.

canonicalizeBaseURL seems like a good option, thanks for the suggestion.
I will update the name in the followup.

cheftako · 2026-07-29T20:55:42Z

This seems cleaner. I like it. It would be nice if we can come up with a better name than "maybeGSURL".
/lgtm

hakman · 2026-07-30T05:55:12Z

/approve

kubernetes-prow · 2026-07-30T05:55:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

When KOPS_BASE_URL is an s3:// URL, the bootstrap script downloads nodeup with curl, signing the request with the instance profile credentials from IMDS using curl's native AWS SigV4 support. The credentials are piped in via --config -, so they never reach the disk, the process arguments, or the console log. An explicit x-amz-content-sha256 header lowers the required curl version from 8.0 to 7.86. The s3:// URL does not carry the bucket region, which SigV4 needs, so kops resolves it at script generation time through vfs, which already discovered and cached it when reading the nodeup hash from the bucket. spec.assets.fileRepository now accepts an s3:// URL on AWS, so node assets can live in the same private bucket. nodeup reads such assets through vfs, which signs requests with the instance credentials and resolves the bucket region. Also rename the kubetest2 deployer's maybeGSURL to canonicalizeBaseURL, as promised in the review of kubernetes#18623, and teach it to convert the public S3 https forms of staged artifacts to s3:// on AWS.

rifelpet · 2026-08-07T12:38:23Z

@hakman Looks like the curl version in some distros doesn't support gs://:

https://storage.googleapis.com/kubernetes-ci-logs/logs/e2e-kops-grid-gce-calico-deb13-k33-ko35/2083211374263013376/artifacts/136.109.232.232/journal.log

google_metadata_script_runner[1154]: Metadata key("startup-script"), command("/bin/bash"): == Failed to download gs://k8s-staging-kops/kops/releases/1.35.3+v1.35.1-39-gc89e13599b/linux/amd64/nodeup using wget --compression=auto -O nodeup --connect-timeout=20 --tries=6 --wait=10 ==
google_metadata_script_runner[1154]: Metadata key("startup-script"), command("/bin/bash"): == Downloading gs://k8s-staging-kops/kops/releases/1.35.3+v1.35.1-39-gc89e13599b/linux/amd64/nodeup using curl -f -Lo nodeup --connect-timeout 20 --retry 6 --retry-delay 10 ==
google_metadata_script_runner[1154]: Metadata key("startup-script"), command("/bin/bash"): curl: (1) Protocol "gs" not supported

https://storage.googleapis.com/kubernetes-ci-logs/logs/e2e-kops-grid-gce-calico-cos121arm64-k34-ko35/2084495582784655360/artifacts/136.115.187.38/journal.log

cloud-init[1242]: == Downloading gs://k8s-staging-kops/kops/releases/1.35.3+v1.35.1-39-gc89e13599b/linux/arm64/nodeup using curl -f --compressed -Lo nodeup --connect-timeout 20 --retry 6 --retry-delay 10 ==
cloud-init[1242]: curl: (1) Protocol "gs" not supported
cloud-init[1242]: == Failed to download gs://k8s-staging-kops/kops/releases/1.35.3+v1.35.1-39-gc89e13599b/linux/arm64/nodeup using curl -f --compressed -Lo nodeup --connect-timeout 20 --retry 6 --retry-delay 10 ==

https://storage.googleapis.com/kubernetes-ci-logs/logs/e2e-kops-grid-gce-calico-umini2404arm64-k33-ko33/2085325656857513984/artifacts/104.154.114.135/journal.log

cloud-init[1160]: == Failed to download gs://k8s-staging-kops/kops/releases/1.33.3+v1.33.1-46-g1d5f38108d/linux/arm64/nodeup using wget --compression=auto -O nodeup --connect-timeout=20 --tries=6 --wait=10 ==
cloud-init[1160]: == Downloading gs://k8s-staging-kops/kops/releases/1.33.3+v1.33.1-46-g1d5f38108d/linux/arm64/nodeup using curl -f -Lo nodeup --connect-timeout 20 --retry 6 --retry-delay 10 ==
cloud-init[1160]: curl: (1) Protocol "gs" not supported or disabled in libcurl

hakman · 2026-08-07T14:37:44Z

@rifelpet it looks like bug, curl should not get any 'gs://' url.

kubernetes-prow Bot requested review from cheftako, justinsb and rifelpet July 25, 2026 11:31

kubernetes-prow Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 25, 2026

hakman force-pushed the gce-nodeup-private-gcs-curl branch 4 times, most recently from fa05780 to e361409 Compare July 25, 2026 12:17

kubernetes-prow Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 25, 2026

hakman force-pushed the gce-nodeup-private-gcs-curl branch from e361409 to 63b4bcc Compare July 25, 2026 12:43

kubernetes-prow Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 25, 2026

hakman force-pushed the gce-nodeup-private-gcs-curl branch 2 times, most recently from b7735c5 to bf7d6fe Compare July 25, 2026 13:02

kubernetes-prow Bot added the area/api label Jul 25, 2026

hakman force-pushed the gce-nodeup-private-gcs-curl branch from bf7d6fe to 8edd197 Compare July 25, 2026 13:04

hakman force-pushed the gce-nodeup-private-gcs-curl branch from 8edd197 to 3cdd2c0 Compare July 25, 2026 13:42

kubernetes-prow Bot added the area/documentation label Jul 25, 2026

hakman marked this pull request as ready for review July 25, 2026 13:49

kubernetes-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 25, 2026

hakman force-pushed the gce-nodeup-private-gcs-curl branch from 3cdd2c0 to ccc958f Compare July 25, 2026 15:34

cheftako reviewed Jul 28, 2026

View reviewed changes

hakman added 2 commits July 29, 2026 08:09

./hack/update-expected.sh

be10c9e

hakman force-pushed the gce-nodeup-private-gcs-curl branch from ccc958f to be10c9e Compare July 29, 2026 05:11

cheftako reviewed Jul 29, 2026

View reviewed changes

kubernetes-prow Bot assigned cheftako Jul 29, 2026

kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 29, 2026

kubernetes-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 30, 2026

kubernetes-prow Bot merged commit a4ab99c into kubernetes:master Jul 30, 2026
28 checks passed

hakman deleted the gce-nodeup-private-gcs-curl branch July 30, 2026 06:56

hakman mentioned this pull request Aug 8, 2026

tests/e2e: use gs:// base URL only for kops versions supporting it #18670

Merged

Conversation

hakman commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kubernetes-prow Bot commented Jul 25, 2026

Uh oh!

hakman commented Jul 25, 2026

Uh oh!

hakman commented Jul 25, 2026

Uh oh!

hakman commented Jul 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hakman commented Jul 29, 2026

Uh oh!

hakman commented Jul 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cheftako commented Jul 29, 2026

Uh oh!

hakman commented Jul 30, 2026

Uh oh!

kubernetes-prow Bot commented Jul 30, 2026

Uh oh!

Uh oh!

rifelpet commented Aug 7, 2026

Uh oh!

hakman commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hakman commented Jul 25, 2026 •

edited

Loading