Sitelet https://github.com/kubernetes/kops/commit/ba3664bd8a7789ea2cd9093af10b6261172f2f91
Skip to content

Commit ba3664b

Browse files
hakmanhakman
committed
azure: download nodeup from Blob Storage with curl
When KOPS_BASE_URL is an azureblob://<account>/<container>/<key> URL on an Azure cluster, the bootstrap script downloads nodeup with curl. It requests an OAuth token for the system-assigned managed identity from the instance metadata service and passes the Authorization and x-ms-version headers to curl through stdin, so the token never reaches disk, process arguments, or console logs. The download URL hard-codes the public blob.core.windows.net endpoint, so validation rejects non-public AZURE_ENVIRONMENT values. Source locations are percent-escaped and validated when the script is rendered, so malformed URLs fail during kops update rather than in the boot retry loop. spec.assets.fileRepository now accepts an azureblob:// URL on Azure, so that the node assets can be hosted in the same private container. The URL must include a container, and nodes on other clouds cannot authenticate to Azure Blob Storage, so validation keeps rejecting it there. Nodeup reads these assets through VFS using the managed identity credentials, and "kops get assets --copy" can now also write to an azureblob:// repository. Access is not granted automatically: the docs describe granting Storage Blob Data Reader on the assets container only, never on the state-store account, which would let nodes read the cluster PKI.
1 parent bc8e90b commit ba3664b

11 files changed

Lines changed: 292 additions & 12 deletions

File tree

‎docs/operations/asset-repository.md‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,20 @@ spec:
7171
fileRepository: s3://example-bucket/files
7272
```
7373

74+
{{ kops_feature_table(kops_added_default='1.37') }}
75+
76+
On Azure, the repository can also be an `azureblob://<account>/<container>/<prefix>` URL.
77+
Nodes then read it with their system-assigned managed identity, which allows the storage
78+
account to be private. The managed identities of the instance groups have to be granted
79+
`Storage Blob Data Reader` on the assets container. Do not grant access to the state-store
80+
storage account, as that would let nodes read the cluster PKI.
81+
82+
```yaml
83+
spec:
84+
assets:
85+
fileRepository: azureblob://exampleaccount/assets/files
86+
```
87+
7488
## Copying assets into repositories
7589

7690
{{ kops_feature_table(kops_added_default='1.22') }}
@@ -80,9 +94,11 @@ You can copy assets into their repositories either by running `kops get assets -
8094
When running `kops get assets --copy`, kOps copies assets into their respective repositories if
8195
they do not already exist there.
8296

83-
For file assets, kOps only supports copying to a repository that is either an S3 or GCS bucket.
97+
For file assets, kOps only supports copying to a repository that is an S3 bucket, a GCS bucket,
98+
or an Azure Blob Storage container.
8499
An S3 bucket must be configured with a prefix of `s3://` or using the [regional naming conventions of S3](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region).
85100
A GCS bucket must be configured with a prefix of `https://storage.googleapis.com/` or `gs://`.
101+
An Azure Blob Storage container must be configured with a prefix of `azureblob://`.
86102

87103
## Listing assets
88104

‎docs/releases/1.37-NOTES.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@ As part of this removal, the `protokube` component, whose only remaining respons
3434

3535
* Private cluster asset repositories now support AWS `s3://` URLs in addition to existing GCE `gs://` URLs, both for `KOPS_BASE_URL` (nodeup download) and `spec.assets.fileRepository`. Nodes authenticate with their instance credentials; see the [asset repository documentation](https://kops.sigs.k8s.io/operations/asset-repository/) for the required permissions. The AWS nodeup download requires node images with curl 8.0 or newer.
3636

37+
* Private cluster asset repositories now also support Azure `azureblob://<account>/<container>/<prefix>` URLs, both for `KOPS_BASE_URL` (nodeup download) and `spec.assets.fileRepository`. Nodes authenticate with their system-assigned managed identity, which has to be granted the `Storage Blob Data Reader` role on the assets container; see the [asset repository documentation](https://kops.sigs.k8s.io/operations/asset-repository/) for the details and warnings.
38+
3739
# Breaking changes
3840

3941
* Support for AWS Classic Load Balancer (CLB) for the API has been removed. Clusters with `spec.api.loadBalancer.class: Classic` (or with no explicit `class`, which previously defaulted to Classic) fail validation, and the long-deprecated `kops create cluster --api-loadbalancer-class` flag has been removed. Existing clusters using a CLB must migrate to a Network Load Balancer (NLB) using kOps 1.36 or earlier before upgrading to kOps 1.37, following the [CLB to NLB migration guide](https://github.com/kubernetes/kops/blob/master/permalinks/acm_nlb.md). Attaching instance groups to externally-managed Classic Load Balancers via `spec.externalLoadBalancers[].loadBalancerName` remains supported.

‎pkg/apis/kops/validation/validation.go‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -806,8 +806,17 @@ func validateFileRepository(s string, fieldPath *field.Path, cloudProvider kops.
806806
if cloudProvider != kops.CloudProviderAWS {
807807
allErrs = append(allErrs, field.Invalid(fieldPath, s, fmt.Sprintf("s3:// fileRepository is only supported on AWS, but the cloud provider is %q", cloudProvider)))
808808
}
809+
case "azureblob":
810+
// Only Azure instances can authenticate to Azure Blob Storage with their managed identity.
811+
if cloudProvider != kops.CloudProviderAzure {
812+
allErrs = append(allErrs, field.Invalid(fieldPath, s, fmt.Sprintf("azureblob:// fileRepository is only supported on Azure, but the cloud provider is %q", cloudProvider)))
813+
}
814+
// Without a container, each remapped asset would treat its first path segment as the container.
815+
if container, _, _ := strings.Cut(strings.TrimPrefix(u.Path, "/"), "/"); container == "" {
816+
allErrs = append(allErrs, field.Invalid(fieldPath, s, "azureblob:// fileRepository must include a container: azureblob://<account>/<container>/<path>"))
817+
}
809818
default:
810-
allErrs = append(allErrs, field.Invalid(fieldPath, s, "fileRepository must be an http://, https://, gs://, or s3:// URL"))
819+
allErrs = append(allErrs, field.Invalid(fieldPath, s, "fileRepository must be an http://, https://, gs://, s3://, or azureblob:// URL"))
811820
}
812821
if u.Host == "" {
813822
allErrs = append(allErrs, field.Invalid(fieldPath, s, "fileRepository must include a host"))

‎pkg/apis/kops/validation/validation_test.go‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2305,6 +2305,25 @@ func TestValidateFileRepository(t *testing.T) {
23052305
Input: "s3://example-k8s-assets/kops",
23062306
ExpectedErrors: []string{"Invalid value::spec.assets.fileRepository"},
23072307
},
2308+
{
2309+
Input: "azureblob://exampleaccount/assets/kops",
2310+
CloudProvider: kops.CloudProviderAzure,
2311+
},
2312+
{
2313+
Input: "azureblob://exampleaccount/assets/kops",
2314+
CloudProvider: kops.CloudProviderGCE,
2315+
ExpectedErrors: []string{"Invalid value::spec.assets.fileRepository"},
2316+
},
2317+
{
2318+
Input: "azureblob://exampleaccount/assets/kops",
2319+
ExpectedErrors: []string{"Invalid value::spec.assets.fileRepository"},
2320+
},
2321+
{
2322+
// A container is required so that remapped assets share one container.
2323+
Input: "azureblob://exampleaccount",
2324+
CloudProvider: kops.CloudProviderAzure,
2325+
ExpectedErrors: []string{"Invalid value::spec.assets.fileRepository"},
2326+
},
23082327
{
23092328
// Nodes download from GCS with the credentials of their service account.
23102329
Input: "gs://example-k8s-assets/kops",

‎pkg/assets/assetcopy/copyfile.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -176,11 +176,11 @@ func writeFile(ctx context.Context, cluster *kops.Cluster, p vfs.Path, data []by
176176
return nil
177177
}
178178

179-
// buildVFSPath returns local paths and memfs://, file://, gs://, and s3:// URLs unchanged.
179+
// buildVFSPath returns local paths and memfs://, file://, gs://, s3://, and azureblob:// URLs unchanged.
180180
// It converts recognized S3 or GCS HTTPS URLs to their native VFS form.
181181
func buildVFSPath(target string) (string, error) {
182182
if !strings.Contains(target, "://") || strings.HasPrefix(target, "memfs://") || strings.HasPrefix(target, "file://") ||
183-
strings.HasPrefix(target, "gs://") || strings.HasPrefix(target, "s3://") {
183+
strings.HasPrefix(target, "gs://") || strings.HasPrefix(target, "s3://") || strings.HasPrefix(target, "azureblob://") {
184184
return target, nil
185185
}
186186

‎pkg/assets/assetcopy/copyfile_test.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,11 @@ func Test_BuildVFSPath(t *testing.T) {
5656
"s3://k8s-for-greeks-kops/kubernetes-release/release/v1.7.2/bin/linux/amd64/kubectl",
5757
true,
5858
},
59+
{
60+
"azureblob://exampleaccount/assets/kubernetes-release/release/v1.7.2/bin/linux/amd64/kubectl",
61+
"azureblob://exampleaccount/assets/kubernetes-release/release/v1.7.2/bin/linux/amd64/kubectl",
62+
true,
63+
},
5964
{
6065
"https://foo/k8s-for-greeks-kops/kubernetes-release/release/v1.7.2/bin/linux/amd64/kubectl",
6166
"",

‎pkg/model/resources/nodeup.go‎

Lines changed: 67 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,9 @@ imds-get() {
8080
curl -s -f --noproxy '*' --connect-timeout 2 --max-time 5 \
8181
-H "X-aws-ec2-metadata-token: $1" "http://169.254.169.254/latest/$2"
8282
}
83+
{{- end }}
84+
85+
{{- if or UseS3Download UseAzureBlobDownload }}
8386
8487
# Extract a string field from a JSON object. args: json, field
8588
json-field() {
@@ -123,6 +126,30 @@ download-or-bust() {
123126
echo "== Downloaded ${url} with hash ${hash} =="
124127
return 0
125128
fi
129+
{{- else if UseAzureBlobDownload }}
130+
local rest account response token
131+
echo "== Downloading ${url} =="
132+
rest="${url#azureblob://}"
133+
account="${rest%%/*}"
134+
rest="${rest#*/}"
135+
# Use the IP of the metadata server, to not depend on DNS this early in boot.
136+
# The token request is brokered to Entra ID, so allow more time than for other clouds.
137+
if ! response=$(curl -s -f --noproxy '*' --connect-timeout 5 --max-time 30 -H 'Metadata: true' "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fstorage.azure.com%2F"); then
138+
echo "== Failed to get a managed identity token =="
139+
elif ! token=$(json-field "${response}" access_token); then
140+
echo "== Failed to parse the managed identity token =="
141+
# Pass the token through stdin so it does not appear in files, logs, or process arguments.
142+
elif ! printf 'Authorization: Bearer %s\nx-ms-version: 2017-11-09\n' "${token}" |
143+
curl -f -Lo "${file}" --connect-timeout 20 --retry 6 --retry-delay 10 -H @- \
144+
"https://${account}.blob.core.windows.net/${rest}"; then
145+
echo "== Failed to download ${url} =="
146+
elif ! validate-hash "${file}" "${hash}"; then
147+
echo "== Failed to validate hash for ${url} =="
148+
rm -f "${file}"
149+
else
150+
echo "== Downloaded ${url} with hash ${hash} =="
151+
return 0
152+
fi
126153
{{- else if UseS3Download }}
127154
local imds_token profile creds access_key secret_key session_token
128155
echo "== Downloading ${url} =="
@@ -267,10 +294,16 @@ func (b *NodeUpScript) nodeUpSource(arch architectures.Architecture) (string, er
267294

268295
locations := slices.Clone(asset.Locations)
269296
for i, location := range locations {
270-
if !strings.HasPrefix(location, "s3://") {
297+
var escape func(string) (string, error)
298+
switch {
299+
case strings.HasPrefix(location, "s3://"):
300+
escape = escapeS3Location
301+
case strings.HasPrefix(location, "azureblob://"):
302+
escape = escapeAzureBlobLocation
303+
default:
271304
continue
272305
}
273-
escaped, err := escapeS3Location(location)
306+
escaped, err := escape(location)
274307
if err != nil {
275308
return "", fmt.Errorf("escaping nodeup source %q: %w", location, err)
276309
}
@@ -291,6 +324,21 @@ func escapeS3Location(location string) (string, error) {
291324
return "s3://" + u.Host + httpbinding.EscapePath(u.Path, false), nil
292325
}
293326

327+
func escapeAzureBlobLocation(location string) (string, error) {
328+
u, err := url.Parse(location)
329+
if err != nil {
330+
return "", fmt.Errorf("parsing Azure Blob location: %w", err)
331+
}
332+
container, key, _ := strings.Cut(strings.TrimPrefix(u.Path, "/"), "/")
333+
// Reject ports, IPv6 hosts, userinfo, queries, and fragments, which the account-based
334+
// blob.core.windows.net URL cannot represent, so they fail here instead of in the boot retry loop.
335+
if u.Scheme != "azureblob" || u.Host == "" || u.Hostname() != u.Host || u.User != nil || u.RawQuery != "" || u.Fragment != "" || container == "" || key == "" {
336+
return "", fmt.Errorf("invalid Azure Blob location; expected azureblob://<account>/<container>/<key>")
337+
}
338+
339+
return "azureblob://" + u.Host + httpbinding.EscapePath(u.Path, false), nil
340+
}
341+
294342
func (b *NodeUpScript) Build() (fi.Resource, error) {
295343
if b.ProxyEnv == nil {
296344
b.ProxyEnv = funcEmptyString
@@ -303,6 +351,14 @@ func (b *NodeUpScript) Build() (fi.Resource, error) {
303351
return nil, fmt.Errorf("ResolveS3Region must be called before building a nodeup script with an s3:// source")
304352
}
305353

354+
if b.useAzureBlobDownload() {
355+
// The script hard-codes the public cloud blob.core.windows.net endpoint suffix.
356+
// Azure environment names are case-insensitive; AzureCloud is the CLI name of the public cloud.
357+
if azureEnv := os.Getenv("AZURE_ENVIRONMENT"); azureEnv != "" && !strings.EqualFold(azureEnv, "AzurePublicCloud") && !strings.EqualFold(azureEnv, "AzureCloud") {
358+
return nil, fmt.Errorf("downloading nodeup from an azureblob:// URL is not supported in Azure environment %q", azureEnv)
359+
}
360+
}
361+
306362
functions := template.FuncMap{
307363
"NodeUpSourceAmd64": func() (string, error) {
308364
return b.nodeUpSource(architectures.ArchitectureAmd64)
@@ -349,9 +405,10 @@ func (b *NodeUpScript) Build() (fi.Resource, error) {
349405
"ProxyEnv": b.ProxyEnv,
350406
"EnvironmentVariables": b.EnvironmentVariables,
351407

352-
"UseGCSDownload": b.useGCSDownload,
353-
"UseS3Download": b.useS3Download,
354-
"S3Region": func() string { return b.S3Region },
408+
"UseGCSDownload": b.useGCSDownload,
409+
"UseS3Download": b.useS3Download,
410+
"UseAzureBlobDownload": b.useAzureBlobDownload,
411+
"S3Region": func() string { return b.S3Region },
355412
}
356413

357414
return newTemplateResource("nodeup", nodeUpTemplate, functions, nil)
@@ -384,6 +441,11 @@ func (b *NodeUpScript) useS3Download() bool {
384441
return b.CloudProvider == string(kops.CloudProviderAWS) && b.firstLocationWithScheme("s3://") != ""
385442
}
386443

444+
// Azure Blob downloads require a managed identity on the instance.
445+
func (b *NodeUpScript) useAzureBlobDownload() bool {
446+
return b.CloudProvider == string(kops.CloudProviderAzure) && b.firstLocationWithScheme("azureblob://") != ""
447+
}
448+
387449
// ResolveS3Region resolves the bucket region because SigV4 requires it but s3:// URLs omit it.
388450
func (b *NodeUpScript) ResolveS3Region(ctx context.Context, vfsContext *vfs.VFSContext) error {
389451
if b.CloudProvider != string(kops.CloudProviderAWS) {

‎pkg/model/resources/nodeup_test.go‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,152 @@ func Test_S3Download(t *testing.T) {
207207
}
208208
}
209209

210+
func TestEscapeAzureBlobLocation(t *testing.T) {
211+
for _, tc := range []struct {
212+
name string
213+
location string
214+
expected string
215+
expectErr bool
216+
}{
217+
{
218+
name: "plain path",
219+
location: "azureblob://exampleaccount/assets/kops/1.37.0/linux/amd64/nodeup",
220+
expected: "azureblob://exampleaccount/assets/kops/1.37.0/linux/amd64/nodeup",
221+
},
222+
{
223+
name: "plus sign",
224+
location: "azureblob://exampleaccount/assets/kops/1.37.0+abcdef/linux/amd64/nodeup",
225+
expected: "azureblob://exampleaccount/assets/kops/1.37.0%2Babcdef/linux/amd64/nodeup",
226+
},
227+
{
228+
name: "existing escape",
229+
location: "azureblob://exampleaccount/assets/kops/1.37.0%2Babcdef/linux/amd64/nodeup",
230+
expected: "azureblob://exampleaccount/assets/kops/1.37.0%2Babcdef/linux/amd64/nodeup",
231+
},
232+
{
233+
name: "missing account",
234+
location: "azureblob:///assets/kops/nodeup",
235+
expectErr: true,
236+
},
237+
{
238+
name: "missing container",
239+
location: "azureblob://exampleaccount",
240+
expectErr: true,
241+
},
242+
{
243+
name: "missing key",
244+
location: "azureblob://exampleaccount/assets",
245+
expectErr: true,
246+
},
247+
{
248+
name: "invalid escape",
249+
location: "azureblob://exampleaccount/assets/100%/nodeup",
250+
expectErr: true,
251+
},
252+
{
253+
name: "port in host",
254+
location: "azureblob://exampleaccount:443/assets/kops/nodeup",
255+
expectErr: true,
256+
},
257+
{
258+
name: "userinfo",
259+
location: "azureblob://user@exampleaccount/assets/kops/nodeup",
260+
expectErr: true,
261+
},
262+
{
263+
name: "query string",
264+
location: "azureblob://exampleaccount/assets/kops/nodeup?sig=secret",
265+
expectErr: true,
266+
},
267+
} {
268+
t.Run(tc.name, func(t *testing.T) {
269+
actual, err := escapeAzureBlobLocation(tc.location)
270+
if tc.expectErr {
271+
if err == nil {
272+
t.Fatalf("expected an error escaping %q", tc.location)
273+
}
274+
return
275+
}
276+
if err != nil {
277+
t.Fatalf("escaping %q: %v", tc.location, err)
278+
}
279+
if actual != tc.expected {
280+
t.Errorf("expected %q, got %q", tc.expected, actual)
281+
}
282+
})
283+
}
284+
}
285+
286+
func Test_AzureBlobDownload(t *testing.T) {
287+
azureBlobMarker := "blob.core.windows.net"
288+
standardMarker := "wget --compression=auto"
289+
290+
for _, tc := range []struct {
291+
name string
292+
cloudProvider string
293+
location string
294+
expectedSource string
295+
expectAzureBlob bool
296+
}{
297+
{
298+
name: "azureblob source",
299+
cloudProvider: "azure",
300+
location: "azureblob://exampleaccount/assets/kops/1.34.0+abcdef/linux/amd64/nodeup",
301+
expectedSource: "NODEUP_URL_AMD64=azureblob://exampleaccount/assets/kops/1.34.0%2Babcdef/linux/amd64/nodeup",
302+
expectAzureBlob: true,
303+
},
304+
{
305+
name: "default https source",
306+
cloudProvider: "azure",
307+
location: "https://artifacts.k8s.io/binaries/kops/1.34.0/linux/amd64/nodeup",
308+
expectAzureBlob: false,
309+
},
310+
{
311+
name: "azureblob source on another cloud provider",
312+
cloudProvider: "hetzner",
313+
location: "azureblob://exampleaccount/assets/kops/1.34.0/linux/amd64/nodeup",
314+
expectAzureBlob: false,
315+
},
316+
} {
317+
t.Run(tc.name, func(t *testing.T) {
318+
script := &NodeUpScript{
319+
CloudProvider: tc.cloudProvider,
320+
NodeUpAssets: singleNodeUpAsset(tc.location),
321+
}
322+
rendered := renderNodeUpScript(t, script)
323+
if got := strings.Contains(rendered, azureBlobMarker); got != tc.expectAzureBlob {
324+
t.Errorf("authenticated Azure Blob download rendered=%v, expected %v", got, tc.expectAzureBlob)
325+
}
326+
if tc.expectedSource != "" && !strings.Contains(rendered, tc.expectedSource) {
327+
t.Errorf("rendered script does not contain escaped source %q", tc.expectedSource)
328+
}
329+
if got := strings.Contains(rendered, standardMarker); got != !tc.expectAzureBlob {
330+
t.Errorf("standard download commands rendered=%v, expected %v", got, !tc.expectAzureBlob)
331+
}
332+
})
333+
}
334+
}
335+
336+
func Test_AzureBlobDownloadRequiresPublicCloud(t *testing.T) {
337+
script := &NodeUpScript{
338+
CloudProvider: "azure",
339+
NodeUpAssets: singleNodeUpAsset("azureblob://exampleaccount/assets/kops/1.34.0/linux/amd64/nodeup"),
340+
}
341+
342+
// Environment names are case-insensitive, and AzureCloud is the CLI name of the public cloud.
343+
for _, publicCloud := range []string{"AzurePublicCloud", "azurepubliccloud", "AzureCloud"} {
344+
t.Setenv("AZURE_ENVIRONMENT", publicCloud)
345+
if _, err := script.Build(); err != nil {
346+
t.Errorf("building an azureblob:// nodeup script with AZURE_ENVIRONMENT=%s: %v", publicCloud, err)
347+
}
348+
}
349+
350+
t.Setenv("AZURE_ENVIRONMENT", "AzureChinaCloud")
351+
if _, err := script.Build(); err == nil {
352+
t.Errorf("expected an error building an azureblob:// nodeup script in a non-public cloud")
353+
}
354+
}
355+
210356
func Test_S3DownloadRequiresRegion(t *testing.T) {
211357
script := &NodeUpScript{
212358
CloudProvider: "aws",

0 commit comments

Comments
 (0)