Sitelet https://github.com/kubernetes/kops/pull/18683/files
Skip to content

containerd: default to 2.3.4 / runc 1.4.3 #18683

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kubernetes-prow merged 6 commits into from
Aug 13, 2026
Merged

containerd: default to 2.3.4 / runc 1.4.3 #18683

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
31ff59a
validation: require containerd 2.1.0 or newer
hakman Aug 13, 2026
e801963
containerd: default to 2.3.4 / runc 1.4.3
hakman Aug 13, 2026
73a8924
nodeup: remove containerd 1.x support
hakman Aug 13, 2026
72745c9
release notes: containerd 2.1 floor and 2.3.4 default
hakman Aug 13, 2026
0f9411b
./hack/update-expected.sh
hakman Aug 13, 2026
395caf8
assetdata: add hashes for latest Kubernetes patch releases
hakman Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
6 changes: 5 additions & 1 deletion docs/releases/1.37-NOTES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ This is a document to gather the release notes prior to the release.

* On AWS, the new `spec.cloudProvider.aws.useIPBasedNodeNames` field (`spec.cloudConfig.useIPBasedNodeNames` in the v1alpha2 API) names Kubernetes nodes after the EC2 private DNS name (e.g. `ip-10-0-0-1.eu-west-1.compute.internal`) instead of the EC2 instance ID, restoring the naming used before kOps 1.24. Only newly launched nodes are affected; roll the cluster to rename existing nodes.

* The default containerd version is now 2.3.4, with runc 1.4.3. When pinning `spec.containerd.version` to the 2.3 series, use 2.3.4 or newer: containerd 2.3.0 through 2.3.3 fail to start pods when the sandbox image is pinned by digest, which kOps configures by default ([containerd#13529](https://github.com/containerd/containerd/issues/13529)).

* containerd versions older than 2.1.0 are no longer supported. See the breaking changes section below for the required actions.

## Removal of gossip DNS

Support for gossip DNS, deprecated since kOps 1.29, has been removed. New gossip clusters will be rejected, and existing clusters will need to migrate before they can be upgraded. This affects clusters whose name ends in `.k8s.local` and that were not created with `--dns=none`; clusters using `--dns=none`, even with a `.k8s.local` name, are not affected.
Expand Down Expand Up @@ -45,7 +49,7 @@ As part of this removal, the `protokube` component, whose only remaining respons

## Other breaking changes

* None at this time
* Support for containerd versions older than 2.1.0 has been removed; `spec.containerd.version` values below 2.1.0 now fail validation. The containerd 1.6, 1.7, and 2.1 release lines are end of life upstream, and 2.0 reaches end of life in March 2027, during the kOps 1.37 support window. Remove the pinned version to use the default (2.3.4), or pin 2.1.0 or newer; nodes install the new runtime as they are replaced.

# Known Issues

Expand Down
12 changes: 6 additions & 6 deletions hack/generate-asset-hashes.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,24 +165,24 @@ generate_k8s_hashes 1.29 15
generate_k8s_hashes 1.30 14
generate_k8s_hashes 1.31 14
generate_k8s_hashes 1.32 13
generate_k8s_hashes 1.33 12
generate_k8s_hashes 1.34 8
generate_k8s_hashes 1.35 5
generate_k8s_hashes 1.36 1
generate_k8s_hashes 1.33 13
generate_k8s_hashes 1.34 10
generate_k8s_hashes 1.35 7
generate_k8s_hashes 1.36 3

# Generate runc hashes.
# The first argument is the major and minor version, the second is the maximum patch version.
generate_runc_hashes 1.1 15
generate_runc_hashes 1.2 9
generate_runc_hashes 1.3 5
generate_runc_hashes 1.4 2
generate_runc_hashes 1.4 3

# Generate containerd hashes.
generate_containerd_hashes 1.7 32
generate_containerd_hashes 2.0 9
generate_containerd_hashes 2.1 7
generate_containerd_hashes 2.2 4
generate_containerd_hashes 2.3 1
generate_containerd_hashes 2.3 4

# Generate CNI network plugins hashes.
# Note: argument is now base path and list of versions (because there is no 1.6.1)
Expand Down
121 changes: 7 additions & 114 deletions nodeup/pkg/model/containerd.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,15 +125,6 @@ func (b *ContainerdBuilder) installContainerd(c *fi.NodeupModelBuilderContext) e

// Add containerd binaries from containerd release package
f := b.Assets.FindMatches(regexp.MustCompile(`^bin/(containerd|ctr)`))
if len(f) == 0 {
// Add containerd binaries from containerd bundle package
f = b.Assets.FindMatches(regexp.MustCompile(`^(\./)?usr/local/bin/(containerd|crictl|ctr)`))
}
if len(f) == 0 {
// Add containerd binaries from Docker package (for ARM64 builds < v1.6.0)
// https://github.com/containerd/containerd/pull/6196
f = b.Assets.FindMatches(regexp.MustCompile(`^docker/(containerd|ctr)`))
}
if len(f) == 0 {
return fmt.Errorf("unable to find any containerd binaries in assets")
}
Expand All @@ -150,15 +141,6 @@ func (b *ContainerdBuilder) installContainerd(c *fi.NodeupModelBuilderContext) e
// Add runc binary from https://github.com/opencontainers/runc
// https://github.com/containerd/containerd/issues/6541
f = b.Assets.FindMatches(regexp.MustCompile(`/runc\.(amd64|arm64)$`))
if len(f) == 0 {
// Add runc binary from containerd package (for builds < v1.6.0)
f = b.Assets.FindMatches(regexp.MustCompile(`^(\./)?usr/local/sbin/runc$`))
}
if len(f) == 0 {
// Add runc binary from Docker package (for ARM64 builds < v1.6.0)
// https://github.com/containerd/containerd/pull/6196
f = b.Assets.FindMatches(regexp.MustCompile(`^docker/runc$`))
}
if len(f) != 1 {
return fmt.Errorf("error finding runc asset")
}
Expand Down Expand Up @@ -491,101 +473,14 @@ func (b *ContainerdBuilder) buildCNIConfigTemplateFile(c *fi.NodeupModelBuilderC
return nil
}

// containerdV3MinVersion is the lowest containerd release that requires the v3 config schema.
var containerdV3MinVersion = semver.MustParse("2.0.0")

// buildContainerdConfig dispatches between the legacy v2 schema and the v3 schema introduced in containerd 2.0.
// Callers must short-circuit ConfigOverride before calling this.
func (b *ContainerdBuilder) buildContainerdConfig() (string, error) {
containerd := b.NodeupConfig.ContainerdConfig
v3, err := useContainerdConfigV3(fi.ValueOf(containerd.Version))
if err != nil {
return "", err
}
if v3 {
return b.buildContainerdConfigV3()
}
return b.buildContainerdConfigV2()
}

// useContainerdConfigV3 reports whether the configured containerd version expects the v3 config schema.
// containerd 2.0 introduced v3 and split the io.containerd.grpc.v1.cri plugin into separate runtime/images plugins.
// An empty version defaults to v2 (the safer default; containerd < 2.0 cannot read v3 at all).
// An unparseable version is a configuration error and surfaces as an error.
func useContainerdConfigV3(version string) (bool, error) {
if version == "" {
return false, nil
}
sv, err := semver.ParseTolerant(version)
if err != nil {
return false, fmt.Errorf("parsing containerd version %q: %w", version, err)
}
return sv.GTE(containerdV3MinVersion), nil
}

// buildContainerdConfigV2 builds the containerd v2 schema config used for containerd < 2.0.
//
// LEGACY: this is kept for the containerd 1.7 default that ships with k8s < 1.32 (see
// pkg/model/components/containerd.go where Containerd.Version is set). When kops drops support
// for k8s < 1.32, delete this function and switch the dispatcher to always call V3.
func (b *ContainerdBuilder) buildContainerdConfigV2() (string, error) {
containerd := b.NodeupConfig.ContainerdConfig

// toml.Load("") never fails for empty input; we use it to obtain an empty tree.
config, _ := toml.Load("")
config.SetPath([]string{"version"}, int64(2))

if containerd.NRI != nil && (containerd.NRI.Enabled == nil || fi.ValueOf(containerd.NRI.Enabled)) {
config.SetPath([]string{"plugins", "io.containerd.nri.v1.nri", "disable"}, false)
if containerd.NRI.PluginRequestTimeout != nil {
config.SetPath([]string{"plugins", "io.containerd.nri.v1.nri", "plugin_request_timeout"}, containerd.NRI.PluginRequestTimeout)
}
if containerd.NRI.PluginRegistrationTimeout != nil {
config.SetPath([]string{"plugins", "io.containerd.nri.v1.nri", "plugin_registration_timeout"}, containerd.NRI.PluginRegistrationTimeout)
}
}
if containerd.SeLinuxEnabled {
config.SetPath([]string{"plugins", "io.containerd.grpc.v1.cri", "enable_selinux"}, true)
}
if containerd.SandboxImage != nil {
config.SetPath([]string{"plugins", "io.containerd.grpc.v1.cri", "sandbox_image"}, fi.ValueOf(containerd.SandboxImage))
}
if len(containerd.RegistryMirrors) > 0 {
config.SetPath([]string{"plugins", "io.containerd.grpc.v1.cri", "registry", "config_path"}, containerdRegistryDirPath)
}
config.SetPath([]string{"plugins", "io.containerd.grpc.v1.cri", "containerd", "default_runtime_name"}, "runc")
config.SetPath([]string{"plugins", "io.containerd.grpc.v1.cri", "containerd", "runtimes", "runc", "runtime_type"}, "io.containerd.runc.v2")
config.SetPath([]string{"plugins", "io.containerd.grpc.v1.cri", "containerd", "runtimes", "runc", "options", "SystemdCgroup"}, true)
if b.NodeupConfig.UsesKubenet {
// Using containerd with Kubenet requires special configuration.
// This is a temporary backwards-compatible solution for kubenet users and will be deprecated when Kubenet is deprecated:
// https://github.com/containerd/containerd/blob/master/docs/cri/config.md#cni-config-template
config.SetPath([]string{"plugins", "io.containerd.grpc.v1.cri", "cni", "conf_template"}, "/etc/containerd/config-cni.template")
}

if b.InstallNvidiaRuntime() {
if err := appendNvidiaGPURuntimeConfig(config, []string{"plugins", "io.containerd.grpc.v1.cri", "containerd", "runtimes"}); err != nil {
return "", fmt.Errorf("appending nvidia gpu runtime to v2 containerd config: %w", err)
}
}

if b.InstallGVisorRuntime() {
if err := appendGVisorRuntimeConfig(config, []string{"plugins", "io.containerd.grpc.v1.cri", "containerd", "runtimes"}); err != nil {
return "", fmt.Errorf("appending gvisor runtime to v2 containerd config: %w", err)
}
}

if err := applyConfigAdditions(config, containerd.ConfigAdditions); err != nil {
return "", fmt.Errorf("applying ConfigAdditions to v2 containerd config: %w", err)
}

return config.String(), nil
}

// buildContainerdConfigV3 builds the containerd v3 schema config used for containerd >= 2.0.
// buildContainerdConfig builds the containerd v3 schema config used for containerd >= 2.0.
// containerd 2.0 split io.containerd.grpc.v1.cri into io.containerd.cri.v1.runtime and io.containerd.cri.v1.images.
// See https://github.com/containerd/containerd/blob/main/docs/cri/config.md
func (b *ContainerdBuilder) buildContainerdConfigV3() (string, error) {
// containerd 2.3 introduced config version 4, but 2.1 and 2.2 reject config versions above 3,
// so we emit v3 as long as those releases are supported; 2.3+ migrates v3 on load without
// affecting any of the fields set here.
// Callers must short-circuit ConfigOverride before calling this.
func (b *ContainerdBuilder) buildContainerdConfig() (string, error) {
containerd := b.NodeupConfig.ContainerdConfig

// toml.Load("") never fails for empty input; we use it to obtain an empty tree.
Expand Down Expand Up @@ -680,7 +575,6 @@ func applyConfigAdditions(config *toml.Tree, additions map[string]intstr.IntOrSt
}

// appendNvidiaGPURuntimeConfig adds the "nvidia" runtime entry under runtimesPath.
// runtimesPath is schema-specific so the same helper can serve both v2 and v3 builders.
func appendNvidiaGPURuntimeConfig(config *toml.Tree, runtimesPath []string) error {
gpuConfig, err := toml.TreeFromMap(
map[string]interface{}{
Expand Down Expand Up @@ -709,7 +603,6 @@ func appendNvidiaGPURuntimeConfig(config *toml.Tree, runtimesPath []string) erro
}

// appendGVisorRuntimeConfig adds the "runsc" runtime entry under runtimesPath.
// runtimesPath is schema-specific so the same helper can serve both v2 and v3 builders.
func appendGVisorRuntimeConfig(config *toml.Tree, runtimesPath []string) error {
gvisorConfig, err := toml.TreeFromMap(
map[string]interface{}{
Expand Down Expand Up @@ -750,7 +643,7 @@ func (b *ContainerdBuilder) buildGVisorShimConfig(c *fi.NodeupModelBuilderContex
// buildRegistryHosts emits one hosts.toml per RegistryMirrors entry under containerdRegistryDirPath.
// The directory is referenced by registry.config_path in the main containerd config; the
// emit-files-iff-mirrors-non-empty condition here must stay in sync with the registry.config_path
// emission in buildContainerdConfigV2/V3.
// emission in buildContainerdConfig.
// containerd watches this directory at runtime, so no daemon reload is needed when a hosts.toml changes.
// containerd uses host declaration order as mirror priority, so endpoints are emitted in the order
// the user provided them rather than sorted (go-toml's Tree.String alphabetizes subtables).
Expand Down
35 changes: 0 additions & 35 deletions nodeup/pkg/model/containerd_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,41 +181,6 @@ func runContainerdBuilderTest(t *testing.T, key string, distro distributions.Dis
testutils.ValidateTasks(t, filepath.Join(basedir, "tasks.yaml"), context)
}

func TestUseContainerdConfigV3(t *testing.T) {
tests := []struct {
name string
version string
want bool
wantErr bool
}{
{name: "empty defaults to v2", version: "", want: false},
{name: "1.7.0 is v2", version: "1.7.0", want: false},
{name: "1.999.999 is v2", version: "1.999.999", want: false},
{name: "2.0.0 boundary is v3", version: "2.0.0", want: true},
{name: "2.0.0 with v prefix is v3", version: "v2.0.0", want: true},
{name: "2.1.6 is v3", version: "2.1.6", want: true},
{name: "3.0.0 is v3", version: "3.0.0", want: true},
{name: "unparseable returns error", version: "not-a-semver", wantErr: true},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got, err := useContainerdConfigV3(tc.version)
if tc.wantErr {
if err == nil {
t.Fatalf("useContainerdConfigV3(%q) error = nil, want non-nil", tc.version)
}
return
}
if err != nil {
t.Fatalf("useContainerdConfigV3(%q) error = %v, want nil", tc.version, err)
}
if got != tc.want {
t.Errorf("useContainerdConfigV3(%q) = %v, want %v", tc.version, got, tc.want)
}
})
}
}

func TestContainerdConfig(t *testing.T) {
b := &ContainerdBuilder{
NodeupModelContext: &NodeupModelContext{
Expand Down
2 changes: 1 addition & 1 deletion nodeup/pkg/model/tests/channels/cluster.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
cloudProvider: aws
configBase: memfs://clusters.example.com/minimal.example.com
containerd:
version: 1.3.4
version: 2.3.4
containerRuntime: containerd
etcdClusters:
- etcdMembers:
Expand Down
2 changes: 1 addition & 1 deletion nodeup/pkg/model/tests/containerdbuilder/complex/cluster.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
configBase: memfs://clusters.example.com/minimal.example.com
containerRuntime: containerd
containerd:
version: 1.4.4
version: 2.3.4
configAdditions:
plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler.test_bool_false: "false"
plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler.test_bool_true: "true"
Expand Down
32 changes: 21 additions & 11 deletions nodeup/pkg/model/tests/containerdbuilder/complex/tasks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,36 +45,46 @@ path: /etc/containerd/config-cni.template
type: file
---
contents: |
version = 2
version = 3

[plugins]

[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.k8s.io/pause:3.10.1"
[plugins."io.containerd.cri.v1.images"]

[plugins."io.containerd.cri.v1.images".pinned_images]
sandbox = "registry.k8s.io/pause:3.10.1"

[plugins."io.containerd.cri.v1.images".registry]
config_path = "/etc/containerd/certs.d"

[plugins."io.containerd.grpc.v1.cri".cni]
[plugins."io.containerd.cri.v1.runtime"]

[plugins."io.containerd.cri.v1.runtime".cni]
conf_template = "/etc/containerd/config-cni.template"

[plugins."io.containerd.grpc.v1.cri".containerd]
[plugins."io.containerd.cri.v1.runtime".containerd]
default_runtime_name = "runc"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc.options]
SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri"]

[plugins."io.containerd.grpc.v1.cri".containerd]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler]
test_bool_false = false
test_bool_true = true
test_int = 1
test_intstr = "1"
test_str = "test"

[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
path: /etc/containerd/config.toml
type: file
---
Expand Down
2 changes: 1 addition & 1 deletion nodeup/pkg/model/tests/containerdbuilder/flatcar/cluster.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
configBase: memfs://clusters.example.com/minimal.example.com
containerRuntime: containerd
containerd:
version: 1.4.4
version: 2.3.4
etcdClusters:
- etcdMembers:
- instanceGroup: master-us-test-1a
Expand Down
20 changes: 12 additions & 8 deletions nodeup/pkg/model/tests/containerdbuilder/flatcar/tasks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,25 +21,29 @@ path: /etc/containerd/config-cni.template
type: file
---
contents: |
version = 2
version = 3

[plugins]

[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.k8s.io/pause:3.10.1"
[plugins."io.containerd.cri.v1.images"]

[plugins."io.containerd.grpc.v1.cri".cni]
[plugins."io.containerd.cri.v1.images".pinned_images]
sandbox = "registry.k8s.io/pause:3.10.1"

[plugins."io.containerd.cri.v1.runtime"]

[plugins."io.containerd.cri.v1.runtime".cni]
conf_template = "/etc/containerd/config-cni.template"

[plugins."io.containerd.grpc.v1.cri".containerd]
[plugins."io.containerd.cri.v1.runtime".containerd]
default_runtime_name = "runc"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc.options]
SystemdCgroup = true
path: /etc/containerd/config.toml
type: file
Expand Down
2 changes: 1 addition & 1 deletion nodeup/pkg/model/tests/containerdbuilder/simple/cluster.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
configBase: memfs://clusters.example.com/minimal.example.com
containerRuntime: containerd
containerd:
version: 1.4.4
version: 2.3.4
etcdClusters:
- etcdMembers:
- instanceGroup: master-us-test-1a
Expand Down
Loading
Loading