feat: add gVisor RuntimeClass support for containerd - #18406
Conversation
Add support for running workloads under gVisor (runsc) as a containerd runtime handler, following the existing NVIDIA GPU runtime pattern. gVisor is gated to Debian-family distributions only. The runsc apt package provides both runsc and containerd-shim-runsc-v1. Assisted by Opus 4.6 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
7c5a8e8 to
60b47c6
Compare
|
/test pull-kops-e2e-k8s-gce-distro-debian13 |
|
/retest |
|
@ameukam no need to deal with the tests for now, all good |
|
/test pull-kops-e2e-k8s-gce-distro-debian13 |
| func (c *NodeupModelContext) InstallGVisorRuntime() bool { | ||
| return c.NodeupConfig.GVisor != nil && | ||
| fi.ValueOf(c.NodeupConfig.GVisor.Enabled) && | ||
| c.Distribution.IsDebianFamily() |
There was a problem hiding this comment.
Nit: we should probably warn if we're not on debian but gvisor is enabled
Also, this probably gets to the whole "package" field naming :-)
| return nil | ||
| } | ||
|
|
||
| c.AddTask(&nodetasks.AptSource{ |
There was a problem hiding this comment.
Personally I would put the IsDebianFamily logic here, rather than in InstallGvisorRuntime.
| } | ||
|
|
||
| if (cluster.Spec.Containerd != nil && cluster.Spec.Containerd.GVisor != nil) || (instanceGroup.Spec.Containerd != nil && instanceGroup.Spec.Containerd.GVisor != nil) { | ||
| config.GVisor = buildGVisorConfig(cluster, instanceGroup) |
There was a problem hiding this comment.
Nit: we could probably put the if (cluster.Spec.Containerd != nil && cluster.Spec.Containerd.GVisor != nil) || (instanceGroup.Spec.Containerd != nil && instanceGroup.Spec.Containerd.GVisor != nil) into buildGVisorConfig
| } | ||
|
|
||
| // Label nodes that have gVisor enabled so the RuntimeClass nodeSelector works. | ||
| clusterGVisor := cluster.Spec.Containerd != nil && cluster.Spec.Containerd.GVisor != nil && fi.ValueOf(cluster.Spec.Containerd.GVisor.Enabled) |
There was a problem hiding this comment.
Aside: I wish we had cluster.Spec.GetContainerd().GetGVisor().GetEnabled(), like we get in proto.
| // everywhere including VMs) or "kvm" (bare-metal with KVM support). | ||
| Platform string `json:"platform,omitempty"` | ||
| // Packages overrides the URL and hash for the gVisor packages. | ||
| Packages *PackagesConfig `json:"packages,omitempty"` |
There was a problem hiding this comment.
Do we actually do anything with packages? I think I posted it before, but maybe I didn't... "packages" feels very specific to install deb or rpm packages. What if we want to install it from a tar.gz or a future format (e.g. a container image).
If we're not using it yet, that's awesome, we can just remove it for now and avoid the naming problems!
There was a problem hiding this comment.
Looks we want a SourcesSpec where we can be more explicity how we install binaries ? we may it limitations with immutable OS ?
hakman
left a comment
There was a problem hiding this comment.
Besides the Packages field being unused, there is also a more philosophical Q, should this be scoped to all types of IGs or just to workers and should it be allowed at cluster level or just meant to be specified for each IG.
@hakman I have been thinking about this. I theorized we may have use-cases where third-party workloads on the Control plane IGs but it's very speculative. I'll scope this only to the workers and probably allow specificity for each worker. |
I guess you men scope it per IG and block with validation the Cluster level change. |
| kOps now supports running workloads under [gVisor](https://gvisor.dev/) (runsc) as a containerd runtime handler. When enabled, kOps installs the `runsc` runtime via apt, configures containerd with a `runsc` runtime handler, and creates a `gvisor` RuntimeClass. Pods targeting the RuntimeClass are automatically scheduled to nodes with gVisor enabled via the `kops.k8s.io/gvisor` node label. | ||
|
|
||
| gVisor can be enabled at the cluster or instance group level: | ||
| gVisor can be enabled at the cluster level or on worker instance groups. Cluster-level configuration installs gVisor only on worker instance groups, and instance group-level configuration is accepted only for workers: |
There was a problem hiding this comment.
That's what I wanted to avoid, no cluster level setting with other weird gotchas as warm pools.
I would prefer to block the cluster setting via validation. If someone wants to enable it, do it one instance group at a time.
Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
gVisor (runsc) was previously installable on any instance group role. Restrict it to nodes with role Node: reject cluster/IG configs that enable gVisor on control plane, apiserver, or bastion roles, strip the gVisor config from non-worker nodeup configs, and only apply the gVisor node label and RuntimeClass addon when a worker has it enabled. Also harden nil handling for cluster.Spec.Containerd in nodeup config and bootstrapchannelbuilder. Update release notes and add tests across validation, nodeup config, gvisor builder, and instancegroup spec.
bb2a944 to
90d5ed3
Compare
|
/test pull-kops-e2e-k8s-gce-distro-debian13 |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hakman The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/override pull-kops-e2e-k8s-gce-distro-debian13 |
|
@hakman: Overrode contexts on behalf of hakman: pull-kops-e2e-k8s-gce-distro-debian13 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Add support for running workloads under gVisor (runsc) as a containerd runtime handler, following the existing NVIDIA GPU runtime pattern.
gVisor is gated to Debian-family distributions only. The runsc apt package provides both runsc and containerd-shim-runsc-v1.
Assisted by Opus 4.6