Sitelet https://github.com/kubeflow/spark-operator/pull/3113
Skip to content

feat(scheduler): add Kubernetes-native Workload/PodGroup backend - #3113

Open
Rajadi16 wants to merge 18 commits into
kubeflow:masterfrom
Rajadi16:kep-2962/workload-scheduler
Open

feat(scheduler): add Kubernetes-native Workload/PodGroup backend#3113
Rajadi16 wants to merge 18 commits into
kubeflow:masterfrom
Rajadi16:kep-2962/workload-scheduler

Conversation

@Rajadi16

@Rajadi16 Rajadi16 commented Aug 23, 2026 •
edited
Loading

Copy link
Copy Markdown

Important

Stacked PR — depends on #3093.

Please review and merge #3093 first. Do not merge this PR before #3093.
Until #3093 lands, this PR intentionally shows the cumulative PR1 + PR2 diff.

Part of #2962.

Stack

  1. feat(api): add SchedulingGroup and MinMember fields for KEP-2962 workload scheduling #3093 — API types, validation, RBAC, CRDs, and generated API artifacts
  2. This PR — Kubernetes-native Workload/PodGroup scheduler implementation
  3. test: add workload scheduler documentation and end-to-end coverage #3116 — user documentation and end-to-end coverage

Purpose of this PR

Adds the Kubernetes-native workload batch-scheduler backend using scheduling.k8s.io/v1alpha2.

Proposed changes

  • Upgrade Kubernetes dependencies and code-generation tooling to v0.36.0, controller-runtime to v0.24.1, and the Go toolchain/builder image to Go 1.26.
  • Register the workload scheduler behind controller.batchScheduler.workload.enable.
  • Validate scheduling.k8s.io/v1alpha2 availability when the backend is selected.
  • Create one owner-referenced Workload per SparkApplication and one PodGroup per submission.
  • Gang-schedule executor pods only; the cluster-mode driver remains outside the PodGroup so it can create executors without deadlocking.
  • Use batchSchedulerOptions.minMember when specified; otherwise derive the DRA-aware initial executor count and clamp it to at least one.
  • Propagate the generated PodGroup reference into executor pod templates.
  • Clean up stale per-submission PodGroups before retries and terminal cleanup while retaining the application-owned Workload.
  • Reject foreign or policy-incompatible immutable Workload/PodGroup objects instead of silently reusing them.
  • Apply operator-managed scheduling metadata only to an ephemeral submission copy, so it is never persisted to the SparkApplication spec or rejected by the validating webhook.
  • Align root/E2E module dependencies and regenerate OpenAPI, Python API, CRDs, clients, informers, and deepcopy artifacts.
  • Add scheduler, controller, webhook-safety, retry-cleanup, ownership, and pod-template unit tests.

Ordering and webhook safety

The controller assigns a non-empty status.submissionID before scheduling; the workload scheduler also rejects an empty submission ID defensively.

Scheduling mutations are applied to a deep copy passed only to the submitter. The reconciled SparkApplication retains no operator-managed schedulingGroup in its spec, preventing the operator from triggering PR1’s webhook rejection on a later full-object update.

Prerequisites

The backend requires Kubernetes v1.36+ with:

  • scheduling.k8s.io/v1alpha2 served by the API server
  • GenericWorkload enabled on kube-apiserver and kube-scheduler
  • GangScheduling enabled on kube-scheduler

If batchSchedulerOptions.priorityClassName is used, the alpha WorkloadAwarePreemption feature gate must also be enabled on kube-apiserver and kube-scheduler.

Compatibility note

The Kubernetes/controller-runtime upgrade also required migrating the v1alpha1 SparkConnect scheme registration from deprecated controller-runtime/pkg/scheme.Builder to runtime.NewSchemeBuilder.

This is a compatibility-only refactor with no SparkConnect behavior change; it removes the staticcheck SA1019 failure introduced by controller-runtime v0.24.1.

The remaining broad generated-file changes are deterministic fallout from the Kubernetes v0.36.0 API and code-generator upgrade.

Change category

  • Bugfix
  • Feature
  • Dependency/toolchain update
  • Documentation update

Verification

  • make generate manifests
  • make verify-codegen
  • make build-api-docs
  • make detect-crds-drift
  • make go-fmt
  • make build-operator
  • make unit-test
  • make go-vet
  • make go-lint
  • make helm-unittest (230 tests)
  • git diff --check
  • All commits include Signed-off-by

Review note

This PR targets upstream master because the PR1 branch exists only in the contributor fork. After #3093 merges, its files should disappear from this PR’s diff.

If #3093 is squash-merged, this branch will be rebased onto the updated upstream master to remove the PR1 commits cleanly.

Rajadi16 added 17 commits August 21, 2026 21:49
@Rajadi16
feat(api): add SchedulingGroup and MinMember fields for KEP-2962 work…
9854a61 
…load scheduling

Adds the preparatory API surface for the upcoming native Kubernetes
Workload/PodGroup (scheduling.k8s.io/v1alpha2) batch-scheduler backend.
No user-facing behavior change in this release — the workload scheduler
name is not yet registered.

Changes:
- api/v1beta2: Add PodSchedulingGroup type and SchedulingGroup field on
  SparkPodSpec, plus MinMember field on BatchSchedulerConfiguration.
  SchedulingGroup is operator-managed (set by the workload scheduler backend);
  MinMember is an optional user override for the gang minCount.
- internal/webhook: Add validateWorkloadSchedulerFields() that rejects
  user-set schedulingGroup fields and emits a warning when
  batchSchedulerOptions.queue is set with batchScheduler=workload (queue
  has no effect on the Workload API). Table-driven unit tests included.
- charts: Add scheduling.k8s.io RBAC rule (workloads, podgroups) gated on
  new controller.batchScheduler.workload.enable Helm value (default false).
- Generated: deepcopy, CRD schemas (both SparkApplication and
  ScheduledSparkApplication), OpenAPI spec, Python SDK models.

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
test(webhook): remove unnecessary warning conversion
1994647 
Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
chore(api): sync chart CRDs and API docs
3e7818e 
Regenerate the Helm chart CRDs and API reference documentation for the workload scheduling API fields.

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
fix(api): validate workload scheduler gang size
20f5245 
Require a positive minMember, reject values above the initial executor gang size for the workload scheduler, and synchronize API documentation and generated artifacts.

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
chore: upgrade K8s dependencies to v0.36.0 for scheduling/v1alpha2 API
118e568 
Upgrade Kubernetes dependencies from v0.35.4 to v0.36.0 to access the
scheduling.k8s.io/v1alpha2 API (Workload and PodGroup resources) required
for the workload scheduler implementation.

Changes:
- k8s.io/api: v0.35.4 -> v0.36.0
- k8s.io/apimachinery: v0.35.4 -> v0.36.0
- k8s.io/client-go: v0.35.4 -> v0.36.0
- k8s.io/apiserver: v0.35.4 -> v0.36.0
- k8s.io/apiextensions-apiserver: v0.35.4 -> v0.36.0
- k8s.io/component-base: v0.35.4 -> v0.36.0
- k8s.io/code-generator: v0.35.4 -> v0.36.0
- All replace directives updated to v0.36.0

Note: K8s 1.36 APIs require Go 1.26+. Dockerfile and go.mod Go version
will be updated in subsequent commit.

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
chore: upgrade Go version to 1.26 for K8s 1.36 compatibility
649a187 
K8s 1.36 APIs require Go 1.26 or later. Update both Dockerfile and go.mod
to meet this requirement.

Changes:
- Dockerfile: golang:1.25.11 -> golang:1.26
- go.mod: go 1.25.0 -> go 1.26.0
- go.mod: toolchain go1.25.11 -> toolchain go1.26

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
chore: update go.sum for K8s v0.36.0 dependencies
55e02af 
Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
chore: update go.sum after go mod tidy with K8s v0.36.0
73a7e65 
Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
chore: upgrade controller-runtime to v0.24.1 for K8s v0.36 compatibility
e1a248a 
controller-runtime v0.23.x pairs with K8s v0.35 (client-go v0.35).
For K8s v0.36 compatibility, controller-runtime v0.24.x is required.

Upgraded:
- sigs.k8s.io/controller-runtime: v0.23.3 -> v0.24.1
- github.com/onsi/ginkgo/v2: v2.27.2 -> v2.27.4 (transitive)
- github.com/onsi/gomega: v1.38.2 -> v1.39.0 (transitive)

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
chore: regenerate manifests and OpenAPI with K8s v0.36.0
4d83455 
Run 'make manifests' to regenerate CRDs and OpenAPI code against
the upgraded K8s v0.36.0 dependencies.

Generated files updated:
- api/v1alpha1/zz_generated.openapi.go
- api/v1beta2/zz_generated.openapi.go
- api/openapi-spec/swagger.json
- config/crd/bases/*.yaml

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
feat: add workload scheduler implementation
4d7c834 
Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
chore: regenerate Python API for Kubernetes v0.36.0
b321009 
Include Python models and schema updates introduced by Kubernetes v0.36.0.

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
feat: complete Kubernetes workload scheduler integration
d72bf7a 
Register the workload backend, propagate PodGroup references to executor pods, align the E2E module with Kubernetes v0.36.0, and add scheduler and controller tests.

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
refactor(api): replace deprecated scheme builder
8ce7145 
Use runtime.NewSchemeBuilder for the v1alpha1 API to maintain compatibility with controller-runtime v0.24.1.

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
chore: pin Go 1.26 builder image
0c520ca 
Pin the Go 1.26.7 builder image by digest and normalize Dockerfile line endings.

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
chore: regenerate Kubernetes v0.36 codegen artifacts
bf0730c 
Update generated clients, informers, deepcopy code, chart CRDs, and the code-generator tool version for Kubernetes v0.36.0.

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16
fix(scheduler): harden workload submission lifecycle
78b756f 
Keep operator-managed scheduling metadata on an ephemeral submission copy, clean up stale per-submission PodGroups before retries, validate gang sizes and submission IDs, and reject conflicting immutable scheduling objects.

Document the complete Kubernetes feature-gate prerequisites and add regression coverage for ownership, policy drift, cleanup failures, and webhook-safe submission.

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@google-oss-prow
google-oss-prow Bot requested review from ImpSy and tariq-hasan August 23, 2026 13:44
@google-oss-prow

google-oss-prow Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jacobsalway for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown

🎉 Welcome to the Kubeflow Spark Operator! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

Join the community:

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

@Rajadi16
fix(scheduler): use kube-scheduler for workload backend
69f887d 
Avoid copying the workload backend name into pod.spec.schedulerName because Kubernetes-native workload scheduling is handled by kube-scheduler's GangScheduling plugin.

Preserve role-specific scheduler overrides and add regression coverage for driver and executor pods.

Part of kubeflow#2962

Signed-off-by: Rajadi16 <rajputadityasingh2407@gmail.com>
@Rajadi16 Rajadi16 mentioned this pull request Aug 23, 2026
Open
19 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ImpSy ImpSy Awaiting requested review from ImpSy
@tariq-hasan tariq-hasan Awaiting requested review from tariq-hasan

Assignees

No one assigned

Labels

size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Rajadi16