Sitelet https://github.com/GoogleCloudPlatform/evalbench/pull/695
Skip to content

feat(container): Kubernetes Job backend for per-case containers - #695

Open
prernakakkar-google wants to merge 3 commits into
mainfrom
container-k8s-backend
Open

prernakakkar-google wants to merge 3 commits into
mainfrom
container-k8s-backend

Conversation

@prernakakkar-google

Copy link
Copy Markdown
Collaborator

Third part of running each agent eval case in its own container. Builds on #694 (case runner and backend interface).

Adds KubernetesJobBackend, which runs each case as a Kubernetes Job on a dedicated worker node pool. Nothing calls it yet, so there is no behaviour change. The evaluator wiring follows in the next PR.

How it works

Each case becomes a ConfigMap holding its spec plus a Job, pinned to a worker pool. The result comes back through the pod's logs: the cluster's PVC is ReadWriteOnce, so there's no filesystem shared across nodes.

Designed for hundreds of concurrent cases

  • One shared cluster snapshot. Waiting cases share one LIST of case Jobs and one of case Pods per poll interval. Polling each Job separately would cost about 60 requests/s at 300 cases.
  • Nothing leaks if the server dies. Lifetimes are left to the Kubernetes garbage collector: the server pod owns each Job, and each Job owns its ConfigMap and Pod. On startup, the backend also deletes any still-running Jobs left behind by an earlier server process in the same pod.
  • Fast failure.
    • A pod that can't start (missing Secret key, bad image name) fails immediately.
    • One stuck pulling or mounting fails after a grace period instead of waiting out the whole Job deadline.
    • A missing case service account fails the run before anything is submitted.
  • Cancel and timeout stop the pod. A case that was cancelled, timed out or failed fast is always deleted rather than kept for inspection, so it stops spending tokens.

Safe defaults

  • Image: cases run the eval server's own image, pinned by digest. Every case in a run gets identical code even if :latest is pushed mid-run, and the case runner always matches the orchestrator.
  • Identity: case pods run as a separate service account, and its token isn't mounted, so the agent under eval can't reach the Kubernetes API.
  • Dependency: adds kubernetes>=36.0.3,<37. It's loaded only when this backend is constructed, so other runs never import it. Capped below 37, which pulls in a pydantic upgrade.

Testing

  • 54 unit tests in container_k8s_backend_test.py, built on real V1Job/V1ConfigMap objects with the API mocked. They cover the manifest, ownership, image resolution, preflight, the orphan sweep, every wait outcome (including "five waiting cases cost one LIST") and cleanup.
  • pycodestyle and pyflakes are clean, and uv lock --check passes.

Notes for reviewers

  • The RBAC and the evalbench-case-ksa service account this needs come in the deploy PR. Until then nothing calls this backend.
  • Nothing here names a GCP project: namespace and service account come from config, and the image from the server's own pod.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant