Sitelet https://github.com/kubernetes/kops/pull/18075
Skip to content

[aiconformance]: test for schedulingOrchestration pod autoscaling - #18075

Merged
k8s-ci-robot merged 9 commits into
kubernetes:masterfrom
justinsb:aiconformance_schedulingOrchestration_pod_autoscaling
Mar 19, 2026
Merged

[aiconformance]: test for schedulingOrchestration pod autoscaling#18075
k8s-ci-robot merged 9 commits into
kubernetes:masterfrom
justinsb:aiconformance_schedulingOrchestration_pod_autoscaling

Conversation

@justinsb

@justinsb justinsb commented Mar 16, 2026

Copy link
Copy Markdown
Member

No description provided.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 16, 2026
@k8s-ci-robot
k8s-ci-robot requested review from hakman and zetaab March 16, 2026 12:37
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 16, 2026
@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from 6fbe442 to 4e1c84b Compare March 16, 2026 12:40
@justinsb

justinsb commented Mar 16, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from 4e1c84b to 4241adf Compare March 16, 2026 14:14
@justinsb

justinsb commented Mar 16, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

1 similar comment
@justinsb

justinsb commented Mar 16, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from 6501d64 to 4241adf Compare March 16, 2026 16:20
@justinsb

justinsb commented Mar 16, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from 4241adf to d9095a5 Compare March 16, 2026 19:20
@justinsb

justinsb commented Mar 16, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from 6e4add4 to 0d473da Compare March 16, 2026 21:37
@justinsb

justinsb commented Mar 16, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from 0d473da to 7644f80 Compare March 16, 2026 22:47
@justinsb

justinsb commented Mar 16, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch 2 times, most recently from 9248521 to 0730545 Compare March 17, 2026 09:59
@justinsb

justinsb commented Mar 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch 2 times, most recently from 911cd9e to 0046d4d Compare March 17, 2026 12:11
@justinsb

justinsb commented Mar 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from 0046d4d to c3f81fa Compare March 17, 2026 13:49
@justinsb

justinsb commented Mar 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from c3f81fa to 8d8c394 Compare March 17, 2026 14:58
@justinsb

justinsb commented Mar 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch 2 times, most recently from 4dfbaa1 to 7604df6 Compare March 17, 2026 21:02
@justinsb

justinsb commented Mar 19, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from 0551e08 to 02f3908 Compare March 19, 2026 00:27
@justinsb

justinsb commented Mar 19, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from 02f3908 to c4472e0 Compare March 19, 2026 02:29
@justinsb

justinsb commented Mar 19, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

1 similar comment
@justinsb

justinsb commented Mar 19, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from 377dfed to b189380 Compare March 19, 2026 09:58
@justinsb
justinsb force-pushed the aiconformance_schedulingOrchestration_pod_autoscaling branch from b189380 to 11c2749 Compare March 19, 2026 10:00
@justinsb

justinsb commented Mar 19, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb
[aiconformance]: dump resources on namespace deletion failure
a02cacb 
This should show us what resources are blocking namespace deletion.
@justinsb

justinsb commented Mar 19, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-ai-conformance

@justinsb

justinsb commented Mar 19, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-k8s-aws-amazonvpc

@justinsb justinsb changed the title WIP: [aiconformance]: test for schedulingOrchestration pod autoscaling [aiconformance]: test for schedulingOrchestration pod autoscaling Mar 19, 2026
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 19, 2026
@justinsb

justinsb commented Mar 19, 2026

Copy link
Copy Markdown
Member Author

I think this is finally going to pass.

There were two problems:

  1. go test ./... will run package tests in parallel. I never knew that :-) So without cluster autoscaler, we were sometimes running out of resources (GPU)

  2. Deleting the namespace for the vllm custom-metric autoscaling case takes about 4m30s. Sometimes it takes longer. There's a five minute timeout currently. I think we're likely to flake a few more times, but what I have done is add the ability to capture a dump of the resources in the namespace, so we can understand what is going wrong. It feels like a bug in maybe cleaning up GPU Pods?

/assign @hakman @ameukam

@ameukam

ameukam commented Mar 19, 2026

Copy link
Copy Markdown
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 19, 2026
@hakman hakman closed this Mar 19, 2026
@hakman hakman reopened this Mar 19, 2026
@hakman

hakman commented Mar 19, 2026

Copy link
Copy Markdown
Member

Sorry, clicked close by mistake...
/approve

@k8s-ci-robot

k8s-ci-robot commented Mar 19, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 19, 2026
@k8s-ci-robot
k8s-ci-robot merged commit 93887f9 into kubernetes:master Mar 19, 2026
40 of 42 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.36 milestone Mar 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hakman hakman Awaiting requested review from hakman
@zetaab zetaab Awaiting requested review from zetaab

Assignees

@ameukam ameukam

@hakman hakman

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Milestone

v1.36

Development

Successfully merging this pull request may close these issues.

4 participants

@justinsb @ameukam @hakman @k8s-ci-robot