e2e: Filter random AZ selection by instance type availability - #18463
Conversation
|
Skipping CI for Draft Pull Request. |
|
The temp commit I added should override this job's pinning to eu-central-1a: /test pull-kops-e2e-arm64 |
confirmed it used us-east-2a: |
327e960 to
853f979
Compare
|
/lgtm |
| chosenZones = append(chosenZones, chosenRegion[randIndexes[i]]) | ||
| // Try regions in a random order so that a single region without the | ||
| // requested instance types does not deterministically fail. | ||
| regionOrder := rng.Perm(len(eligibleRegions)) |
| // zonesWithInstanceTypes returns the subset of zones in which every one of | ||
| // instanceTypes is offered, according to DescribeInstanceTypeOfferings. | ||
| func zonesWithInstanceTypes(region string, zones []string, instanceTypes []string) ([]string, error) { | ||
| ctx := context.Background() |
There was a problem hiding this comment.
Nit: I always prefer adding the ctx context.Context parameter if it's a new function, but ... not a blocker
| } | ||
| } | ||
|
|
||
| matching := make([]string, 0, len(zones)) |
There was a problem hiding this comment.
Nit: particularly in test code there's really no reason to bother with pre-allocation
| seen := make(map[string]bool) | ||
| var types []string |
There was a problem hiding this comment.
Nit: any reason to keep both, vs just building the slice at the end?
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: justinsb The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
currently kubetest2-kops selects random AWS availability zones if they aren't provided.
For many prow jobs we specify a specific zone because we want to use a specific instance type not offered in every zone:
https://github.com/search?q=repo%3Akubernetes%2Ftest-infra%20path%3Aconfig%2Fjobs%2Fkubernetes%2Fkops%2Fbuild_jobs.py%20--zones&type=code
This results in hot spots of zones, especially if the --zone flag is used in many jobs like in the grid definition. A claude analysis:
Along with a graph of EC2 instance launches in the kops prow account grouped by zone over the past week:
https://us5.datadoghq.com/s/82b785d3-4052-11f0-9c7a-f2f155245a29/vwu-qbw-ug3
Instead of restricting jobs to one zone we can query EC2 for instance type availability in each zone and use that to filter the zone randomization.
Once this merges we can remove all of the AWS
--zonesusage in build_jobs.py, perhaps leaving the scale tests' zones due to quotas.