scaletest: Move Prometheus to addon nodes by serathius · Pull Request #18048 · kubernetes/kops

serathius · 2026-03-09T15:30:17Z

/cc @mborsz @upodroid @hakman

With #18036 we can move prometheus to addons node.

hakman · 2026-03-09T16:08:55Z

/approve

serathius · 2026-03-09T16:30:56Z

/retest

hakman · 2026-03-09T18:38:36Z

/test pull-kops-gce-master-scale-performance-100

serathius · 2026-03-09T20:07:16Z

We have 3 failures on this PR after 3 successes on adding nodes, were we really unlucky?

My guess is that previously the 120s might not be enough to for the nodes to start. It didn't matter before, but it matters now.

serathius · 2026-03-09T20:18:06Z

Prepared fix for instanceDetails being nil, #18050

@hakman any guesses why instanceDetails is nil?

hakman · 2026-03-09T21:50:12Z

Prepared fix for instanceDetails being nil, #18050

@hakman any guesses why instanceDetails is nil?

That should be fixed now by #18050.

serathius · 2026-03-09T22:49:53Z

/test pull-kops-gce-master-scale-performance-100

serathius · 2026-03-09T23:21:16Z

Heh, looks like kops never create addons node. That's why the test passed :(

serathius · 2026-03-10T00:20:23Z

Ok, found the error Mar 10, 2026, 1:19:34 AM UTC+01:00 Instance 'addons-4xfc' creation failed: pd-standard disk type cannot be used by c3-standard-22 machine type.

serathius · 2026-03-10T00:48:15Z

Prometheus logs

ts=2026-03-10T00:47:25.616Z caller=scrape.go:1343 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-proxy/0 target=http://10.128.0.103:10249/metrics msg="Scrape failed" err="Get \"http://10.128.0.103:10249/metrics\": context deadline exceeded"
ts=2026-03-10T00:47:27.042Z caller=scrape.go:1343 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/master/1 target=https://10.128.0.103:10259/metrics msg="Scrape failed" err="Get \"https://10.128.0.103:10259/metrics\": context deadline exceeded"
ts=2026-03-10T00:47:28.299Z caller=scrape.go:1343 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/master/2 target=https://10.128.0.103:10257/metrics msg="Scrape failed" err="Get \"https://10.128.0.103:10257/metrics\": context deadline exceeded"
ts=2026-03-10T00:47:32.043Z caller=scrape.go:1343 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/master/1 target=https://10.128.0.103:10259/metrics msg="Scrape failed" err="Get \"https://10.128.0.103:10259/metrics\": context deadline exceeded"
ts=2026-03-10T00:47:33.299Z caller=scrape.go:1343 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/master/2 target=https://10.128.0.103:10257/metrics msg="Scrape failed" err="Get \"https://10.128.0.103:10257/metrics\": context deadline exceeded"
ts=2026-03-10T00:47:37.044Z caller=scrape.go:1343 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/master/1 target=https://10.128.0.103:10259/metrics msg="Scrape failed" err="Get \"https://10.128.0.103:10259/metrics\": context deadline exceeded"
ts=2026-03-10T00:47:38.300Z caller=scrape.go:1343 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/master/2 target=https://10.128.0.103:10257/metrics msg="Scrape failed" err="Get \"https://10.128.0.103:10257/metrics\": context deadline exceeded"
ts=2026-03-10T00:47:42.045Z caller=scrape.go:1343 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/master/1 target=https://10.128.0.103:10259/metrics msg="Scrape failed" err="Get \"https://10.128.0.103:10259/metrics\": context deadline exceeded"
ts=2026-03-10T00:47:43.301Z caller=scrape.go:1343 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/master/2 target=https://10.128.0.103:10257/metrics msg="Scrape failed" err="Get \"https://10.128.0.103:10257/metrics\": context deadline exceeded"

serathius · 2026-03-10T00:52:15Z

Still getting

W0310 00:43:46.508389   57468 util.go:72] error while calling prometheus api: the server is currently unable to handle the request (get services http:prometheus-k8s:9090), response: k8s�

�v1��Status�]
�
�������Failure�3no endpoints available for service "prometheus-k8s""�ServiceUnavailable0����"�

Don't know why

$ sudo kubectl --kubeconfig /var/lib/kops/kubeconfig get endpoints prometheus-k8s -n monitoring -o yaml
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2026-03-10T00:44:08Z"
  creationTimestamp: "2026-03-10T00:43:46Z"
  labels:
    endpoints.kubernetes.io/managed-by: endpoint-controller
    prometheus: k8s
  name: prometheus-k8s
  namespace: monitoring
  resourceVersion: "8648"
  uid: 8bd1435a-5085-415a-b728-d2fe81d06790
subsets:
- addresses:
  - ip: 10.64.101.4
    nodeName: addons-6jlz
    targetRef:
      kind: Pod
      name: prometheus-k8s-0
      namespace: monitoring
      uid: 7e67f1b4-f75f-4a82-9aff-8f92a1f77703
  ports:
  - name: web
    port: 9090
    protocol: TCP

serathius · 2026-03-10T00:53:33Z

Direct proxy API also works

$ sudo kubectl --kubeconfig /var/lib/kops/kubeconfig get --raw "/api/v1/namespaces/monitoring/services/http:prometheus-k8s:9090/proxy/"
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="shortcut icon" href="/sitelet?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fkops%2Fpull%2Ffavicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,shrink-to-fit=no"/><meta name="theme-color" content="#000000"/><script>const GLOBAL_CONSOLES_LINK="",GLOBAL_AGENT_MODE="false"</script><link rel="manifest" href="/sitelet?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fkops%2Fpull%2Fmanifest.json" crossorigin="use-credentials"/><title>Prometheus Time Series Collection and Processing Server</title><script defer="defer" src="/sitelet?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fkops%2Fpull%2Fstatic%2Fjs%2Fmain.8644e60a.js"></script><link href="/sitelet?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fkops%2Fpull%2Fstatic%2Fcss%2Fmain.e64b9629.css" rel="stylesheet"></head><body class="bootstrap"><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html

serathius · 2026-03-10T12:20:50Z

/retest

serathius · 2026-03-10T13:01:34Z

For some reason it picked old commit 9d4a1281e038e65e85caffa20e76c2d67870edfc instead of 9ab1dc73e51a5cdbfada3d2437f4d4972e59aee9 that includes kubernetes/perf-tests#3885

serathius · 2026-03-10T13:03:40Z

/test pull-kops-gce-master-scale-performance-100

serathius · 2026-03-10T13:39:27Z

It didn't work. Still we don't know why CL2 cannot connect to Prometheus and there are no logs. Close to giving up.

serathius · 2026-03-12T13:02:46Z

/test pull-kops-gce-master-scale-performance-100

hakman · 2026-03-12T14:14:24Z

@serathius @upodroid anything else left to do here?

upodroid · 2026-03-12T15:25:46Z

Yeah. Let's test the aws one

/test pull-kops-gce-master-scale-performance-100

serathius · 2026-03-12T15:36:06Z

The last 2 runs were successful so if the third one started by @upodroid passes I think we can merge.

Thanks @upodroid for fixing network!

upodroid · 2026-03-12T15:43:48Z

Oops wrong one

/test pull-kops-ec2-master-scale-performance-100

serathius · 2026-03-12T16:45:26Z

Can we skip AWS? release blocking tests are on GCE, the AWS is a followup.

AWS is missing firewall rules to allow Prometheus to scrape control plane.

hakman · 2026-03-12T17:09:29Z

Can we skip AWS? release blocking tests are on GCE, the AWS is a followup.

Yes

serathius · 2026-03-12T17:13:51Z

 # this is used as a label to select kube-proxy pods on kops for kube-proxy service
 # used by CL2 Prometheus here https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/pkg/prometheus/manifests/default/kube-proxy-service.yaml#L2
 export PROMETHEUS_KUBE_PROXY_SELECTOR_KEY="k8s-app"
-export PROMETHEUS_SCRAPE_APISERVER_ONLY="true"


FYI PROMETHEUS_SCRAPE_APISERVER_ONLY doesn't exist on CL2 side.

upodroid · 2026-03-12T17:13:52Z

You can't because the aws jobs are release-informing

https://testgrid.k8s.io/sig-release-master-informing#ec2-master-scale-performance

If we merge it, we'll need to fix it within a day or two.

serathius · 2026-03-12T17:35:16Z

By skipping AWS I didn't mean break their testing. Just skip fixing addon nodes.

serathius · 2026-03-12T18:25:05Z

/retest

serathius · 2026-03-12T19:55:17Z

/test pull-kops-ec2-master-scale-performance-100
/test pull-kops-gce-master-scale-performance-100

k8s-ci-robot · 2026-03-13T09:32:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman, upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~tests/e2e/scenarios/scalability/OWNERS~~ [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot requested review from hakman, mborsz and upodroid March 9, 2026 15:30

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 9, 2026

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 9, 2026

serathius force-pushed the addons-node-prometheus branch from 810625b to 37ade34 Compare March 9, 2026 22:04

serathius force-pushed the addons-node-prometheus branch from 37ade34 to 581810b Compare March 9, 2026 23:20

serathius force-pushed the addons-node-prometheus branch 2 times, most recently from 22f590a to cfc2596 Compare March 10, 2026 00:01

serathius force-pushed the addons-node-prometheus branch from cfc2596 to b8b29d5 Compare March 10, 2026 00:21

serathius force-pushed the addons-node-prometheus branch 4 times, most recently from 2dbf6c2 to 4a3596a Compare March 10, 2026 07:21

This was referenced Mar 10, 2026

Fix adding addon node #18051

Closed

Set timeout to checking prometheus targets kubernetes/perf-tests#3885

Merged

Fix adding addon node

cf564bf

serathius force-pushed the addons-node-prometheus branch from 4a3596a to 05535d9 Compare March 12, 2026 08:05

Move Prometheus to addon nodes in GCE

f7c4cf6

AWS is missing firewall rules to allow Prometheus to scrape control plane.

serathius force-pushed the addons-node-prometheus branch from 05535d9 to f7c4cf6 Compare March 12, 2026 16:52

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 12, 2026

serathius commented Mar 12, 2026

View reviewed changes

upodroid approved these changes Mar 13, 2026

View reviewed changes

k8s-ci-robot assigned upodroid Mar 13, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 13, 2026

k8s-ci-robot merged commit 3911794 into kubernetes:master Mar 13, 2026
29 checks passed

k8s-ci-robot added this to the v1.36 milestone Mar 13, 2026

hakman changed the title ~~Move Prometheus to addon nodes~~ scaletest: Move Prometheus to addon nodes Mar 18, 2026

Conversation

serathius commented Mar 9, 2026

Uh oh!

hakman commented Mar 9, 2026

Uh oh!

serathius commented Mar 9, 2026

Uh oh!

hakman commented Mar 9, 2026

Uh oh!

serathius commented Mar 9, 2026

Uh oh!

serathius commented Mar 9, 2026

Uh oh!

hakman commented Mar 9, 2026

Uh oh!

serathius commented Mar 9, 2026

Uh oh!

serathius commented Mar 9, 2026

Uh oh!

serathius commented Mar 10, 2026

Uh oh!

serathius commented Mar 10, 2026

Uh oh!

serathius commented Mar 10, 2026

Uh oh!

serathius commented Mar 10, 2026

Uh oh!

serathius commented Mar 10, 2026

Uh oh!

serathius commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serathius commented Mar 10, 2026

Uh oh!

serathius commented Mar 10, 2026

Uh oh!

serathius commented Mar 12, 2026

Uh oh!

hakman commented Mar 12, 2026

Uh oh!

upodroid commented Mar 12, 2026

Uh oh!

serathius commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

upodroid commented Mar 12, 2026

Uh oh!

serathius commented Mar 12, 2026

Uh oh!

hakman commented Mar 12, 2026

Uh oh!

serathius Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

upodroid commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serathius commented Mar 12, 2026

Uh oh!

serathius commented Mar 12, 2026

Uh oh!

serathius commented Mar 12, 2026

Uh oh!

k8s-ci-robot commented Mar 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

serathius commented Mar 10, 2026 •

edited

Loading

serathius commented Mar 12, 2026 •

edited

Loading

upodroid commented Mar 12, 2026 •

edited

Loading