fix(agent): mark agent infrastructure and simulated-user errors as unscored by omkargaikwad23 · Pull Request #702 · GoogleCloudPlatform/evalbench

omkargaikwad23 · 2026-11-04T11:33:38Z

What this does

Some agent runs fail for reasons outside the tool we test. This PR marks those runs as errors instead of failures.

Two kinds of errors are covered:

The agent's model API returns a quota or server error, such as 429 or 503. Each CLI checks its own error output:
- Gemini CLI: the final [API Error: ...] message.
- Claude Code: a rate_limit or server_error message from the main agent.
- Codex CLI: the turn.failed message.
The simulated user fails or returns an empty reply. The simulated user model returns an empty reply when it runs out of quota.

In both cases, the conversation stops and the error is saved in the generated_error column. A timeout still counts as a failure.

Why

If these errors count as failures, a stable scenario looks flaky. Later, multi-run stats will leave error runs out of the pass rate.
When the simulated user crashed, the remaining scenarios in the dataset were skipped. Now they still run.

Other changes

Averages such as latency and turn count skip error runs. Before, these runs counted as 0 and lowered the average.
The noop simulated user always replies with empty text, so its empty reply is not an error.

Testing

46 new unit tests. The full test suite passes.
4 local runs with Gemini CLI: a simulated user that raises an error, a simulated user that returns None, an agent 404 error (counts as a failure), and an agent 429 error from a fake Vertex server (counts as an error). Logs: https://paste.googleplex.com/6755399436658858 (Google-internal)

Not covered

agy_cli and agent_grpc_proxy errors still count as failures.
Claude Code and Codex CLI quota errors are covered only by unit tests.
The Gemini CLI and Codex CLI checks read the error message text. If a CLI release changes the wording, the error counts as a failure again.
The judge still scores error runs, so summary.csv still counts them. A later PR fixes this.

…scored Agent CLI quota and server errors, and simulated-user errors, now set generated_error instead of being scored as failures. A simulated-user error no longer skips the remaining scenarios of an evalset. The analyzer leaves rows with generated_error out of non-binary averages, because scorers return 0 for them.

The noop simulated user of the agent gRPC proxy flow returns an empty reply by design, so treating it as an error stopped multi-turn proxy scenarios after turn 1. Only a simulated-user exception now sets generated_error.

omkargaikwad23 · 2026-11-06T05:44:07Z

/gcbrun

prernakakkar-google · 2026-11-06T09:44:22Z

+# cause these, so the trial is unscored instead of failed. Status codes must
+# follow a word such as "HTTP" or "error", so a traceback line number such as
+# "line 503" does not match.
+_INFRA_ERROR_RE = re.compile(


The regex searches all of stderr, so real failures get marked as infra errors: gemini-cli's "Attempt 1 failed with status 429" retry log, errors from the tool being tested (UNAVAILABLE, 500), and the gRPC proxy's error_message. Match only the model API's error formats, ideally with one check per agent CLI.

Fixed. I removed the stderr regex. Each CLI now checks only its own final model API error, so retry logs, tool errors, and gRPC proxy errors no longer match. Other generators, including the gRPC proxy, count every error as a failure.

prernakakkar-google · 2026-11-06T09:45:08Z

+# "line 503" does not match.
+_INFRA_ERROR_RE = re.compile(
+    r"\b(?:HTTP|[Ss]tatus|[Cc]ode|[Ee]rror)\b\D{0,10}\b(?:429|50[0234])\b"
+    r"|Too Many Requests|RESOURCE_EXHAUSTED|UNAVAILABLE"


Real infra errors are missed: 529, ERROR 429, status_code=503, HTTP/1.1 and HTTP/2 status lines, OpenAI's wording, and Claude/Codex errors that appear on stdout. Use re.IGNORECASE.

Fixed. Claude Code now uses the API error category (rate_limit, server_error), which covers 529. Codex now reads the final turn.failed message, including OpenAI wording such as "at capacity" and "high demand". Both read stdout, and the match is case-insensitive.

prernakakkar-google · 2026-11-06T09:59:37Z

-                        conversation_history,
-                        result.stdout
-                    )
+                    try:


On a simulated-user quota error, QueryGenerator.generate returns "", so this try never fires. Raise on an empty reply unless the generator is NOOPGenerator.

Fixed. An empty or None reply is now saved as a simulated user error. The noop simulated users still continue the conversation.

…derr The stderr regex marked real failures as infrastructure errors: Gemini CLI retry logs, errors from the tool under test, and gRPC proxy error messages. It also missed errors that Claude Code and Codex CLI write to stdout. AgentCliGenerator.model_api_error() now returns the final model API error if it is a quota or server error. Gemini CLI reads the "[API Error: ...]" result event, Claude Code reads the rate_limit and server_error categories of top-level assistant messages, and Codex CLI reads the turn.failed message. The default returns None, so agy_cli and agent_grpc_proxy errors still count as failures. An empty or None simulated-user reply is now an error, because QueryGenerator.generate returns "" after a quota error and ClaudeGenerator returns None. The noop simulated users still continue the conversation.

omkargaikwad23 · 2026-11-11T10:13:22Z

/gcbrun

prernakakkar-google · 2026-11-11T11:48:04Z

+    r"|too many requests|stream disconnected before completion"
+    r"|at capacity|high demand",
+    re.IGNORECASE)
+


Codex regex misclassifies a content-filter stop as an infra error

Fixed. When the model stops a response by itself, for example because of a content filter, it now counts as a failure. A dropped connection still counts as an infra error.

prernakakkar-google · 2026-11-11T11:50:47Z

+                        # A subagent error is skipped, because the main
+                        # agent receives it as a tool result and continues.
+                        final_obj["api_error"] = {
+                            "type": event["error"],


api_error is set on the first errored main-agent message and never cleared, so a run that recovers afterwards is still marked as an infra error

Fixed. Only the last message from the main agent sets api_error now. If the agent recovers after a 429, the run is not marked as an infra error.

prernakakkar-google · 2026-11-11T11:51:50Z

+    r"\b(?:429|500|502|503|504)\b|RESOURCE_EXHAUSTED|UNAVAILABLE|\bINTERNAL\b"
+    r"|DEADLINE_EXCEEDED|quota exceeded|exceeded[^\n]{0,40}quota|exhausted"
+    r"|rate limit|too many requests|overloaded",
+    re.IGNORECASE)


regex needs update. limit of 500 items will aslo pass

Fixed. A status code now counts only when it follows its label, such as "code":429 or status: 503, so "limit of 500 items" no longer matches. The Codex regex had the same problem and has the same fix.

- Gemini CLI and Codex CLI: a status code counts only after its label, such as '"code":429' or "status 503". A bare number such as "limit of 500 items" no longer matches. - Gemini CLI: status names such as UNAVAILABLE count only in upper case. - Codex CLI: "Incomplete response returned" (content filter, output limit) is a failure, not a server error. - Claude Code: only the last main-agent message sets the API error, so a run that recovers is not an infra error.

… CLI A bare "exhausted" also matched errors that are not quota errors, such as an exhausted context window. The regex now lists the quota messages that Gemini CLI and the API use. Also shorten the comments on the three error checks.

Gemini CLI can report a quota error with only a reason code, such as "RATE_LIMIT_EXCEEDED", which the message regex does not match. The result event names the error class in `error.type`, so TerminalQuotaError and RetryableQuotaError now count as infra errors. The regex still applies to all other errors.

omkargaikwad23 requested review from IsmailMehdi, helloeve and prernakakkar-google as code owners November 4, 2026 11:33

omkargaikwad23 and others added 2 commits November 4, 2026 13:38

Merge branch 'main' into fix/agent-infra-errors-unscored

30f93b1

prernakakkar-google reviewed Nov 6, 2026

View reviewed changes

omkargaikwad23 requested a review from prernakakkar-google November 6, 2026 16:07

omkargaikwad23 added 2 commits November 6, 2026 21:38

Merge branch 'main' into fix/agent-infra-errors-unscored

8c3f98c

Merge branch 'main' into fix/agent-infra-errors-unscored

8d0c3b4

prernakakkar-google reviewed Nov 11, 2026

View reviewed changes

omkargaikwad23 requested a review from prernakakkar-google November 11, 2026 13:43

omkargaikwad23 added 2 commits November 11, 2026 13:55

Conversation

omkargaikwad23 commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

Why

Other changes

Testing

Not covered

Uh oh!

omkargaikwad23 commented Nov 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

omkargaikwad23 commented Nov 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

omkargaikwad23 commented Nov 4, 2026 •

edited

Loading