Repository navigation
Conversation
Clients that drive the gRPC service only receive the aggregate summary
({"total", "scores": {metric: passing_count}}), so a CI job that sees a
scorer drop from 22/22 to 20/22 cannot tell which scenarios failed, or
why, without the viewer or the reporting store. The server already has
every per-scenario score row in hand when it builds that summary.
Add summary["scenario_scores"]: one entry per (id, comparator) row with
id, comparator and score, plus comparison_error / generated_error when
set and, only for rows without full marks, the judge's comparison_logs
truncated to 4000 characters. A clean run therefore adds a few hundred
bytes per scenario, while every failure keeps its reasoning. Whole-number
scores are emitted as integers even when pandas has upcast the column to
float. The field is additive, so existing clients are unaffected, and it
reaches all three paths (unary Eval with summary_in_response, interactive
and agent streams) through _process_results.
|
/gcbrun |
| # Per-scenario outcomes, so a client that only sees this response (e.g. a | ||
| # CI log) can tell which scenarios failed which scorer, and why, without | ||
| # the viewer or the reporting store. | ||
| summary["scenario_scores"] = analyzer.scenario_scores(scores_df) |
There was a problem hiding this comment.
This service is also used byNL@SQL cases where large run happens and it can break that.
There was a problem hiding this comment.
Good call — the client sets no gRPC message-size options, so a large run really could blow past the 4 MB default. Pushed e2d50c2: scenario_scores is now capped at 256 KB (fit_scenario_scores: keep only failures → drop their logs → truncate, with scenario_scores_truncated: true so clients can tell), scenario_scores_in_response: false turns it off, and the judge's logs are only attached to judged scorers that failed (measured metrics just carry their value). On a 22-scenario agent run that's ~14 KB; a synthetic 30k-entry NL2SQL run collapses to its failures at ~48 KB. The Summary: log lines now print <N entries> instead of the list.
| # Per-scenario outcomes, so a client that only sees this response (e.g. a | ||
| # CI log) can tell which scenarios failed which scorer, and why, without | ||
| # the viewer or the reporting store. | ||
| summary["scenario_scores"] = analyzer.scenario_scores(scores_df) |
There was a problem hiding this comment.
Wrap it in try/except and log a warning.
There was a problem hiding this comment.
Done — wrapped in try/except with a warning that names the job id; the summary is returned without the field if anything goes wrong.
Review follow-ups on the per-scenario scores added to the summary: - Cap the JSON size of scenario_scores at 256 KB (fit_scenario_scores), well under gRPC's 4 MB default receive limit and Cloud Logging's entry limit, so a large NL2SQL run cannot grow the response until the client rejects it. Over budget it keeps only the failures, then drops their logs, then truncates the list, and sets scenario_scores_truncated: true. - Attach the judge's comparison_logs only to judged scorers that failed; measured metrics (NON_BINARY_METRICS, hoisted from analyze_one_metric) carry just their value instead of a one-line restatement of it. - Wrap the whole thing in try/except and log a warning with the job id, so a problem building the list can never fail a run. - Add scenario_scores_in_response: false to leave the field out entirely. - Log "<N entries>" instead of the list in the Summary: log lines, and the job id plus byte count instead of the whole final payload. On a 22-scenario agent run the field is ~14 KB; a synthetic 30k-entry run collapses to its failures at ~48 KB.
What
Adds
summary["scenario_scores"]to the summary the service returns to clients: one entry per(id, comparator)row of the scores dataframe withid,comparatorandscore, pluscomparison_error/generated_errorwhen set and, only for rows that did not get full marks, the judge'scomparison_logs(truncated to 4000 characters).{"job_id": "…", "summary": { "total": 22, "scores": {"goal_completion": 20, "trajectory_matcher": 22, "…": "…"}, "scenario_scores": [ {"id": "fetch-pull-request-list", "comparator": "trajectory_matcher", "score": 100}, {"id": "fetch-pull-request-list", "comparator": "goal_completion", "score": 0, "comparison_logs": "FAIL Reasoning: … the tool call failed due to an external error (API rate limit) …"}, "…" ] }}Why
Clients that drive the gRPC service only get the aggregate summary back, so a CI job that watches
goal_completiondrop from 22/22 to 20/22 cannot tell which scenarios failed or why without opening the viewer or querying the reporting store. We hit exactly this: the viewer was down, the CI log only had totals, and the three failures (all the same upstream API rate limit, not a model regression) could only be found by querying BigQuery by hand. The server already has every per-scenario score row in memory when it builds the summary, so the data is free.Design notes
summary.total/summary.scoresare unaffected._process_results: unaryEval(whensummary_in_response: true), and the interactive and agent streams' final message.comparison_logsis attached only whenscore != 100(the pass value for binary scorers inanalyze_result) and capped at 4000 chars, so a clean run adds a few hundred bytes per scenario while every failure keeps its reasoning.float64).json.dumpsworks unchanged; rows without a scenarioidare skipped.Testing
evalbench/test/scenario_scores_test.py(7 cases: ordering, logs only on non-passing rows, error fields, truncation marker, skipped rows, numpy/NaN JSON safety).pytest evalbench/test/scenario_scores_test.py evalbench/test/analyzer_empty_fix_test.py evalbench/test/evalbench_test.py→ 35 passed.pycodestyle --config=.pycodestyleclean on the touched files.