Repository navigation
feat(agent): run num_trials trials of each scenario - #700
Open
omkargaikwad23 wants to merge 5 commits into
Open
omkargaikwad23 wants to merge 5 commits into
omkargaikwad23 wants to merge 5 commits into
Conversation
The agent evaluator now reads num_trials and runs the full suite once per trial, one trial after another. With more than one trial, each row gets id <scenario>_trial_<k>, prompt_id, and trial_index, the same as NL2SQL. A scenario can set a lower num_trials. With one trial, the output does not change.
omkargaikwad23
requested review from
IsmailMehdi,
helloeve and
prernakakkar-google
as code owners
November 4, 2026 06:43
Collaborator
Author
|
/gcbrun |
The old sequencing test passed without a barrier between trials. The new test holds one trial 0 scenario and checks that no trial 1 scenario starts before trial 0 ends. Document the scenario-level num_trials key, and that trials share the sandbox home and work_dir. Say why trials run one after another.
Collaborator
|
LEts close doc design first |
| | `query_types` | Optional | Specifies the query_types (`dql`, `dml`, `dd`). This filters the dataset to the list of evals that are of the query_types provided. If not provided, all eval types (dql, dml and ddl) found in the dataset_config json file will be tried. | | ||
| | `dataset_format` | Conditional (if needed) | Defines the dataset format, with `evalbench-standard-format` as the default. For BIRD datasets, it must be set to `bird-standard-format`. For scenario-based agent runs use `agent-format` (orchestrator `agent`) or `gemini-cli-format` (orchestrator `geminicli`); other supported values are `bird-interact-format`, `cortado-format`, and `dea-format`.| | ||
| | `num_trials` | Optional | Number of trials to run for each prompt. | | ||
| | `num_trials` | Optional | Number of trials to run for each prompt (NL2SQL) or scenario (agent runs). | |
Collaborator
There was a problem hiding this comment.
Lets add examples to be more clear for both.
Collaborator
|
Currently this PR is not resetting before any trial. |
| | `databases` | Optional | Specifies the databases (e.g., `db_blog`, `california_schools`, etc.). This filters the dataset to the provided list of databases and ignores all other evals. If not provided, all databases found in the dataset_config json file will be tried. | | ||
| | `query_types` | Optional | Specifies the query_types (`dql`, `dml`, `dd`). This filters the dataset to the list of evals that are of the query_types provided. If not provided, all eval types (dql, dml and ddl) found in the dataset_config json file will be tried. | | ||
| | `dataset_format` | Conditional (if needed) | Defines the dataset format, with `evalbench-standard-format` as the default. For BIRD datasets, it must be set to `bird-standard-format`. For scenario-based agent runs use `agent-format` (orchestrator `agent`) or `gemini-cli-format` (orchestrator `geminicli`); other supported values are `bird-interact-format`, `cortado-format`, and `dea-format`.| | ||
| | `num_trials` | Optional | Number of trials to run for each prompt. | |
Collaborator
There was a problem hiding this comment.
Update existing run config as well
Collaborator
Author
I have created separate PR #709 on top of this as this PR was getting larger. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Agent evals can now run each scenario more than once. Set
num_trials: Nin the run config, and the whole suite runs N times in one job, one trial after another. A trial finishes before the next one starts.With N > 1, each eval row and score row says which scenario and which trial it belongs to:
id:capitals_trial_1prompt_id:capitalstrial_index:1This is the same format that NL2SQL already uses.
A scenario can set its own
num_trialsto run fewer times, for example1to run once. It cannot run more times than the run config. A higher value is capped at the run config value, with a warning.num_trialsmust be a whole number of 1 or more.With the default of 1, the output does not change.
Why
One run cannot tell a stable pass from a lucky one. This is the first step of multi-run stats. Later PRs use these rows to compute pass rate and flakiness for each scenario.
Testing
evalbench/test/agent_num_trials_test.py. The full test suite passes.num_trials: 2and two scenarios, one set to run once. It producedcapitals_trial_0,capitals_trial_1, andsum_once_trial_0, and the score rows had the same labels. Logs: https://paste.googleplex.com/6755399436651482 (Google-internal)Not covered
With N > 1, two things are not right yet. Later PRs fix them.
summary.csvis not correct for N > 1. Numeric metrics such astoken_consumptionandturn_countshow a count instead of an average, and theexecutabletotal is too high.work_dir, so files that one trial writes stay for the next trial. Until a reset is added, setnum_trials: 1on scenarios that write files or create named cloud resources. The dataset config doc has a note about this.