Sitelet https://github.com/GoogleCloudPlatform/evalbench/pull/698
Skip to content

feat(mcp_readability): reconcile dropped findings on re-judged tools - #698

Open
akangsha7 wants to merge 1 commit into
evalbench/mcp-readability-html-provenancefrom
evalbench/mcp-readability-dropped-finding-reconcile
Open

akangsha7 wants to merge 1 commit into
evalbench/mcp-readability-html-provenancefrom
evalbench/mcp-readability-dropped-finding-reconcile

Conversation

@akangsha7

@akangsha7 akangsha7 commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Third part of wiring baseline carry-forward into mcp_readability. Builds on evalbench/mcp-readability-html-provenance (#697).

When a changed tool is re-judged and a baseline finding for that tool is not repeated in the new response, runs a lightweight second pass (reconcile()) to check whether the schema change actually fixed the issue or the model simply omitted it.

What changed

  • evalbench/scorers/mcp_readability/style_readability.py:
    • Add _RECONCILE_PROMPT, _reconcile_dropped(), and reconcile().
    • When cf.unreported_findings(decision, judged) is non-empty, ask the judge for a boolean still_applies verdict per omitted finding_id.
    • Keep findings where still_applies is true (preserving original wording) and drop only findings ruled false.
    • Fail closed on API or JSON parse errors by keeping all findings under review.

Testing

  • pytest evalbench/test/mcp_carry_forward_scorer_test.py evalbench/test/mcp_readability_test.py: 59 passed.
  • pycodestyle . --config=.pycodestyle: clean.

@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-html-provenance branch from ab7e6bb to fe24985 Compare November 2, 2026 23:30
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-dropped-finding-reconcile branch 2 times, most recently from 9e59eab to 9a2c850 Compare November 3, 2026 04:05
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-html-provenance branch from fe24985 to 36b3f54 Compare November 3, 2026 04:05
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-dropped-finding-reconcile branch from 9a2c850 to d46f2b2 Compare November 3, 2026 04:38
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-html-provenance branch 2 times, most recently from 6e7ef98 to b4239e2 Compare November 3, 2026 04:47
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-dropped-finding-reconcile branch from d46f2b2 to e6ee65e Compare November 3, 2026 04:47
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-dropped-finding-reconcile branch from e6ee65e to 821a0f5 Compare November 3, 2026 06:21
@akangsha7
akangsha7 force-pushed the evalbench/mcp-readability-html-provenance branch from b4239e2 to 947d4e0 Compare November 3, 2026 06:21
@akangsha7 akangsha7 self-assigned this Nov 3, 2026
@akangsha7

Copy link
Copy Markdown
Collaborator Author

/gcbrun

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant