Repository navigation
feat(mcp_readability): reconcile dropped findings on re-judged tools - #698
Open
akangsha7 wants to merge 1 commit into
Conversation
akangsha7
requested review from
IsmailMehdi,
helloeve and
prernakakkar-google
as code owners
November 2, 2026 23:02
akangsha7
force-pushed
the
evalbench/mcp-readability-html-provenance
branch
from
November 2, 2026 23:30
ab7e6bb to
fe24985
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-dropped-finding-reconcile
branch
2 times, most recently
from
November 3, 2026 04:05
9e59eab to
9a2c850
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-html-provenance
branch
from
November 3, 2026 04:05
fe24985 to
36b3f54
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-dropped-finding-reconcile
branch
from
November 3, 2026 04:38
9a2c850 to
d46f2b2
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-html-provenance
branch
2 times, most recently
from
November 3, 2026 04:47
6e7ef98 to
b4239e2
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-dropped-finding-reconcile
branch
from
November 3, 2026 04:47
d46f2b2 to
e6ee65e
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-dropped-finding-reconcile
branch
from
November 3, 2026 06:21
e6ee65e to
821a0f5
Compare
akangsha7
force-pushed
the
evalbench/mcp-readability-html-provenance
branch
from
November 3, 2026 06:21
b4239e2 to
947d4e0
Compare
Collaborator
Author
|
/gcbrun |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third part of wiring baseline carry-forward into
mcp_readability. Builds onevalbench/mcp-readability-html-provenance(#697).When a changed tool is re-judged and a baseline finding for that tool is not repeated in the new response, runs a lightweight second pass (
reconcile()) to check whether the schema change actually fixed the issue or the model simply omitted it.What changed
evalbench/scorers/mcp_readability/style_readability.py:_RECONCILE_PROMPT,_reconcile_dropped(), andreconcile().cf.unreported_findings(decision, judged)is non-empty, ask the judge for a booleanstill_appliesverdict per omittedfinding_id.still_appliesistrue(preserving original wording) and drop only findings ruledfalse.Testing
pytest evalbench/test/mcp_carry_forward_scorer_test.py evalbench/test/mcp_readability_test.py: 59 passed.pycodestyle . --config=.pycodestyle: clean.