Sitelet https://github.com/Netis/heron/pull/203
Skip to content

feat(verification): risk-driven coverage policy, checkers, and CI gates - #203

Open
vaderyang wants to merge 5 commits into
mainfrom
feat/verification-coverage
Open

vaderyang wants to merge 5 commits into
mainfrom
feat/verification-coverage

Conversation

@vaderyang

Copy link
Copy Markdown
Collaborator

Summary

Replaces the single "line coverage ≥ 80%" target with a risk-driven, six-dimension verification policy — and the gates to enforce it.

Policy (docs/design/11-verification.md, verification/)
Code / change / behavior / risk / effectiveness / system coverage. Tiered coverage-policy.json, expiring waivers.json, a scenarios.json behavior matrix, and a measured coverage-baseline.json ratchet.

Checkers (scripts/ci/, stdlib-only so the runner needs no PyYAML)

  • check_coverage_policy.py — tier classification, waiver expiry/owner, floors, no-decrease, --enforce-tiers, --write-baseline
  • check_scenarios.py — @scenario <ID> <kind> tags (unit/integration/e2e/fault)
  • diff_coverage.py — changed-code coverage from lcov + JSON × git diff
  • coverage.sh, mutation.sh, self-tests under scripts/ci/tests/

CI
Cheap static gates in ci.yml; the heavy instrumented build + changed-code gate + advisory mutation run in a separate coverage.yml (so instrumentation never holds the required ci check hostage); a weekly mutation.yml runs the Tier 0 full pass.

Commits

  1. test: raise coverage on storage backends, CLI, and API smoke paths
  2. test: fix silently-skipping pipeline_e2e and turn integration suites
  3. test(verification): tag critical scenarios with @scenario
  4. feat(verification): policy, checkers, and CI gates
  5. ci(mutation): calibrate cargo-mutants --in-diff, arm the advisory PR check

Notable fixes

  • pipeline_e2e and the turn integration suite referenced legacy gitignored fixtures that do not exist, so they skipped silently — and pipeline_e2e hung once fixtures were present because the pair sweeper is an intentional forever-task. Fixtures are now resolved corpus-first with a git-LFS-aware check, the sweeper is aborted after the finite stages drain, and the four genuinely-unreproducible tests are #[ignore]d instead of silently skipping.
  • cargo-mutants --in-diff matches paths against the server/ workspace root, so repo-relative server/... paths silently select zero mutants. The patch is now generated workspace-relative, and mutation runs single-threaded --in-place (required because h-common reads the repo-root VERSION via include_str!).

Test plan

  • cargo test --workspace
  • cargo check --workspace --all-targets
  • python3 scripts/ci/check_scenarios.py → 18 defined, 20 tag(s), OK
  • python3 scripts/ci/check_coverage_policy.py --static
  • checker self-tests (15 / 9 / 11 passing)
  • Reviewer: mark the coverage workflow's rust / console jobs as required checks once merged
  • Reviewer: confirm --enforce-tiers stays on (currently waived for the crates still ratcheting)

Follow-ups

  • Promote the advisory mutation PR check to blocking once its false-positive rate is measured on real diffs.
  • Auto-refresh coverage-baseline.json on merge to main (manual --write-baseline for now).

Vader Yang added 5 commits October 4, 2026 15:17
Add scripted HTTP mocks (test_mock.rs/search_mock.rs) and query/retention tests for the ClickHouse and Aglake backends, unit tests for the storage dialect/convert helpers, CLI smoke coverage for heron, and a whole-router smoke test for the h-api read paths. Lifts the crates that had no live-server coverage.
The fixtures these suites referenced were legacy gitignored captures that do not exist, so they skipped silently (and pipeline_e2e hung once fixtures were present because pair_sweeper is an infinite task). Resolve corpus-first with an LFS-aware check, drain all finite stages before aborting the sweeper, repoint fixtures to the committed corpus, retune assertions to single-turn ground truth, and #[ignore] the four tests that still need uncommitted captures.
Annotate the tests that verify Tier 0/1 scenarios so check_scenarios.py can match each declaration to an implementation.
Add a six-dimension verification policy (code/change/behavior/risk/effectiveness/system) with stdlib-only checkers: coverage-policy.json tiers, expiring waivers, scenarios.json, and a measured baseline ratchet. Wire the cheap static gates into ci.yml and the heavy measurement + changed-code gate into a separate coverage.yml workflow. Add a PR template encoding the scenario convention.
…heck

cargo-mutants matches --in-diff paths against the server/ workspace root, so repo-relative paths silently select zero mutants; generate a workspace-relative patch instead. Run mutation on the PR's changed Tier 0 lines (advisory, continue-on-error) plus a weekly full run. Record the storage equivalent mutants and the single-threaded --in-place requirement.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant