fix(ci): inert workflow test harness and dead composite actions give false CI assurance #3570

Problem

Workflow-and-composite "test" infrastructure in this repo passes CI, has never executed, and its contract tests assert that fact without noticing. The result is a green tick for a test path that does not exist.

Everything below was verified on origin/develop = 25179a560e44feefa9e611126657a4c269fd0fd3.

1. The workflow test harness cannot run — proven

.github/tests/workflow-harness.yml and .github/tests/error-isolation-test.yml are workflow files — both declare on: triggers (workflow-harness.yml:4-13, error-isolation-test.yml:5-6) — but they live in .github/tests/, not .github/workflows/.

GitHub only registers workflows under .github/workflows/. Proof from the registry:

GET /repos/lightspeedwp/.github/actions/workflows
  → workflows[] where path matches "tests/"   →  0 results

The harness is not merely absent from the registry — it has never been dispatched, cannot be dispatched with gh workflow run (which resolves by filename under .github/workflows/), and is not referenced by any script, package.json entry, or active workflow.

The only references to either file are:

• .github/specs/011-workflow-consolidation-phase-2/{plan,quickstart,tasks}.md — the design spec
• .github/docs/COMPOSITE_ACTIONS.md:312 — "Testing: All composite actions tested in workflow-harness.yml"
• .github/actions/__tests__/workflow-consolidation-actions.test.js:454-481 — Jest assertions about the files' YAML

The placement is a design error, not a build miss. Spec 011's own target tree puts them there:

.github/
├── tests/
│   ├── workflow-harness.yml              ← NEW
│   ├── error-isolation-test.yml          ← NEW

(plan.md:60-62, tasks T010/T011). Spec 011 is still marked Active in .github/specs/CATALOG.md:40 with 0 of 122 tasks checked, so the spec records these as unbuilt while the files sit on develop being linted by nothing.

2. The matrix targets a design that was superseded

The harness iterates five "unified" workflow types:

for workflow in labeling validation testing linting quality-gates   # workflow-harness.yml:36

error-isolation-test.yml:91 repeats the same five and builds .github/workflows/${TYPE}-unified.yml paths.

On develop, only labeling-unified.yml exists. The other four exist solely on the unmerged PR #3359 (refactor/workflow-consolidation-phase-2, open):

So the harness can never run even after #3359 merges — the directory is wrong regardless. And the five-type matrix encodes the 71→5 consolidation plan, which #3488 supersedes with a single reusable "PR checks" workflow. #3488 is open and its target architecture is the current direction.

3. The contract tests run, and they pass, asserting the fiction

.jest.config.cjs sets testMatch to include **/__tests__/**/*.test.js, so the suite is in the real test run. Verified locally against develop with the repo's own config and toolchain:

$ npx jest --config .jest.config.cjs --listTests | grep workflow-consolidation
.github/actions/__tests__/workflow-consolidation-actions.test.js      ← discovered

$ npx jest --config .jest.config.cjs .github/actions/__tests__/workflow-consolidation-actions.test.js
Test Suites: 1 passed, 1 total
Tests:       44 passed, 44 total

The two harness tests pass in isolation as well (-t "harness" → 1 passed, -t "error isolation" → 1 passed).

They pass because they only assert structure — job names, matrix contents, needs arrays, dispatch input options. None of them checks that the file lives in .github/workflows/, that its triggers can fire, or that the workflows it names exist. So a suite whose entire purpose is "the workflow harness works" is green, while the harness has never run and 4 of its 5 targets are absent from develop.

Note the only Jest currently gated in CI is a different file: labeling-unified.yml:66 runs scripts/agents/__tests__/label-contracts.test.js. The workflow/composite contract tests are not gated today; #3479 / #3487 would add that gate, which is why this needs fixing before the gate lands rather than after.

4. Two composite actions with no caller survived a closed issue

#3478 asked for exactly this:

• Delete the two unused composite actions, or wire them in if a planned workflow needs them.

#3478 was closed as COMPLETED by #3483, which fixed shellcheck debt and pinned the consumption examples. It did not touch the composite actions. Both are still on develop:

collect-metrics and apply-labels are genuinely used (labeling-unified.yml:107,113,142) and are out of scope.

COMPOSITE_ACTIONS.md:312 compounds it: it tells contributors that all composite actions are tested in workflow-harness.yml, which is both untrue and a dead end.

Change

Per file, delete or make real. Do not leave them inert.

• .github/tests/workflow-harness.yml — either move it under .github/workflows/ and retarget the matrix at workflows that exist, or delete it. If moved it must have a real trigger, per-PR concurrency and a timeout, and its floating pins fixed: actions/checkout@v4 appears 11 times across the two files (workflow-harness.yml:26,66,112,136,149,162,175; error-isolation-test.yml:27,40,76,115) with no SHA, against a repo that is otherwise fully SHA-pinned.
• .github/tests/error-isolation-test.yml — same decision. It is workflow_dispatch-only and asserts an isolation property across five workflows; with epic: central, tested workflow layer for all org repos (GitHub Free) #3488's single shared workflow that property has no meaning, so deletion is likely correct.
• .github/actions/aggregate-tests/ and .github/actions/validate-check/ — delete, or wire into a workflow that needs them.
• .github/actions/__tests__/workflow-consolidation-actions.test.js:454-481 — replace the structural assertions with reachability assertions: any file under .github/tests/ that declares on: must live under .github/workflows/; any workflow filename referenced by a test or doc must exist. Keep the composite-action contract tests — those are the useful half.
• .github/docs/COMPOSITE_ACTIONS.md — fix :312, and :116,173,289,300 if the composites are deleted.
• .github/specs/011-workflow-consolidation-phase-2/ — T010/T011 stay unticked, or the spec records what actually happened. A spec marked Active with 0/122 tasks while its outputs sit on develop is how this survived.

Acceptance

• grep -rl '^on:' .github/tests/ returns nothing, or every hit is under .github/workflows/
• GET /actions/workflows lists any harness workflow that was added, proving registration
• grep -rn "actions/aggregate-tests\|actions/validate-check" .github/workflows/ .github/examples/ returns 0 hits, and both directories are deleted or have a documented caller
• A test asserts every workflow filename referenced under .github/** resolves to a real file, so a missing workflow fails CI
• A test asserts no workflow-shaped YAML lives outside .github/workflows/, so this cannot regress
• npm run test:js is green, and npx jest --config .jest.config.cjs .github/actions/__tests__/workflow-consolidation-actions.test.js is green for a reason
• No floating actions/*@vN in any file added or moved by this issue
• .github/docs/COMPOSITE_ACTIONS.md describes how composite actions are actually tested
• Spec 011 T010/T011 state matches reality

Not in scope

• Reusable-workflow architecture, per-repo callers, rulesets, Dependabot, template centralisation — all epic: central, tested workflow layer for all org repos (GitHub Free) #3488.
• Standing up a Jest CI gate — fix(ci): no workflow runs the Jest suite, so test regressions merge green #3479, in flight in fix: ci - run Jest on every PR and fail only on new failures #3487.
• Jest discovery conventions (tests/ vs __tests__/, dead root jest.config.js) — task: support tests/ and __tests__/ and remove dead root jest.config.js #3469.
• .test.cjs files that match no runner — ci: 20 .test.cjs suites run in no known runner #3560.
• refactor: Phase 2 Workflow Consolidation - 71→5 unified workflows implementation #3359's five unified workflows. This issue does not decide whether they land.
• collect-metrics / apply-labels behaviour, label-authority conflicts (fix: labeling - workflows fight over PR labels (broken labeler rules, non-canonical type:documentation, stale payloads) #3545 / fix: reconcile label ownership so routing, labeler and agent converge (#3545) #3549), or the wider workflow backlog.
• jest.setup.localstorage.js legitimately lives in .github/tests/ (referenced by .jest.config.cjs) and must stay.

Evidence

• origin/develop = 25179a560e44feefa9e611126657a4c269fd0fd3, default branch develop
• GET /repos/lightspeedwp/.github/actions/workflows → 0 workflows under tests/
• .github/tests/workflow-harness.yml:4-13,36, .github/tests/error-isolation-test.yml:5-6,91
• .github/actions/__tests__/workflow-consolidation-actions.test.js:454-481 — 44/44 pass locally
• .github/specs/011-workflow-consolidation-phase-2/plan.md:60-62, tasks.md T010/T011, CATALOG.md:40
• .github/docs/COMPOSITE_ACTIONS.md:116,173,289,300,312
• git grep "actions/aggregate-tests\|actions/validate-check" develop -- .github/ → docs only
• chore(ci): remove unused composite actions, clear shellcheck debt and pin example workflows #3478 closed as COMPLETED by chore: ci - clear workflow shellcheck debt and fix consumption examples #3483, whose changed files do not include either composite
• git ls-tree origin/develop .github/workflows/refactor… — the four unified workflows exist only on refactor/workflow-consolidation-phase-2 (PR refactor: Phase 2 Workflow Consolidation - 71→5 unified workflows implementation #3359, open)