Slim Basecamp skill and rely on live CLI help by robzolkos · Pull Request #835 · basecamp/basecamp-cli

robzolkos · 2026-10-04T05:17:04Z

Intent

Keep the Basecamp skill focused on durable operating and safety rules instead of duplicating the CLI's command manual.

The existing skill had grown to 89 KB / 1,565 lines. Loading it for a simple question such as “how many Basecamp accounts do I have?” added roughly 13,000 tokens of tool output before the command could run, even though the installed CLI already exposes structured, command-specific --agent --help.

What changed

reduce the main skill to 8.3 KB / 235 lines (about 91% smaller)
make live basecamp ... --agent --help the source of truth for command syntax, flags, scope, and notes
retain cross-cutting rules for credentials, URLs and replies, output modes, project/account scope, stdin content, attachments, retries, authentication, and mutations
teach the eval harness to serve structured help from the locally built CLI
add an eval for the original account-count question
enforce a 16 KiB skill budget and discovery-first contract in Go tests
document deterministic checks and model-eval comparison workflows

Why the eval harness changed

A discovery-first skill cannot be evaluated faithfully if help calls receive the generic empty mock. The harness now executes only structured --agent --help requests against the local binary; normal Basecamp operations remain deterministic mocks.

Verification

GOWORK=off bin/ci
eval patterns: 162 compiled across 30 cases
structured-help harness self-test
skill drift checks

The hosted model eval was not run locally because ANTHROPIC_API_KEY is not configured; the existing PR eval workflow can run it when the repository secret is available.

Summary by cubic

Slims the Basecamp skill from 89 KB / 1,565 lines to 8.3 KB / 235 lines by removing the embedded command manual and treating live basecamp ... --agent --help output as the source of truth for syntax, flags, and scope. Loading the old skill added roughly 13,000 tokens for simple questions; the new one keeps only durable operating and safety rules like credentials, output modes, and mutation retries. Discovery now goes leaf-first: accounts list --agent --help directly instead of walking down the command tree.

Eval harness

Serves structured --agent --help from the locally built CLI and enforces leaf-first discovery; other Basecamp operations remain deterministic mocks.
Adds eval cases for the account-count question and Docs & Files folder downloads, with tightened mock and matcher regexes that enforce correct command and flag sequences, plus a self-test for the structured help path, runnable via make check-eval-harness.
Go tests enforce a 16 KiB skill budget and the discovery-first contract.
Clarifies that the todolist --description flag accepts rich text HTML, with a test guarding the updated help text.

^{Written for commit 575a735. Summary will update on new commits.}

cubic-dev-ai

All reported issues were addressed across 7 files

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

Copilot

Copilot review overview

🟡 Changes recommended

The help-token check can execute real mutating commands against developer credentials, and retry guidance misclassifies unclassified failures.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Restrict eval execution to approved command words and flags · New
Clarify that retryable=false is not always a definitive verdict · New

What changed in this PR

Slims the Basecamp skill by replacing its command catalog with live structured CLI help, while retaining durable safety guidance.

Changes:

Introduces a discovery-first skill with a 16 KiB budget.
Serves live structured help in evals and adds an account-count case.
Documents and integrates deterministic harness checks.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File	Description
`skills/basecamp/SKILL.md`	Replaces the embedded manual with durable operating rules.
`skill-evals/run`	Executes structured help through the local CLI.
`skill-evals/README.md`	Documents checks and model-eval workflows.
`skill-evals/Makefile`	Passes the local CLI binary to evals.
`skill-evals/cases/accounts-count.yml`	Adds discovery-first account-count coverage.
`Makefile`	Adds harness checking and builds before evals.
`internal/commands/doc_contract_test.go`	Enforces size and discovery-first constraints.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

robzolkos · 2026-10-04T05:28:29Z

Fixed — addressed the structured-help coverage, JSON validation, failure diagnostics, JSON-mode discovery, and eval documentation findings.

robzolkos · 2026-10-04T05:28:29Z

Fixed — structured help now routes only allowlisted discovery requests through the non-mutating help command, and retry guidance now distinguishes an absent retry signal from a definitive verdict.

Copilot

Copilot review overview

🔵 Needs a closer look

The new eval rejects valid count commands, and its deterministic harness check is not wired into remote CI.

Review effort: Balanced
Findings: None

Resolved since last review (2)

Restrict eval execution to approved command words and flags
Clarify that retryable=false is not always a definitive verdict

Previously missed (2)

In code that hasn't changed since last review

Run deterministic eval harness self-test in integration workflow

Makefile:415

The remote integration workflow invokes its checks individually and currently runs make check-eval-patterns, but neither make check nor make check-eval-harness. Consequently, whenever the secret-gated model-eval job is skipped, CI never runs this new deterministic harness self-test. Add make check-eval-harness to the integration workflow so the advertised no-credential check is enforced for every PR.

Accept valid direct-count and filtered-list answers in evaluation

skill-evals/cases/accounts-count.yml:8

This expectation rejects valid answers that live root help can lead the model to: accounts list --count directly emits the list length, and accounts list --agent --jq 'length' correctly filters the data-only payload. Either invocation answers the task after introspection, but the eval would fail despite receiving and returning 2. Accept those forms as well as envelope-based .data | length.

cubic-dev-ai

1 issue found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="skill-evals/cases/accounts-count.yml">

<violation number="1" location="skill-evals/cases/accounts-count.yml:14">
P3: `max_commands: 2` leaves zero headroom for the skill's own fallback path from the new workflow. Step 3 of the rewritten SKILL.md tells agents that a leaf-miss (a plausible first guess here, e.g. `accounts count --agent --help`) is recovered by inspecting the parent group, which makes a fully skill-conformant run take 3 commands: wrong leaf help, parent help, then the count. With one sample per case, any first-guess miss or error-reading step flunks the case even though the behavior matches the changed skill exactly.</violation>
</file>

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

cubic-dev-ai · 2026-10-04T05:41:36Z

+  - '^accounts (?!list\b).*--help'
+accept_response:
+  - '\b2\b'
+max_commands: 2


P3: max_commands: 2 leaves zero headroom for the skill's own fallback path from the new workflow. Step 3 of the rewritten SKILL.md tells agents that a leaf-miss (a plausible first guess here, e.g. accounts count --agent --help) is recovered by inspecting the parent group, which makes a fully skill-conformant run take 3 commands: wrong leaf help, parent help, then the count. With one sample per case, any first-guess miss or error-reading step flunks the case even though the behavior matches the changed skill exactly.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At skill-evals/cases/accounts-count.yml, line 14: <comment>`max_commands: 2` leaves zero headroom for the skill's own fallback path from the new workflow. Step 3 of the rewritten SKILL.md tells agents that a leaf-miss (a plausible first guess here, e.g. `accounts count --agent --help`) is recovered by inspecting the parent group, which makes a fully skill-conformant run take 3 commands: wrong leaf help, parent help, then the count. With one sample per case, any first-guess miss or error-reading step flunks the case even though the behavior matches the changed skill exactly.</comment> <file context> @@ -4,8 +4,11 @@ mocks: accept_response: - '\b2\b' -max_commands: 4 +max_commands: 2 </file context>

Suggested change

max_commands: 2

max_commands: 3

Copilot

Copilot review overview

🟡 Changes recommended

The deterministic harness check is not wired into GitHub Actions, and the discovery contract test does not actually enforce leaf-first help.

Review effort: Balanced
Findings: 1

Open (1)

Test does not enforce the leaf-first help contract · New

Previously missed (1)

In code that hasn't changed since last review

CI skips the deterministic structured-help self-test

Makefile:415

This target is only added to the local aggregate check, but GitHub Actions does not invoke make check: .github/workflows/test.yml runs make check-eval-patterns directly, and the model-eval job is skipped when ANTHROPIC_API_KEY is unavailable. As a result, the new deterministic structured-help self-test can regress while required CI stays green. Add make check-eval-harness to the deterministic workflow job as well.

robzolkos · 2026-10-04T05:48:11Z

Flagged for human review — wiring a new required GitHub Actions step changes security-sensitive CI configuration, which this review-processing workflow does not modify.

robzolkos · 2026-10-04T05:48:11Z

Fixed — the account-count eval now accepts direct count output, envelope filtering, and data-only agent filtering.

robzolkos · 2026-10-04T05:48:11Z

Not doing this — the two-command ceiling is the deliberate efficiency contract for this obvious account-list case; increasing it would stop the eval from detecting the discovery overhead it was created to catch.

robzolkos · 2026-10-04T05:48:11Z

Fixed — the documentation contract now asserts the leaf-help template and its ordering before root fallback.

Copilot

Copilot review overview

🟢 Approved

The discovery workflow, safety boundary, tests, and documentation are consistent and complete.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Test does not enforce the leaf-first help contract

cubic-dev-ai

All reported issues were addressed across 2 files (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

cubic-dev-ai

All reported issues were addressed across 2 files (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

robzolkos · 2026-10-04T15:21:31Z

Fixed — the Docs & Files eval URL mock no longer intercepts help invocations.

robzolkos · 2026-10-04T15:21:31Z

Fixed — the account-count eval now validates the exact counting expression for envelope and data-only modes instead of accepting any jq filter containing length.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

robzolkos · 2026-10-04T15:59:02Z

Fixed — live todolists create and todolists update leaf help now identifies descriptions as rich-text HTML, with regression coverage.

Slim Basecamp skill around live CLI help

73a7da7

Copilot AI balanced review requested due to automatic review settings October 4, 2026 05:17

github-actions Bot added commands CLI command implementations tests Tests (unit and e2e) skills Agent skills labels Oct 4, 2026

Copilot started reviewing on behalf of robzolkos October 4, 2026 05:17 View session

cubic-dev-ai Bot reviewed Oct 4, 2026

View reviewed changes

Comment thread skill-evals/run Outdated

Comment thread skill-evals/run Outdated

Comment thread Makefile

Comment thread skill-evals/README.md Outdated

Comment thread skill-evals/run Outdated

Copilot AI reviewed Oct 4, 2026

View reviewed changes

Comment thread skill-evals/run

Comment thread skills/basecamp/SKILL.md Outdated

Harden structured help evals

8573602

Copilot AI balanced review requested due to automatic review settings October 4, 2026 05:27

Copilot started reviewing on behalf of robzolkos October 4, 2026 05:28 View session

Copilot AI reviewed Oct 4, 2026

View reviewed changes

Prefer leaf-first Basecamp command discovery

1d62896

Copilot AI balanced review requested due to automatic review settings October 4, 2026 05:38

Copilot started reviewing on behalf of robzolkos October 4, 2026 05:39 View session

cubic-dev-ai Bot reviewed Oct 4, 2026

View reviewed changes

Copilot AI reviewed Oct 4, 2026

View reviewed changes

Comment thread internal/commands/doc_contract_test.go Outdated

Broaden account count eval assertions

e75b2f2

Copilot AI balanced review requested due to automatic review settings October 4, 2026 05:47

Copilot started reviewing on behalf of robzolkos October 4, 2026 05:48 View session

Copilot AI previously approved these changes Oct 4, 2026

View reviewed changes

cubic-dev-ai Bot reviewed Oct 4, 2026

View reviewed changes

Comment thread skill-evals/cases/accounts-count.yml Outdated

Route Docs and Files requests to leaf help

1b84bd6

Copilot AI balanced review requested due to automatic review settings October 4, 2026 15:15

Copilot AI dismissed their stale review, a newer Copilot review was requested October 4, 2026 15:15

Copilot started reviewing on behalf of robzolkos October 4, 2026 15:15 View session

Copilot AI reviewed Oct 4, 2026

cubic-dev-ai Bot reviewed Oct 4, 2026

View reviewed changes

Comment thread skill-evals/cases/docs-file-download.yml Outdated

Tighten skill eval mocks and matchers

c41fc7d

Copilot AI balanced review requested due to automatic review settings October 4, 2026 15:21

Copilot started reviewing on behalf of robzolkos October 4, 2026 15:21 View session

Copilot AI reviewed Oct 4, 2026

Clarify todolist description format

575a735

Copilot AI balanced review requested due to automatic review settings October 4, 2026 15:55

Copilot started reviewing on behalf of robzolkos October 4, 2026 15:56 View session

Copilot AI reviewed Oct 4, 2026

robzolkos merged commit a65e26f into main Oct 4, 2026
30 of 31 checks passed

robzolkos deleted the skill-cleanup branch October 4, 2026 16:09

Conversation

robzolkos commented Oct 4, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Intent

What changed

Why the eval harness changed

Verification

Summary by cubic

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

robzolkos commented Oct 4, 2026

Uh oh!

robzolkos commented Oct 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

robzolkos commented Oct 4, 2026

Uh oh!

robzolkos commented Oct 4, 2026

Uh oh!

robzolkos commented Oct 4, 2026

Uh oh!

robzolkos commented Oct 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approved

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

robzolkos commented Oct 4, 2026

Uh oh!

robzolkos commented Oct 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

robzolkos commented Oct 4, 2026 •

edited by cubic-dev-ai Bot

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading