ReleaseBench

ReleaseBench is a Claude Code plugin for developers who are about to take a private repository public. Its ten focused skills handle the risky steps of that move (auditing repository health, scanning for committed secrets, generating the standard public-repo files, polishing the README, and preparing releases), and each one stops at a result you can review, never a push or publish it performs on its own. Without checks like these, mistakes become public the moment the repository does: a committed API key, a missing license, or an untested install command ships to the world in one push.

Prerequisites: Python 3.10+ (standard library only) for the CLI, Node 18+ to run the bundled audit and secret scripts directly, and Claude Code to use ReleaseBench as a plugin.

Install:

• In Claude Code (one command): /plugin marketplace add https://gitlab.com/krahul02004/ReleaseBench then /plugin install releasebench. That reads the self-hosted marketplace manifest committed in this repository (.claude-plugin/marketplace.json); nothing is published to an external or central registry.
• Python CLI: pip install releasebench.

Check it works: hand the router one typed request and read the single-line receipt it prints (write the file with plain Set-Content so no byte-order mark is added):

'{"contract_version":"releasebench.route-request/v1","request_id":"getting-started","intents":["audit-repository"]}' |
  Set-Content request.json
releasebench request.json

{"authority_state":"candidate-inactive","closed_actions":["remote-project-creation-and-first-push","host-visibility-metadata-topics-avatar-writes","host-public-api-and-network-verification","local-tag-creation-and-outward-tag-push","host-release-creation-or-backfill","package-registry-name-availability-reads","package-publication","remote-image-reachability-checks"],"considered_leaves":["releasebench-audit-repository"],"decision":"selected","leaf_id":"releasebench-audit-repository","outward_actions_authorized":false,"outward_actions_executed":false,"reason_code":"ONE_DIRECT_INTENT","receipt_sha256":"c6bf6d7d150da84b8f5870e3aac9bba4504700e088dfdd37c0a609b34c4d5b35","receipt_version":"releasebench.route-receipt/v1","request_id":"getting-started","router_id":"releasebench-route","visibility_state":"private-prepublic"}

The "decision":"selected" and "leaf_id":"releasebench-audit-repository" fields confirm the router is installed and routing. The detailed demos below explain the receipt in full.

At the center of ReleaseBench is a small deterministic router. You hand it a typed JSON request saying what you want; it answers with exactly one skill to run, or refuses when the request is ambiguous. After installing (pip install releasebench):

'{"contract_version":"releasebench.route-request/v1","request_id":"demo-1","intents":["scan-secrets"]}' |
  Set-Content request.json
releasebench request.json

{"authority_state":"candidate-inactive","closed_actions":["remote-project-creation-and-first-push","host-visibility-metadata-topics-avatar-writes","host-public-api-and-network-verification","local-tag-creation-and-outward-tag-push","host-release-creation-or-backfill","package-registry-name-availability-reads","package-publication","remote-image-reachability-checks"],"considered_leaves":["releasebench-scan-secrets"],"decision":"selected","leaf_id":"releasebench-scan-secrets","outward_actions_authorized":false,"outward_actions_executed":false,"reason_code":"ONE_DIRECT_INTENT","receipt_sha256":"bbe6cc0ae3ae305bdb5c134521fae0b8d745f6777784b54aabcc3d8bdfbd93ef","receipt_version":"releasebench.route-receipt/v1","request_id":"demo-1","router_id":"releasebench-route","visibility_state":"private-prepublic"}

That one-line answer is a receipt: a JSON record of what was decided and why. leaf_id is the one skill it chose (the router calls skills "leaves"), reason_code says why, closed_actions lists everything it refused to even consider doing (every push, tag, publish, and settings change), and receipt_sha256 is a checksum of the record itself. The same request always produces byte-identical output, which is what the test suite verifies.

Ten skills. Each does one job and hands you something you can read before anything else happens.

1. You ask for something: "get this repo ready to go public", "check for leaked keys".
2. The router picks exactly one skill, or declines with a reason rather than guessing.
3. The chosen skill runs locally and stops at a reviewable result: a report, a diff, or prepared commands.
4. Anything that would leave your machine (pushing, tagging, creating a release, changing project settings, publishing a package) is written down for you but left for you to run.

flowchart LR
  A["Your request"] --> B["Router picks exactly one skill"]
  B -- "ambiguous" --> C["Declines with a reason"]
  B --> D["Skill runs locally"]
  D --> E["Reviewable report or diff"]
  E --> F["You run any push or publish yourself"]

Load the plugin from a checkout:

claude.cmd --plugin-dir .

Then let the router decide:

/releasebench:releasebench-route

Or call one skill directly, for example:

/releasebench:releasebench-scan-secrets

Each skill also bundles metadata for OpenAI Codex under agents/openai.yaml; automatic invocation is switched off there, so a skill runs only when you name it.

A receipt names a skill, and skills are procedures that Claude Code executes, so with the plugin loaded, the named skill is simply the next thing you run. Without Claude Code, ReleaseBench is two things: the releasebench router CLI shown above, and the bundled scripts under skills/*/scripts, which are plain dependency-free Node programs you can run directly against any repository path. Here is the repository-health audit pointed at a small demo app that has code and a README but nothing else:

node skills/releasebench-audit-repository/scripts/audit-repo.mjs R:\demo-app

[audit-repo] R:\demo-app
present: README, .gitignore

  [MUST] LICENSE: add a license (e.g. MIT); with none, the repo is "all rights reserved"
  [RECOMMENDED] CONTRIBUTING: add CONTRIBUTING: how to contribute + the project guardrails
  [RECOMMENDED] SECURITY: add SECURITY: how to report vulnerabilities privately
  [RECOMMENDED] CODE_OF_CONDUCT: add CODE_OF_CONDUCT: community standards
  [RECOMMENDED] CHANGELOG: add CHANGELOG: notable changes
  [RECOMMENDED] issue templates: add .github/ISSUE_TEMPLATE/
  [RECOMMENDED] PR template: add .github/PULL_REQUEST_TEMPLATE.md
  [RECOMMENDED] CI: add a .github/workflows CI (e.g. run tests on push/PR)
  [RECOMMENDED] package.json "license": add a "license" field
  [INFO] secrets scan: run releasebench-scan-secrets for committed-secret risks
  [INFO] config schema: if there is a JSON config, run releasebench-document-config for a schema + validator + Make-it-yours

summary: 1 must-have missing, 8 recommended missing, 2 present.
Run releasebench-scaffold-governance to generate the missing files.

It exits non-zero when a must-have is missing, so it can guard a CI job. The secret scanner (skills/releasebench-scan-secrets/scripts/scan-secrets.mjs) works the same way and masks every value it finds, printing at most the first three and last two characters.

The router is on PyPI, with no dependencies beyond the Python 3.10+ standard library:

pip install releasebench

Usage: releasebench [-h | --help] [REQUEST_FILE] reads one typed JSON routing request from REQUEST_FILE (or from stdin when the argument is omitted or -) and writes one single-line JSON receipt to stdout. Exit code 0 means a receipt was written (including "no safe route" answers), and exit code 2 means the request itself was malformed, with the reason on stderr. The file argument is the most reliable path on Windows. One caution for Windows PowerShell 5.1: write the request file with plain Set-Content as shown above: both > redirection and -Encoding UTF8 prepend a byte-order mark, which the router rejects with an INVALID_JSON ... utf-8-sig error.

Equivalently, run python -m releasebench. To install from a checkout instead: pip install ..

ReleaseBench is new: no download counts, no stars, no users to quote. What it has is proof you can re-run yourself in about a minute:

python -B tests/releasebench/run_tests.py
node --test tests/releasebench/test_node_tools.mjs

releasebench-atomic-tests tests=22 failures=0 errors=0 skipped=0
candidate_root=.
PASS

The Node runner reports 2 tests passed, 0 failed. Those 24 checks (22 Python contracts plus 2 Node tool regressions) pin down the exact set of skills, every routing decision (including every pairwise ambiguous request failing safely), byte-identical CLI output, and a privacy check that forbids machine-specific paths and secret-shaped strings throughout the product files and tests. GitLab CI runs the same suite on every push (badge above), and the PyPI package was re-installed into a fresh virtual environment to verify the install command while writing this README.

The repository audit recognizes both GitHub-native (.github/...) and GitLab-native (.gitlab/..., .gitlab-ci.yml) project surfaces. The Node regression suite exercises both layouts so a host-specific false warning cannot silently return.

Status in one line: source version 0.1.1 is tagged and released on GitLab, PyPI carries 0.1.1, and the plugin skills remain inactive candidates. The documented suite has 22 Python checks and 2 Node checks; the full lifecycle ledger is in STATUS.md.

ReleaseBench is available under the MIT License. Contributions follow CONTRIBUTING.md, the Code of Conduct, and private security reporting guidance. Planned work is recorded in ROADMAP.md.

• Rahul Krishna - creator and contributor.
• Claude Code - AI development assistance.
• Codex - AI development assistance.