Split @benchsdk/client (REST) and @benchsdk/runner (framework); rework benchmark authoring to config + task - #253

Summary

Splits the two SDK packages around a clean boundary and reworks benchmark authoring into a fully declarative model. No published users, so this is a breaking rework rather than a compat shim; scale stays on its own orchestration path.

• @benchsdk/client = REST transport + worker engine. Removed the authoring factories defineStep/defineTask/defineWorker/defineBench and the runBenchmarkWorker free function. client.runWorker({ task }) takes a raw TaskFunction whose context now exposes three channels: step() (imperative named steps), measure(data), and log(msg, meta?).
• @benchsdk/runner = the authoring framework (renamed from @benchsdk/cli). A *.bench.ts exports exactly two things — config and task — and never calls the runner itself; the new bench run <file> CLI owns the entrypoint.

Authoring model (the whole surface)

export const config = defineBenchmarkConfig({
  benchmarkSlug, benchmarkName, iterations, concurrency,
  participants,                       // orchestration data lives in config
  onComplete: (outcome) => writeLegacyResults(outcome.participants),  // run-level aggregate hook
});

export const task = defineTask(async ({ participant, step, measure, log }) => {
  const sandbox = await step('create', () => participant.createCompute().sandbox.create()); // return threads to next step
  try {
    const t0 = performance.now();
    await step('exec', () => sandbox.runCommand('node -v'));
    measure({ ttiMs: performance.now() - t0 });   // metrics → platform (step data / record data)
  } finally {
    await step('destroy', () => sandbox.destroy());
  }
});

bench run benchmarks/sandbox/sequential.bench.ts --iterations 100 --concurrency 20 --provider e2b,modal

Key semantics:

• step() returns are control-flow values, never auto-recorded — you thread live objects (a sandbox, a client) between steps. measure(data) is the explicit metric channel: inside a step() it merges into that step's data; at task top level it merges into the record. log() is human-readable timeline narration.
• A task with no explicit step() calls is recorded as one implicit 'task' step spanning the task's wall-clock, carrying any task-level measurements. Measurements are preserved even when a task throws.
• onResult is gone from the author surface (the runner still prints a default per-record line); per-iteration concerns live in the task, aggregate concerns in config.onComplete.

bench run entrypoint

cli.ts imports the module, validates config/task, and calls the now-internal runBenchmark(config, task, argv); NoAvailableParticipantsError maps to a clean exit. The bin re-imports the package by name (marked external in tsup) so the bin and the dynamically-imported *.bench.ts share one @benchsdk/runner instance — otherwise instanceof TaskError / NoAvailableParticipantsError would break across separately-bundled entries.

Migrations & housekeeping

• All 9 *.bench.ts migrated to config + task; each file's legacy aggregate writer moved into config.onComplete; removed runBenchmark tails, logTti/logAiGateway/logDax, and the bench-exit util. AI Gateway keeps its cold/warm phase parsing and zero-phase skip.
• Root package.json bench:* scripts now run tsx …/dist/bin.js run <file>.
• GitHub Actions benchmark workflows repointed to the runner CLI. Because the migrated *.bench.ts no longer self-run, the 7 workflows that invoked npx tsx <file>.bench.ts (sandbox-tti, sandbox-dax, storage, snapshot-fork, browser, browser-throughput, ai-gateway) would have silently run zero benchmarks. They now call npx tsx packages/benchsdk-runner/dist/bin.js run <file> … (dist is produced by the runner's prepare: tsup on pnpm install), and each bench step's load-vault-secrets allowlist gains the org-scoped BENCHMARKS_PLATFORM_API_KEY the runner requires (prod platform URL is the default, so no URL var needed). The other sandbox-* micro-benchmarks use the separate benchmarks/src/run.ts harness and are unaffected.
• create-bench scaffolds a declarative config+task project on @benchsdk/runner.
• READMEs, the local-e2e skill, and changesets updated; CLI unit tests + fixtures added.

Testing

• pnpm -r --filter "./packages/**" build — builds (CJS/ESM/DTS)
• pnpm typecheck — clean
• pnpm --filter @benchsdk/client test — 118 passed, 1 skipped
• pnpm --filter @benchsdk/runner test — 53 passed (incl. new CLI tests)
• pnpm --filter create-bench test — 1 passed
• Local platform E2E: bench run smoke against a local benchmarks-platform (Postgres/MinIO/ClickHouse) created a run and reported success (exit 0).
• CI invocation path verified locally: npx tsx packages/benchsdk-runner/dist/bin.js run <file> for sequential/storage/browser/ai-gateway all exit 0 via the "no participants" path with no creds; pnpm install --frozen-lockfile rebuilds dist/bin.js via prepare.

Tests were updated to match the intentionally-changed public API, not to paper over failures.

Note: the runner requires benchmarks-platform at origin/main (org-scoped API-key auth + organizationSlug returned from createRun); confirm prod is at that revision before dispatching the workflows.

Link to Devin session: https://app.devin.ai/sessions/3dbf4532c9e94ab6811ff85b49f1a203
Requested by: @HeyGarrison