ShellRisk-Bench

ShellRisk-Bench is a reproducible benchmark for context-free risk classification of individual shell commands. It tests whether a system can distinguish generally risky commands from ordinary development and operations traffic when it sees only the submitted command.

The benchmark is built from six pinned public sources. Upstream data is downloaded locally and normalized into a common schema; it is not vendored in this repository.

ShellRisk-Bench v0.1 evaluates one binary question:

Does this individual shell-command submission pose meaningful cyber or system risk?

It does not infer user intent, inspect surrounding task context, evaluate multi-command sessions, or make a final authorization decision. Multi-line scripts and sessions are excluded rather than flattened into misleading atomic labels.

See DATASETS.md for pinned revisions, transformations, provenance, and upstream license information.

The published comparison uses a deterministic same-source, in-distribution split:

• 161,383 command rows before global deduplication
• 160,220 unique command strings
• 9 cross-label collisions removed
• All 966 risky commands retained
• Benign commands deterministically capped at 20,000
• Stratified 80/20 split with seed 13
• Test set: 4,194 commands—193 risky and 4,001 not risky

This split measures performance on unseen strings from known source distributions. It is not a source-transfer result. Source-grouped evaluation is the appropriate test for novel command dialects; every generated row retains its source so that evaluation can be performed separately.

The approximately 20:1 test mix is a constructed operating point for comparing precision under class imbalance. It is not presented as an empirical measurement of all production shell traffic.

All systems were scored on the same 4,194 commands. Hosted-model latency was measured sequentially and includes the API round trip. Shieldstral used its default 0.5 threshold; Llama Guard counted any unsafe category as risky. Quality results, prompts, and the Kestrel per-example verdicts are under results/. The Kestrel implementation is not part of this benchmark repository. The portable Kestrel v0.1.0 model artifact is published separately on Hugging Face at the pinned release revision and is not duplicated here. Its SHA-256 checksum is 1df8b3e5f2bfc4e1fe95230ee9b3d37f63aaa461a8551982fcc0dd7103c8221b.

Python 3.11 or newer is recommended.

python3 -m venv .venv
.venv/bin/pip install -e '.[test]'
.venv/bin/python -m shellrisk_bench.build
.venv/bin/python -m shellrisk_bench.prepare --verify

The first command downloads several gigabytes of upstream trajectories. For a quick adapter smoke test:

.venv/bin/python adapters/swesmith.py --limit 50
.venv/bin/python adapters/terminalbench.py --limit 50

After building the fixed split, score a JSONL prediction file:

.venv/bin/python -m shellrisk_bench.score \
  --gold data/splits/test.jsonl \
  --predictions results/kestrel.predictions.jsonl

Prediction rows contain a stable command hash and a binary verdict:

{"id":"sha256:…","prediction":"risky"}

Create the files used by the Hugging Face Dataset Viewer and datasets.load_dataset() from the verified local split:

.venv/bin/python -m shellrisk_bench.export_huggingface

This writes dist/huggingface/README.md, one Parquet file per split, the canonical split manifest, and an export manifest containing file hashes. The command does not upload anything. The dataset card is maintained under huggingface/README.md.

ShellRisk-Bench is maintained by Kontext Security. To see Kestrel evaluating the cyber risk of agent tool calls as part of Kontext, visit kontext.security.

This repository processes potentially destructive commands as inert text. Nothing in the build or evaluation path executes benchmark commands. Do not pipe dataset contents into a shell.

The benchmark code is licensed under Apache-2.0. Upstream datasets retain their own licenses and terms; see DATASETS.md. Generated data and Hub export files are git-ignored; publishing them requires a separate source-by-source redistribution review.