fix: align repetition handling with vLLM (cutoff off by default) - #82

Problem

Qwen models generating large markdown tables (a |----------------------------------| rule, or many byte-identical rows) get silently truncated. The native check_repetition_cutoff runs every decode step and stops generation on N identical tokens in a row (Tier 1) or a repeating n-gram block (Tier 2). It fires on legitimate repeated structure — table rules, ASCII boxes, separators, repeated rows — and the "repetition" finish reason is then mapped to a clean end_turn, so the cut looks like a normal answer.

PR #81 only loosened the TS Qwen launch preset (256/8/64). Every other path — non-launch servers, Gemma4/LFM2, per-request configs that omit the fields, and the napi GenerationConfig::default() — still fell back to the tight native defaults (16/3/64), so as few as ~6 repeated tokens still truncated tables.

How vLLM handles it (the model for this change)

vLLM ships no repetition-stop heuristic by default. It shapes repetition only with logit-level penalties (repetition_penalty 1.0, frequency_penalty/presence_penalty 0.0 — all no-ops by default) and stops generation only on real conditions (EOS, stop tokens, stop strings, max_tokens). Its opt-in loop detector (repetition_detection, recent main) defaults to None, and when enabled surfaces its own distinct FINISHED_REPETITION finish reason rather than disguising the cut.

Change: fully align with vLLM — cutoff OFF by default, opt-in

• fix(sampling) — Introduce three named constants (all 0) in sampling.rs and route every former 16/3/64 default-fallback site through them (params.rs, qwen3 / qwen3.5 / qwen3.5-MoE / gemma4 / qianfan_ocr model files, and the napi GenerationConfig::default()). With the knobs unset the detectors resolve to disabled (0 already trips the existing check_consecutive / check_ngram guards). The detection algorithm is unchanged; GRPO training (grpo/engine.rs) and paddleocr keep their explicit values. New unit tests lock default-off and prove a positive value still re-enables detection (opt-in).
• fix(server) — Remove the now-unnecessary loosened cutoff (256/8/64) from all four Qwen launch presets; repetition is now shaped by the sampling penalties and bounded by maxOutputTokens, with per-request opt-in still winning via ChatSession.mergeConfig. Presets test flipped to assert the fields are absent while keeping the opt-in/override coverage.
• fix(types) — Sync the build-generated packages/core/index.d.cts cutoff doc comments to match the hand-synced crates/mlx-core/index.d.cts (both now say default: 0 = disabled), and rename a stale test describe block.

Behavior change / migration

A degenerate small model now runs to max_tokens (default max_new_tokens = 2048) instead of stopping at ~16 tokens. This is intended and vLLM-aligned — max_tokens is the backstop. Operators who want the old early repetition-stop must now set maxConsecutiveTokens / maxNgramRepeats / ngramSize explicitly (per-request or via registered samplingDefaults).

Testing

• Rust: cargo test -p mlx-core --lib repetition → 28 passing (TDD RED→GREEN; algorithm tests unchanged). Full lib suite 1878 pass; the 7 failures are pre-existing flaky f32/Metal numerical tests (banded-attn, attention-vjp, packed-affine embedding, sparse-moe gather, chunked-prefill parity) — verified to fail identically on base 73f4a89e, no path to this change.
• TS: vp test __test__/server/presets.test.ts → 7/7.

Out of scope (noted, not fixed here)

An adversarial review flagged that /v1/responses never clamps max_output_tokens against the registry's per-model outputTokenLimit (unlike /v1/messages). Independently verified as pre-existing (byte-identical at base 73f4a89e; this branch doesn't touch responses.ts/messages.ts) and not a regression of this change (the cutoff only ever caught exact-repeat loops, never bounded long coherent output). Worth a separate server-hardening PR.

🤖 Generated with Claude Code