Batched indexer intermittently wedges under sustained import pressure ("Stuck req_key") — still reproducible on 31.0.rc14, single node #3032

Summary

Under sustained bulk-import pressure, the batched indexer intermittently wedges on one
request: batched_indexer.cpp logs Stuck req_key: <id> (line 441 on v30.2, line 395 on
v31.0.rc14), the apply lane stops (queued_writes stays flat, applying: 0, raft
last_index frozen), and every write queues behind the stuck request. The stuck request
clears only when the submitting client disconnects (the "Running GC for aborted requests"
sweep then reaps it); the lane flows briefly and wedges again. We can reproduce this on
demand and have before/after data across v30.2 → v31.0.rc14.

Environment

• Self-hosted, single node, official Docker images (typesense/typesense:30.2, then
  31.0.rc14), Debian host, 251 GB RAM (no OOM, no swap-in during incidents, disk far
  from full — the Queued writes stuck #2137-style resource explanations are excluded).
• One large collection: ~1.1M documents, ~83 KB each, four 1024-dim float vector fields
  per document (precomputed, no auto-embedding), plus a handful of small collections.
• Read load during incidents: 15–45 search rps, ~300 ms mean (searches stay healthy
  throughout — the wedge affects writes only).
• Write pattern: a single writer process issuing one JSONL import at a time
  (/documents/import?action=upsert), 100–500 docs per request, sustained ~80–90 docs/s.

Observed on v30.2

• After tens of minutes of sustained importing, the node enters a wedge mode: imports of
  ≥ ~100-doc bodies hang indefinitely while tiny writes sometimes pass; effective apply
  throughput drops to single digits/s.
• Stuck req_key lines appear every few minutes. Status lines show e.g.
  pending_writes: 0, queued_writes: 3417, applying: 0 with last_index frozen while
  search threads are busy and NO indexer thread consumes CPU (verified via per-thread
  /proc deltas — the CPU burn is entirely the search pool).
• Probe matrix during a wedge: 1-doc upsert to a fresh throwaway collection: 12 ms;
  1-doc upsert to the big collection: sometimes 16 ms, sometimes hangs; 100-doc import
  (8.6 MB body): hangs past 60 s, alias or physical name alike.
• A 45-minute total write pause does NOT clear the mode; the first new import wedges
  within minutes.
• A node restart clears it — for ~8 minutes of full-speed imports, then it re-wedges.
• The same signature exists at a low background rate (~6/day) on a second, independent
  single-node installation with a similar collection shape.

Observed on v31.0.rc14 (upgraded specifically for the three 2026 batched_indexer fixes)

Clear improvement, not a cure:

• First wedge only after ~25 minutes / ~122k docs of sustained 85–91 docs/s importing
  (v30.2 wedged within minutes under the same backlog).
• The hourly timed snapshot completed cleanly in ~2 s mid-import-pressure (on v30.2 we
  have a history of snapshot-window write stalls).
• Alias state survived the version restart correctly (Fix broken reference on restart if alias collection name is used. #2919 works for us).
• But Stuck req_key then recurs (now from batched_indexer.cpp:395) at ~1 per 5
  minutes under continued pressure, with the same park-until-client-disconnect behavior
  and single-digit effective throughput between bursts.

What we can provide / questions

• Full timelines, raft status excerpts, per-thread CPU traces, and the probe matrices
  for both versions; we can reproduce within tens of minutes on demand against rc14.
• Is there a debug flag / log level that captures what the batched indexer is waiting on
  when it declares a req_key stuck? We can run it at the next occurrence.
• Is the park-until-disconnect behavior expected? A server-side timeout/requeue for a
  stuck head-of-line request would make the failure self-healing.

Related: #2137 (jasonbosco noted write-queue fixes in v28.0.rc36 — the family evidently
persists), community threads on stagnant queued_writes (v26–v29 reports).