fix(qwen3.5-VL): correct image inference (interleaved M-RoPE, patch-embed, compressed-position decode + delta lifetime) - #80

qwen3_5 / qwen3_5_moe image inference rotated Q/K with PaddleOCR-VL's
sectioned (contiguous-chunk) multimodal-RoPE selector instead of
qwen3_5's interleaved (stride-3) selector. For image tokens (where
temporal != height != width) this assigns ~17% of frequencies to the
wrong spatial axis, producing garbage visual features. Text tokens were
unaffected because temporal == height == width makes the three cos/sin
axis rows bit-identical, so any selector yields the same angles — which
is exactly why text worked and images did not.

Add `apply_multimodal_rotary_pos_emb_interleaved`, which builds the
per-frequency axis selector matching mlx-vlm's
`_interleaved_position_selector` (height: idx 1,4,7,… up to
section[1]*3; width: idx 2,5,8,… up to section[2]*3; temporal
otherwise), mirrors it across the doubled cos/sin, and gathers the
selected t/h/w axis per frequency via `take_along_axis` before the same
rotate_half + partial-rotary tail. Switch qwen3_5 attention's two M-RoPE
call sites (forward + forward_paged) to it; this covers qwen3_5_moe too
(shared `Qwen3_5Attention`). PaddleOCR-VL keeps the sectioned path
(unchanged; its production forward uses the C++ sectioned kernel).

Tests (no model): interleaved selection correctness against the
hand-computed selector, and a text-invariance hard gate asserting the
interleaved apply is bit-identical to the sectioned apply when
t==h==w (proving the text path cannot regress).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The qwen3_5 shared vision patch-embed loader collapsed the 5D Conv3d
weight [out, kD=2, kH, kW, in] by taking only temporal slice 0 and never
loaded the Conv bias. The image processor duplicates the static frame
across the temporal axis (mlx-vlm qwen3_vl Conv3d(bias=True), kD=2), so
the effective 2D kernel is the SUM of the temporal slices plus the bias.

Sum over the temporal axis (robust to kD != 2) and plumb the optional
patch_embed.proj.bias through set_patch_embed / PatchEmbedding::new into
the Conv2d bias (was hardcoded None). get_parameters now round-trips the
bias. Shared path (qwen3_5 / qwen3_5_moe / Qwen3.6-35B-A3B), not forked.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

`MxArray::addmm` called `mlx_array_addmm`, but the vendored `mlx::core::addmm`
in this build returns only `alpha*(A@B)` and silently drops `beta*C`. That was
invisible for the bias-free LM linears (Q/K/V/O/MLP and the bias-free MoE router
gate all pass a zero C), but it corrupted every biased linear — most visibly the
Qwen3.5-VL vision tower, whose qkv/proj/fc1/fc2 and merger projections all carry
a bias. Dropping those biases produced semantically wrong image features (the
model described the ocr.png financial table as "stone/fabric").

Compute the result explicitly (matmul, optional alpha scale, then add beta*C) so
the C term is actually applied. Numeric bisection vs mlx-vlm on converted bf16
ornith: vision-tower image_embeds per-token cosine median 0.9996 after the fix
(block0 rel-err 35x -> 0.0078). Add nn::linear unit tests proving addmm applies
a [4]/[1,4]/[2,4] C and Linear::forward applies its bias.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

An image run compresses its placeholder tokens into fewer M-RoPE positions,
so image prefill rotates Q/K at the compressed positions and records
rope_deltas = max_position + 1 - seq_len (negative). Decode and warm
continuation must keep rotating at that compressed position
(physical_slot + rope_deltas) while K/V is still written at the physical
slot. The scalar-offset RoPE path rotated at the raw physical token count,
ignoring rope_deltas, so post-image decode ran ~725 positions off and
produced repetition garbage.

Carry rope_deltas on VisionMerge, store it as cached_rope_deltas at every
VLM prefill, and thread a rope_position_offset (physical position +
cached_rope_deltas, cast u32->i32 before the negative add) through
forward_paged, the decoder-layer paged forward, and the paged decode/prefill
drivers for both qwen3_5 and qwen3_5_moe. get_rope_index takes the global
max over the (t,h,w) axes for the delta. Text turns store no delta, so the
offset equals the physical position and non-VLM paths stay byte-identical.

Matches paddleocr_vl (cache_offset + rope_deltas) and mlx-vlm
(base_offset + rope_delta).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…che hits

The cross-turn M-RoPE delta an image prefill bakes into the shared
`cached_rope_deltas` was reset only when `cached_prefix_len == 0`. But the
paged turn planner also produces `cached_prefix_len > 0` with
`continued_live_prefix == false` on a non-live prefix-cache hit: a later
text request that merely shares a cached text prefix with an earlier image
request (the model is shared across all sessions). There the old gate did
not fire, so a stale negative delta survived and rotated unrelated text at
`physical_slot + stale_delta` instead of the raw physical position.

Only a live continuation (`continued_live_prefix`) re-attends the image's
physically-resident compressed-position K/V, so only it needs the delta:
image requests prefill with `skip_lookup` and never publish a text stream
that collides with their expanded-placeholder blocks, so every non-live hit
restores pure-text prefix blocks (delta 0).

Factor the decision into `rope_delta_for_paged_turn(current,
continued_live_prefix)` and wire all six paged gates (dense + MoE sync,
stream, engine paths) through it. Add model-free lifecycle regression tests
covering the live-continuation, cold-start, and stale-delta-on-text-hit
cases.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… mlx bug

The earlier addmm rewrite attributed the dropped `beta*C` to a bug in
`mlx::core::addmm`. That premise was wrong: PyPI MLX's addmm applies `C`
correctly and the FFI wrapper passes its args correctly. The real cause was a
corrupt local metallib that miscompiled the fused GEMM kernels (the same bad
build also miscompiled the NAX gemm). The explicit matmul+add form is kept as
robustness against this project's documented non-deterministic metallib
corruption, and the nn::linear C-application tests double as a build canary —
but the code comment no longer claims an mlx source bug.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>