NemotronH LoRA: default target_modules only cover 7% of layers, silent failure #3554

Problem

PEFT's default LoRA target modules for nemotron_h (added in PR #3289) only target attention projections (q_proj, k_proj, v_proj, o_proj). For NemotronH's hybrid Mamba-2 + Attention architecture, this means LoRA only touches 4 out of 56 layers (7.1%) — the other 52 layers are Mamba SSM layers with different parameter names (in_proj, out_proj, A_log, D, dt, x_proj, z_proj).

Worse, PEFT silently skips target module names that don't exist in the model. There's no warning that the configured target modules only matched a small fraction of layers. The user sees training "complete" with loss not improving, and has no idea why.

Reproduction

We ran DPO (LoRA) on nvidia/NVIDIA-Nemotron-Nano-9B-v2 with the default PEFT target modules:

from peft import LoraConfig
config = LoraConfig(
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],  # PEFT default for nemotron_h
    r=16, lora_alpha=32, lora_dropout=0.1,
)

Results:

• Training completed (25/25 steps, 12 minutes)
• Loss stayed at 0.6931 (ln(2)) for all 25 steps — zero learning
• HumanEval: 0.0% (0/164) — the LoRA adapter produced no behavioral change

The Nemotron-Nano-9B-v2 has 56 layers with this hybrid pattern:

M-M-M-MM-M-M-M*-M-M-M*-M-M-M-M*-M-M-M-M*-M-MM-M-M-M-M-M-

Where M = Mamba, * = Attention. Only 4 layers are attention. The other 52 are Mamba layers with parameters in_proj, out_proj, A_log, D — none of which were targeted by LoRA.

Root Cause

1. PEFT PR Add default target_modules for nemotron_h hybrid Mamba-MoE models #3289 added nemotron_h to TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING with only attention modules
2. PEFT PR Prevent applying LoRA to disallowed modules in Mamba-based architectures #2562 added nemotron_h to mamba_model_types in _check_lora_target_modules_mamba(), which forbids targeting Mamba modules (in_proj, out_proj, conv1d)
3. The combination means: PEFT defaults only target attention layers, and PEFT actively blocks you from targeting Mamba layers
4. No warning is emitted when target modules match an unusually low number of parameters

Proposed Fixes

Fix 1: Warn on low coverage

When LoraConfig.target_modules matches fewer than N% of linear layers in the model, emit a warning:

if matched_layers / total_linear_layers < 0.3:
    warnings.warn(
        f"LoRA target modules only matched {matched_layers}/{total_linear_layers} "
        f"linear layers ({100*matched/total:.1f}%). For hybrid architectures like "
        f"NemotronH, consider including Mamba modules (in_proj, out_proj) in target_modules."
    )

Fix 2: Document hybrid architecture target modules

Add documentation for hybrid Mamba+Attention models showing the correct target modules:

# For NemotronH / Jamba / Granite-H hybrid models:
LoraConfig(
    target_modules=[
        # Attention layers
        "q_proj", "k_proj", "v_proj", "o_proj",
        # Mamba layers (requires removing from mamba_model_types check)
        "in_proj", "out_proj", "x_proj", "dt_proj",
    ],
    # Skip: conv1d, A_log, D (structured SSM parameters, not suitable for LoRA)
)

Fix 3: Reconsider the Mamba module ban

PR #2562 banned Mamba modules from LoRA targeting due to shape issues. But in_proj and out_proj are standard nn.Linear layers — they should be LoRA-compatible. The ban should be scoped to conv1d and structured SSM parameters (A_log, D), not all Mamba modules.

Environment

• peft: latest (with PR Add default target_modules for nemotron_h hybrid Mamba-MoE models #3289 merged)
• transformers: 5.16.0.dev0
• Model: nvidia/NVIDIA-Nemotron-Nano-9B-v2 (56 layers, 4 attention + 52 Mamba)
• GPU: NVIDIA GB10 (119GB VRAM)

Related

• PEFT PR Add default target_modules for nemotron_h hybrid Mamba-MoE models #3289: Added default target_modules for nemotron_h (attention only)
• PEFT PR Prevent applying LoRA to disallowed modules in Mamba-based architectures #2562: Added Mamba target module check (bans in_proj, out_proj)
• Unsloth issue #3854: "Trouble fine-tuning Nemotron 3 Nano, failure when merging LoRA"
• Full writeup: https://github.com/davidnichols-ops/claude-yolo-vibes-v5/blob/master/EVALUATION.md