nvidia

/

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

like 403

Follow

NVIDIA 64.3k

Move the two runtime monkey-patches from the eval script into the model file:

1. NemotronHHybridDynamicCache: initialize ssm_states with dtype=torch.float32 (matches vLLM 0.20.0+ DEFAULT_MAMBA_SSM_CACHE_DTYPE = 'float32' for NemotronH).

2. NemotronHMamba2Mixer: replace mamba_chunk_scan_combined with a wrapper (_make_mamba_chunk_scan_combined) that uses vLLM's mamba_chunk_scan_combined_varlen kernel with state_dtype=torch.float32 when vLLM is installed (batch_size=1), falling back to mamba_ssm otherwise. This makes HF single-sample prefill bit-compatible with vLLM 0.20.0 inference.

Verified on ChartQA (first 100 test samples, no reasoning, 256 max tokens):

• HF: 63%, vLLM: 59%, HF/vLLM correct-wrong agreement: 92%
• New model-file patches produce identical outputs to the monkey-patch approach (100% agreement).

This PR is not necessary for matching with vLLM.