Summary

The prefill worker crashes with RuntimeError: Failed: CUDA error /build/DeepEP/csrc/deep_ep.cpp:112 'invalid resource handle' on GB200 GPUs when using moe-a2a-backend: deepep, causing the entire job to fail.

Error

RuntimeError: Failed: CUDA error /build/DeepEP/csrc/deep_ep.cpp:112 'invalid resource handle'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /build/DeepEP/csrc/deep_ep.cpp:278 'an illegal memory access was encountered'
Fatal Python error: Aborted

From: inkwell-copper-cn01_prefill_w0.out:2086

Config

The recipe passes these flags to prefill workers:

• moe-a2a-backend: deepep
• deepep-mode: normal
• tp-size: 8
• ep-size: 8
• dp-size: 8
• ep-dispatch-algorithm: dynamic
• ep-num-redundant-experts: 32
• mem-fraction-static: 0.75
• disable-cuda-graph: true

Job

• Job ID: 5118 (dsr1-fp8-1k1k-ultra-tpt-26993287488)
• Model: deepseek-ai/DeepSeek-R1
• Precision: FP8
• Topology: 2 prefill nodes (4 GPUs each, gb200) + 2 decode nodes (4 GPUs each)
• Container: lmsysorg_sglang_dev-cu13_20260605

Timeline

• 03:28:01 — Prefill workers started
• 03:28:15 — Health check polling began
• 03:34:45–03:40:45 — Repeated health check failures (0 prefills + 0 decodes ready)
• 03:41:29 — Prefill worker 0 crashes with SIGQUIT
• 03:41:42 — Job terminated with exit code 137

Relationship to Issue #171

This is the same bug as issue #171 (opened 2026-05-21). The bug remains unfixed after ~2 weeks. This job (5118) uses a newer container (20260605 vs 20260521) but exhibits the same DeepEP initialization failure.

Suggested Fix

1. Add pre-flight validation to reject DeepEP + GB200 + high ep-size combinations
2. Reduce default ep-size for DeepEP on GB200 from 8 to 4 or lower
3. Add fallback to nixl moe-a2a-backend when DeepEP fails to initialize on GB200
4. Consider adding allow_mnnvl=False or similar flag to disable MNNVL-specific resource handling that may be incompatible with DeepEP on GB200