Summary

The prefill worker (inkwell-copper-cn01) crashes with RuntimeError: Failed: CUDA error /build/DeepEP/csrc/deep_ep.cpp:112 'invalid resource handle' followed by an illegal memory access at line 278, causing the entire job to fail.

Error

RuntimeError: Failed: CUDA error /build/DeepEP/csrc/deep_ep.cpp:112 'invalid resource handle'
terminate called after throwing an instance of 'EPException'
  what():  Failed: CUDA error /build/DeepEP/csrc/deep_ep.cpp:278 'an illegal memory access was encountered'

From: inkwell-copper-cn01_prefill_w0.out:1931 and inkwell-copper-cn01_prefill_w0.out:1946-1947

Config

The recipe passes these flags to prefill workers:

• moe-a2a-backend: deepep
• deepep-mode: normal
• tp-size: 8
• ep-size: 8
• dp-size: 8
• ep-dispatch-algorithm: dynamic
• ep-num-redundant-experts: 32
• mem-fraction-static: 0.75
• disable-cuda-graph: true

Decode workers use different values (tp-size=32, ep-size=32, deepep-mode=low_latency, ep-dispatch-algorithm=static).

Job

• Job ID: 5051 (dsr1-fp8-1k1k-max-tpt-26203405870)
• Model: deepseek-ai/DeepSeek-R1 (DeepSeek-V3)
• Precision: FP8
• Topology: 2 prefill nodes (4 GPUs each) + 8 decode nodes (4 GPUs each)
• Container: lmsysorg_sglang_dev-cu13_20260521

Timeline

• 04:33:30 — Prefill workers complete first DeepGEMM warmup
• 04:34:21 — Prefill workers begin second DeepGEMM JIT session
• 04:35:30 — Prefill worker 0 crashes with SIGQUIT
• 04:35:38 — etcd connection failures begin (consequence)
• 04:35:44 — srtctl cleans up

Suggested Fix

1. Add validation in the recipe to reject flag combinations that DeepEP cannot support — the "invalid resource handle" error suggests the CUDA resource configuration exceeds what the driver/hardware can provide
2. Check DeepEP's max EP/TP limits — consider reducing ep-size from 8 to a validated maximum for prefill workers with deepep backend
3. Consider reducing decode ep-size from 32 to 16 or 8 to stay within validated bounds
4. Add a pre-flight check that validates DeepEP can initialize with the given topology before starting workers