← Back
LMSYS
SGLang adds Elastic Expert Parallelism for MoE models, tolerating up to 16 GPU failures with <10-second recovery
· featureapiperformanceplatformintegration · lmsys.org ↗

Problem: Fragility at Scale

Large MoE model deployments require wide Expert Parallelism (EP) spanning 32+ GPUs per inference instance to maximize batch sizes and reduce cost per token. However, this approach creates a critical reliability vulnerability: as EP size grows, so does the statistical probability of hardware failures or process crashes. Traditional SGLang MoE deployments required full server restarts when failures occurred, causing 2–3 minute outages and significant resource waste.

Solution: Elastic EP with Redundant Experts

Elastic EP decouples the rigid mapping between experts and GPUs by maintaining redundant experts across the cluster. When a failure is detected, the system automatically redistributes expert weights and reroutes inference tokens to surviving GPUs without halting ongoing operations.

The implementation introduces two complementary layers:

  • Scheduler Layer: Continuously monitors Data Parallel (DP) rank health and filters failed ranks from batch assignment, preventing new requests from routing to failed resources.
  • Expert Parallel Layer: Dynamically adjusts expert-to-GPU mappings in real-time, redistributing experts across surviving EP members to maintain mathematical correctness.

Performance and Resilience

Testing on DeepSeek V3.2 with 32 GPUs (4 nodes) and 256 redundant experts demonstrated:

  • Sub-10-second recovery: Service interruption averaged 6.2–6.8 seconds even under extreme 16-rank failures, compared to 120–180 seconds for traditional restarts (90% improvement).
  • Graceful degradation: With reduced resources after recovery, the system continues inference at proportionally reduced throughput (2,825 tokens/sec with 16 failed ranks vs. baseline 5,500+).
  • Zero baseline overhead: Static performance metrics (throughput, TTFT, TPOT) match standard EP implementations under normal conditions.

Implementation Details

Elastic EP leverages Mooncake, a fault-tolerant PyTorch distributed backend, as its communication foundation. Mooncake provides resilient collective operations, specialized MoE primitives (dispatch/combine), GPU Direct RDMA support, and rapid fault detection via timeout mechanisms.

To enable Elastic EP, start SGLang with:

  • --elastic-ep-backend mooncake
  • --moe-a2a-backend mooncake
  • --mooncake-ib-device <devices>
  • --ep-num-redundant-experts <count>

Higher redundancy values tolerate more simultaneous failures at the cost of increased memory overhead.