← Back
NVIDIA
NVIDIA Dynamo 1.0 achieves 7x inference throughput boost on Blackwell with production-grade multi-node support
· releasefeatureperformanceapiplatformintegration · developer.nvidia.com ↗

Production-Ready Distributed Inference

NVIDIA Dynamo 1.0 is now available as a mature, production-grade framework for distributed AI inference across multiple GPU nodes. Built to address the challenges of deploying large reasoning models and agentic AI workflows at scale, Dynamo enables low-latency, high-throughput inference with careful orchestration and coordination across GPUs.

Performance Improvements and Benchmarks

The framework delivers significant performance gains, boosting inference throughput by up to 7x on NVIDIA Blackwell hardware when using disaggregated serving with wide expert parallelism. These results are validated by trusted third-party benchmarks including MLPerf and SemiAnalysis InferenceX, positioning Dynamo as a leading production inference platform.

Ecosystem Integration and Adoption

Dynamo has achieved broad adoption across the industry:

  • Production deployments: AstraZeneca, Baseten, ByteDance, CoreWeave, Crusoe, DigitalOcean, Gcore, Meituan, Pinterest, Prime Intellect, Tencent Cloud, Together AI, Vultr, and many others have deployed Dynamo to scale multi-node inference and optimize throughput.
  • Cloud platform integrations: Alibaba Cloud, AWS, Google Cloud, Microsoft Azure, and Oracle Cloud Infrastructure have built native integrations into their managed Kubernetes environments.
  • Inference engine support: The framework supports leading open-source engines including SGLang, NVIDIA TensorRT LLM, and vLLM, with modular components like NIXL widely adopted for KV cache acceleration.

Key Features and Optimizations

Recent enhancements include:

  • Agentic inference optimizations: Priority-based request routing and cache pinning for improved coordination across models.
  • Multimodal acceleration: Disaggregated encode/prefill/decode stages, embedding caching, and multimodal KV routing for efficient multi-model inference.
  • Native video generation support: Direct support for video-generation models in distributed environments.
  • ModelExpress: Achieves 7x faster model startup via checkpoint restore and weight streaming leveraging NVIDIA NVLink and NIXL.
  • Enhanced Kubernetes orchestration: Grove API for topology-aware scheduling on NVIDIA GB300 NVL72 clusters.
  • Zero-config deployment: DGDR support for simplified cluster setup.
  • Resilient inference: Layered fault detection, request cancellation, and migration capabilities.
  • Flexible KV Block Manager: Pip-installable with object storage integration for flexible deployment options.