← Back
NVIDIA
NVIDIA launches AI Cluster Runtime, open-source tool for reproducible Kubernetes GPU configurations
· open-sourcereleaseplatformintegration · developer.nvidia.com ↗

What Changed

NVIDIA announced AI Cluster Runtime (AICR), a new open-source project designed to solve the persistent problem of cluster configuration drift in AI workloads running on Kubernetes. Instead of manually configuring each cluster or struggling with upgrades that break existing setups, AICR publishes optimized, validated, and reproducible Kubernetes configurations as versioned recipes.

Key Capabilities

Recipe Generation: The aicr CLI generates customized recipes based on your target environment by specifying service provider (e.g., EKS), accelerator type (H100, Blackwell), intent (training/inference), OS, and platform (Kubeflow). Each recipe captures exact component versions, configuration values, deployment order, and constraints (minimum Kubernetes version, required OS/kernel).

Layered Configuration: Recipes are composed from modular layers rather than monolithic configurations:

  • Base layers define universal components and defaults
  • Environment layers add Kubernetes-specific components (e.g., EBS CSI driver on EKS)
  • Intent layers optimize for training or inference workloads, including NCCL tuning
  • Hardware layers pin driver versions and enable accelerator-specific features (CDI, GDRCopy)

A fully specialized recipe (e.g., Blackwell + EKS + Ubuntu + training + Kubeflow) can contain up to 268 configuration values across 16 components.

Snapshot & Validation: Users can snapshot running cluster state (OS release, kernel version, GPU hardware, Kubernetes version, operators) and validate deployments in phases—readiness checks before deployment, then health and conformance checks afterward. Conformance validation aligns with CNCF Certified Kubernetes AI standards, including dynamic resource allocation and gang scheduling.

Deployment Bundling: The bundler converts recipes into deployable artifacts with per-component folders containing values.yaml, integrity checksums, and READMEs, ready for Helm deployment.

Impact

This addresses a critical pain point: cluster operators often spend days replicating validated configurations or face breaking changes with component upgrades. AICR standardizes this process across clouds and hardware, with community-driven recipe updates as new validated configurations become available.