NVIDIA Encodes Kubernetes Adaptation Decisions for Vera Rubin NVL72 into Reproducible Recipes, Validating Cross-Node NCCL Performance for the First Time
NVIDIA engineers Mark Chmarny and Yuan Chen documented how they completed end-to-end Kubernetes validation on early Vera Rubin NVL72 hardware, encoding findings such as driver branches, NIC topology, IMEX exclusivity, and kernel page sizes into versioned NVIDIA AI Cluster Runtime (AICR) recipes. These were validated through deployment, conformance, and performance phases on a reference cluster, though inference performance was deferred due to an unready software stack.
Configuring Kubernetes for new GPU hardware is a process prone to errors. Drivers, kernels, container runtimes, networking, operators, and schedulers must work in concert, with settings tuned specifically to the hardware and workload. Even minor mismatches can prevent workloads from running or limit their performance. In a technical blog post dated November 2, 2026, NVIDIA cloud architect Mark Chmarny and software engineer Yuan Chen detailed how they completed this adaptation on early Vera Rubin NVL72 hardware and solidified their experience into versioned recipes within the NVIDIA AI Cluster Runtime (AICR).
The upgrade from GB200 NVL72 to Vera Rubin NVL72 involves more than fine-tuning; user-visible changes are significant. The GPU driver requires a new branch: the validated R615.23 is an open-source kernel module driver pre-installed on the host, while the official platform support target is R620. On the network side, ConnectX-7 NICs and BlueField-3 DPUs are replaced by ConnectX-9 SuperNICs and BlueField-4 DPUs, with DOCA 3.3.0 being the first GA release supporting ConnectX-9. The device topology also differs: the Vera Rubin NVL72 compute tray exposes eight groups of ConnectX-9 PCI devices totaling 32 functions, whereas the GB200 has only four ConnectX-7 adapters. Multi-node NVLink workloads still rely on IMEX, but DRA-managed IMEX is mutually exclusive with the host's static nvidia-imex service, which must be disabled.
These changes are scattered across various layers, from low-level bindings to the orchestration layer. AICR addresses this by encoding all findings into recipes: host-managed mode drivers (GPU Operator sets driver.enabled: false, while the container toolkit remains operator-managed), DRA drivers enabling ComputeDomains, a minimum Kubernetes version of 1.34.1 (1.34 provides the GA resource.k8s.io/v1 DRA API, and the .1 patch is required because the RKE2 CDI/NRI toolkit path needs containerd 2.1), Ubuntu 26.04 paired with a Vera-optimized arm64 64k-page kernel, and raising the NFD topology updater's default 60Mi memory limit—otherwise, it triggers OOM kills on multi-TB NVL72 worker nodes. The recipe also introduces DraNet for allocating ConnectX-9 devices to Pods and employs NodeWright to enforce an exclusive RDMA network namespace mode, ensuring each Pod sees only its own Host Channel Adapter.
The validation process executes in four steps on a live Vera Rubin NVL72 reference cluster. Step one generates the recipe, offline parsing 14 components and 5 overlay layers. Step two renders the deployment bundle, producing 61 files including ordered Helm charts and deployment scripts. Step three snapshots the cluster and runs validation: the deployment phase passes all four checks; the conformance phase passes eight, covering GPU Operator health, DRA/ComputeDomain support, KAI gang scheduling, Pod autoscaling, and secure accelerator access. Step four is the cross-node ComputeDomain test, verifying IMEX channel injection and NVLink fabric connectivity. The performance phase runs a cross-node NCCL all-reduce, completing a 16 GiB message all-reduce over the NVLink fabric via IMEX ComputeDomain, achieving a bus bandwidth of 744.71 GB/s, meeting the recipe's floor. The entire sequence (recipe, bundle, snapshot, deployment, and conformance validation) completes in about five minutes, with the performance phase adding the runtime of the NCCL job itself.
However, boundaries must be clearly understood. The recipe is in Preview status and does not carry production support or lifecycle qualification. Validation was self-reported by the vendor on NVIDIA reference hardware, with no independent reproduction. Inference serving performance is explicitly deferred: vLLM merged Rubin (sm_107a) support code in v0.27.0, but the SM107 kernel path is gated at build time on CUDA ≥ 13.4, while vLLM’s published images up to v0.28.0 are still built against CUDA 13.0.3. Thus, the latest image carries dispatch code but lacks compiled kernels. ai-dynamo’s latest release, v1.4.2, pins vllm-openai v0.26.0, which predates this code. An available inference performance floor requires a published runtime image built on CUDA 13.4 and verified on real hardware.
The validation process itself drove upstream fixes in reverse. When running on early silicon, AICR discovered an unmapped Rubin architecture value in go-nvlib that caused the DRA driver’s kubelet plugin to crash-loop. Because architecture support lands first in go-nvml, flows into go-nvlib, and then into the GPU Operator, Kubernetes Device Plugin, MIG Manager, and DRA plugins, the fix propagated consistently down the stack. From the first crash report to a validated build on Vera Rubin hardware took about three days, finally shipping with the official DRA driver v0.5.0 release. Validation also revealed that host-managed IMEX daemons and DRA-managed ComputeDomains cannot currently share a node, feedback that shaped the host-managed IMEX option added in DRA driver v0.5.0.
For platform teams, the practical value of this record lies not in the number 744.71 GB/s itself—it is a vendor self-test result on reference hardware—but in how it demonstrates that the complete decision list for new hardware adaptation is solidified into reproducible, verifiable recipes. This allows a second cluster to repeat the same results the next day without having to re-navigate pitfalls like NFD memory limits or IMEX exclusivity conflicts.