NVIDIA Encodes Kubernetes Adaptation Decisions for Vera Rubin NVL72 into Reproducible Recipes, Validating Cross-Node NCCL Performance for the First Time
NVIDIA engineers Mark Chmarny and Yuan Chen documented how they completed end-to-end Kubernetes validation on early Vera Rubin NVL72 hardware, encoding findings such as driver branches, NIC topology, IMEX exclusivity, and kernel page sizes into versioned NVIDIA AI Cluster Runtime (AICR) recipes. These were validated through deployment, conformance, and performance phases on a reference cluster, though inference performance was deferred due to an unready software stack.
Configuring Kubernetes for new GPU hardware is a process prone to errors. Drivers, kernels, container runtimes, networking, operators, and schedulers must work in concert, with settings tuned specifically to the hardware and workload. Even minor mismatches can prevent workloads from running or limit their performance. In a technical blog post dated November 2, 2026, NVIDIA cloud architect Mark Chmarny and software engineer Yuan Chen detailed how they completed this adaptation on early Vera Rubin NVL72 hardware and solidified their experience into versioned recipes within the NVIDIA AI Cluster Runtime (AICR).
The upgrade from GB200 NVL72 to Vera Rubin NVL72 involves more than fine-tuning; user-visible changes are significant. The GPU driver requires a new branch: the validated R615.23 is an open-source kernel module driver pre-installed on the host, while the official platform support target is R620. On the network side, ConnectX-7 NICs and BlueField-3 DPUs are replaced by ConnectX-9 SuperNICs and BlueField-4 DPUs, with DOCA 3.3.0 being the first GA release supporting ConnectX-9. The device topology also differs: the Vera Rubin NVL72 compute tray exposes eight groups of ConnectX-9 PCI devices totaling 32 functions, whereas the GB200 has only four ConnectX-7 adapters. Multi-node NVLink workloads still rely on IMEX, but DRA-managed IMEX is mutually exclusive with the host's static nvidia-imex service, which must be disabled.
These changes are scattered across various layers, from low-level bindings to the orchestration layer. AICR addresses this by encoding all findings into recipes: host-managed mode drivers (GPU Operator sets driver.enabled: false, while the container toolkit remains operator-managed), DRA drivers enabling ComputeDomains, a minimum Kubernetes version of 1.34.1 (1.34 provides the GA resource.k8s.io/v1 DRA API, and the .1 patch is required because the RKE2 CDI/NRI toolkit path needs containerd 2.1), Ubuntu 26.04 paired with a Vera-optimized arm64 64k-page kernel, and raising the NFD topology updater's default 60Mi memory limit—otherwise, it triggers OOM kills on multi-TB NVL72 worker nodes. The recipe also introduces DraNet for allocating ConnectX-9 devices to Pods and employs NodeWright to enforce an exclusive RDMA network namespace mode, ensuring each Pod sees only its own Host Channel Adapter.