ReadingResearchRadarInvestment frameworkAI Company Primary Sources
Sign in / Sign up
← Back to reading
NewsMaterial5 min readDisconfirmAI Editorial

NVIDIA AI Cluster Runtime Separates Dynamo Inference Kubernetes Runtime from Application Deployment into a Five-Step Validated Workflow

NVIDIA AI Cluster Runtime packages Dynamo and its dependent infrastructure—GPU drivers, dynamic resource allocation, gang scheduling, node health, and monitoring—into version-locked recipes. It achieves runtime readiness through a five-step process: snapshotting the cluster, generating the recipe, rendering the deployment bundle, deploying the runtime, and validating the installation, only then allowing the application inference graph to be installed. The article also compares the applicable scenarios for the Dynamo Frontend routing path versus the Gateway API Inference Extension/Endpoint Picker path.

2026-10-31 05:35 GMT+8

NVIDIA Dynamo is a distributed inference framework that coordinates model serving across multiple GPU worker nodes. Deploying it on Kubernetes superficially requires only a DynamoGraphDeployment to specify how vLLM, SGLang, or TensorRT-LLM should run in pods. However, actual production deployment involves much more: the GPU driver stack must match the nodes, GPU nodes need labeling and discovery, the cluster must expose GPUs via Dynamic Resource Allocation, the scheduler must support gang scheduling to meet topology requirements, node health signals must be available before workloads land on degraded nodes, Prometheus must be reachable to collect accelerator and Dynamo metrics, and if the deployment requires Kubernetes-native ingress, the Gateway API and Endpoint Picker paths must also be configured and ready.

The design philosophy of NVIDIA AI Cluster Runtime (AICR) is to decouple these infrastructure dependencies from application deployment. It does not require teams to manually install and debug each component one by one; instead, it packages Dynamo and all its runtime dependencies into a single recipe. The recipe records locked component versions, compatibility constraints, and validation checks, making the deployment reproducible. AICR v1.0.0 locks the following component versions for the Dynamo inference recipe on Amazon EKS: NVIDIA GPU Operator v26.7.1, Node Feature Discovery 0.19.0, cert-manager v1.20.2, AWS EFA v0.5.29, NVSentinel v1.25.0, NVIDIA DRA GPU Driver 0.5.0, KAI Scheduler v0.16.9, Grove v0.1.0-alpha.13, Dynamo Platform 1.4.2, Agent Gateway v1.5.0, and Kube Prometheus Stack 84.4.0. These versions were jointly tested by the AICR team on the specified platforms and hardware.

Sign in / Sign up
ReadingResearchRadarInvestment frameworkAI Company Primary Sources

The entire process consists of five steps. Step 1: The aicr snapshot command captures the current state of the cluster, automatically targeting nodes labeled nvidia.com/gpu.present=true by default and tolerating GPU node taints. Step 2: The aicr recipe command identifies the Kubernetes distribution, GPU accelerators, and operating system based on the snapshot, and generates a recipe file combining workload intent (inference) and platform (dynamo). Step 3: The aicr bundle command renders the recipe into deployable Helm values and manifests, using node selectors and toleration flags to specify placement policies for GPU worker nodes and system nodes. Step 4: Deploy the runtime: when no GitOps controller is present locally, Helmfile can be used to apply resources in dependency order; in production environments, ArgoCD or Flux GitOps bundles can be rendered. Step 5: The aicr validate command performs two-stage validation (deployment and conformance) on the cluster after installation but before application workloads are deployed. Checks cover cluster baseline resources and GPU allocatable capacity, GPU Operator health and version, DRA support and secure accelerator access, gang scheduling and autoscaling, Prometheus metrics collection, and Gateway API and agentgateway readiness. Only upon successful validation is the DynamoGraphDeployment installed.

At the routing level, Dynamo 1.4.2 supports two Kubernetes paths, and AICR provides recipes and validation for both. Path 1 is the Dynamo Frontend route: the Frontend runs with DYN_ROUTER_MODE=kv, tokenizes each request, scores candidate workers using prefix overlap and current load, selects a worker, and dispatches the request via the TCP request plane. This path has the minimal topology and is suitable for scenarios where an application team owns a model-specific endpoint, or an existing edge proxy only needs to forward traffic to the Dynamo Frontend Service. It is currently the most production-proven serving option. Path 2 is the Gateway API Inference Extension (GAIE) / Endpoint Picker Plugin (EPP) route: agentgateway resolves HTTPRoute to an InferencePool. The Dynamo operator creates this InferencePool from the DynamoGraphDeployment, where its selector identifies candidate worker Pods, and its endpointPickerRef identifies the Dynamo EPP Service. The Gateway invokes the EPP via GAIE’s endpoint selection interface; the EPP uses Dynamo’s KV router to select an endpoint and returns it to the Gateway, which then forwards the request to the selected worker’s Frontend sidecar (running with --router-mode direct, thus no longer rerouting). This path is suitable for platform teams managing shared Kubernetes ingress, where timeouts, retries, and fallback strategies need to be configured via Gateway API mechanisms.

Both paths share the same KV-cache event plane. In Dynamo 1.4.2, each vLLM worker engine emits raw KV-cache events via a local ZMQ publisher (configuration example: --enable-prefix-caching --kv-transfer-config setting publisher to zmq, endpoint to tcp://*:5557). The Dynamo worker runtime subscribes to this local event source, normalizes the events, and republishes them onto the Dynamo event plane. Both the Frontend route and the EPP path use this Dynamo-managed KV-cache state for routing decisions. The article specifically notes that Dynamo can also be configured to predict KV-cache state from the location where requests are sent, but prediction cannot see evictions. Consequently, the router’s view may drift from the cache actually held by the worker, and routing quality will degrade under memory pressure. AICR enables real-time events by default, so both paths route based on the KV-cache state reported by the workers.

Notable limitations: This method is bound to the specific version combination of Dynamo 1.4.2 and AICR v1.0.0, with testing limited to Amazon EKS paired with GB200 NVL72 and GB300 NVL72 systems. Different Kubernetes distributions (such as self-built clusters, GKE, AKS) or different hardware require manual adaptation of component versions and placement flags in the recipe. Additionally, Dynamo 1.5 previews replacing the current Go/CGO bridge implementation with a native Rust EPP. When this happens, the component architecture for the GAIE/EPP path will change, and configurations for Agent Gateway and EPP in the current recipe may need updating.

Among the three authors, Stefan Schimanski is a Principal Engineer at NVIDIA and serves as the Technical Lead for Kubernetes SIG API Machinery, having previously worked on kcp and API server projects at Red Hat; Yuan Chen is a Principal Software Engineer at NVIDIA, responsible for maintaining the open-source AICR project, having previously built hyperscale Kubernetes infrastructure at Apple; Jamie Li is a Senior Technical Marketing Engineer at NVIDIA, focusing on AI inference technology.

Sources

  • developer.nvidia.comNVIDIA Technical Blog↗
Continue researching →