LesenForschungRadarAnlageframework
Anmelden / Registrieren
← Zurück zur Lektüre
NewsThema · 英伟达AICR推理运行时Wesentlich5 Min. LesezeitDisconfirmAI Redaktion

NVIDIA AI Cluster Runtime Separates Dynamo Inference Kubernetes Runtime from Application Deployment into a Five-Step Validated Workflow

NVIDIA AI Cluster Runtime packages Dynamo and its dependent infrastructure—GPU drivers, dynamic resource allocation, gang scheduling, node health, and monitoring—into version-locked recipes. It achieves runtime readiness through a five-step process: snapshotting the cluster, generating the recipe, rendering the deployment bundle, deploying the runtime, and validating the installation, only then allowing the application inference graph to be installed. The article also compares the applicable scenarios for the Dynamo Frontend routing path versus the Gateway API Inference Extension/Endpoint Picker path.

2026-10-31 05:35 GMT+8

NVIDIA Dynamo is a distributed inference framework that coordinates model serving across multiple GPU worker nodes. Deploying it on Kubernetes superficially requires only a DynamoGraphDeployment to specify how vLLM, SGLang, or TensorRT-LLM should run in pods. However, actual production deployment involves much more: the GPU driver stack must match the nodes, GPU nodes need labeling and discovery, the cluster must expose GPUs via Dynamic Resource Allocation, the scheduler must support gang scheduling to meet topology requirements, node health signals must be available before workloads land on degraded nodes, Prometheus must be reachable to collect accelerator and Dynamo metrics, and if the deployment requires Kubernetes-native ingress, the Gateway API and Endpoint Picker paths must also be configured and ready.

The design philosophy of NVIDIA AI Cluster Runtime (AICR) is to decouple these infrastructure dependencies from application deployment. It does not require teams to manually install and debug each component one by one; instead, it packages Dynamo and all its runtime dependencies into a single recipe. The recipe records locked component versions, compatibility constraints, and validation checks, making the deployment reproducible. AICR v1.0.0 locks the following component versions for the Dynamo inference recipe on Amazon EKS: NVIDIA GPU Operator v26.7.1, Node Feature Discovery 0.19.0, cert-manager v1.20.2, AWS EFA v0.5.29, NVSentinel v1.25.0, NVIDIA DRA GPU Driver 0.5.0, KAI Scheduler v0.16.9, Grove v0.1.0-alpha.13, Dynamo Platform 1.4.2, Agent Gateway v1.5.0, and Kube Prometheus Stack 84.4.0. These versions were jointly tested by the AICR team on the specified platforms and hardware.

Anmelden / Registrieren
LesenForschungRadarAnlageframework

The entire process consists of five steps. Step 1: The aicr snapshot command captures the current state of the cluster, automatically targeting nodes labeled nvidia.com/gpu.present=true by default and tolerating GPU node taints. Step 2: The aicr recipe command identifies the Kubernetes distribution, GPU accelerators, and operating system based on the snapshot, and generates a recipe file combining workload intent (inference) and platform (dynamo). Step 3: The aicr bundle command renders the recipe into deployable Helm values and manifests, using node selectors and toleration flags to specify placement policies for GPU worker nodes and system nodes. Step 4: Deploy the runtime: when no GitOps controller is present locally, Helmfile can be used to apply resources in dependency order; in production environments, ArgoCD or Flux GitOps bundles can be rendered. Step 5: The aicr validate command performs two-stage validation (deployment and conformance) on the cluster after installation but before application workloads are deployed. Checks cover cluster baseline resources and GPU allocatable capacity, GPU Operator health and version, DRA support and secure accelerator access, gang scheduling and autoscaling, Prometheus metrics collection, and Gateway API and agentgateway readiness. Only upon successful validation is the DynamoGraphDeployment installed.

At the routing level, Dynamo 1.4.2 supports two Kubernetes paths, and AICR provides recipes and validation for both. Path 1 is the Dynamo Frontend route: the Frontend runs with DYN_ROUTER_MODE=kv, tokenizes each request, scores candidate workers using prefix overlap and current load, selects a worker, and dispatches the request via the TCP request plane. This path has the minimal topology and is suitable for scenarios where an application team owns a model-specific endpoint, or an existing edge proxy only needs to forward traffic to the Dynamo Frontend Service. It is currently the most production-proven serving option. Path 2 is the Gateway API Inference Extension (GAIE) / Endpoint Picker Plugin (EPP) route: agentgateway resolves HTTPRoute to an InferencePool. The Dynamo operator creates this InferencePool from the DynamoGraphDeployment, where its selector identifies candidate worker Pods, and its endpointPickerRef identifies the Dynamo EPP Service. The Gateway invokes the EPP via GAIE’s endpoint selection interface; the EPP uses Dynamo’s KV router to select an endpoint and returns it to the Gateway, which then forwards the request to the selected worker’s Frontend sidecar (running with --router-mode direct, thus no longer rerouting). This path is suitable for platform teams managing shared Kubernetes ingress, where timeouts, retries, and fallback strategies need to be configured via Gateway API mechanisms.

Both paths share the same KV-cache event plane. In Dynamo 1.4.2, each vLLM worker engine emits raw KV-cache events via a local ZMQ publisher (configuration example: --enable-prefix-caching --kv-transfer-config setting publisher to zmq, endpoint to tcp://*:5557). The Dynamo worker runtime subscribes to this local event source, normalizes the events, and republishes them onto the Dynamo event plane. Both the Frontend route and the EPP path use this Dynamo-managed KV-cache state for routing decisions. The article specifically notes that Dynamo can also be configured to predict KV-cache state from the location where requests are sent, but prediction cannot see evictions. Consequently, the router’s view may drift from the cache actually held by the worker, and routing quality will degrade under memory pressure. AICR enables real-time events by default, so both paths route based on the KV-cache state reported by the workers.

Notable limitations: This method is bound to the specific version combination of Dynamo 1.4.2 and AICR v1.0.0, with testing limited to Amazon EKS paired with GB200 NVL72 and GB300 NVL72 systems. Different Kubernetes distributions (such as self-built clusters, GKE, AKS) or different hardware require manual adaptation of component versions and placement flags in the recipe. Additionally, Dynamo 1.5 previews replacing the current Go/CGO bridge implementation with a native Rust EPP. When this happens, the component architecture for the GAIE/EPP path will change, and configurations for Agent Gateway and EPP in the current recipe may need updating.

Among the three authors, Stefan Schimanski is a Principal Engineer at NVIDIA and serves as the Technical Lead for Kubernetes SIG API Machinery, having previously worked on kcp and API server projects at Red Hat; Yuan Chen is a Principal Software Engineer at NVIDIA, responsible for maintaining the open-source AICR project, having previously built hyperscale Kubernetes infrastructure at Apple; Jamie Li is a Senior Technical Marketing Engineer at NVIDIA, focusing on AI inference technology.

Quellen

  • developer.nvidia.comNVIDIA Technical Blog↗
Thema

英伟达AICR推理运行时

英伟达推出AI集群运行时(AICR),旨在解决NVIDIA Dynamo在Kubernetes上部署时基础设施依赖复杂的问题。AICR将GPU驱动、动态资源分配、调度器及监控组件打包为版本锁定的“配方”,通过快照、生成、渲染、部署、验证五步…

3 Schritte1 BeiträgeAktualisiert 2026-11-01 16:42

Wie es dazu kam

  1. 2026-10-30NVIDIA发布技术博客,详细介绍AICR如何通过版本锁定配方管理Dynamo推理运行时,并对比Frontend与GAIE/EPP两种路由方案。developer.nvidia.com ↗
  2. 2026-10-30第三方媒体We Are The Frontier报道AICR v1.1.2发布,强调其自动化四阶段工作流及对H100至GB300架构的支持。wearethefrontier.org ↗
  3. 2026-05-15GitHub Issue #933记录AICR在多安全组EKS集群中,因验证器Pod调度位置不确定导致Prometheus连接超时的Bug。github.com ↗

Was noch ungeklärt ist

  • AICR是否计划引入更严格的Pod Affinity规则以解决多安全组环境下的验证器非确定性问题?
  • 对于不使用Gateway API的传统Ingress场景,Dynamo Frontend路由与GAIE/EPP路径的性能差异是否有公开基准测试数据?
  • AICR配方对非NVIDIA GPU Operator管理的集群(如使用其他驱动栈)的支持程度如何?
Vollständiges Thema lesen →
Recherche fortsetzen →