ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

NVIDIA Internal Practice: Five Key Lessons for AI Agent Operations of GPU Clusters

NVIDIA deployed a multi-agent operations system on its DGX SuperPOD production clusters, using domain-specific AI agents to replace the repetitive labor of Site Reliability Engineers (SREs) troubleshooting failures layer by layer. The article distills five design lessons from actual operation: clear agent boundaries, read-only before automation, cross-layer correlation, precise tool definitions, and transparent reasoning.

2026-11-05 06:00 GMT+8

As GPU cluster scale expands, operations span multiple layers including compute, network, hardware, and scheduling. When training task performance degrades, SREs often need to open different dashboards one by one to check scheduler status, GPU temperature, network latency, and application logs separately before locating the root cause. Kavya G Y, a product manager at NVIDIA, and Rohan Adwankar, an engineer responsible for agentic systems and observability, recorded in a technical blog how they used a set of AI agents to solve this repetitive labor problem.

This system is called the AI Factory Operations Agent, running on NVIDIA's internal DGX H100, DGX B200, and DGX GB300 production clusters. The core idea is not to use a single all-powerful agent to replace SREs, but to let a group of domain-specific agents perform their respective duties, communicating with each other through the Model Context Protocol (MCP). SREs can ask any agent questions in natural language, switch investigation directions, and every step of the operation has audit records.

NVIDIA deployed six domain agents. The Job Recovery Agent detects training job failures and goodput degradation, performing multi-layer root cause analysis across the scheduler, application, hardware, and network layers, and recommending recovery steps once the root cause is confirmed. The Hardware Agent triages hardware faults at the device level—PCIe errors, ECC errors, GPU issues, NVMe failures, and thermal anomalies—tracing symptoms down to specific components. The Observability Agent queries historical Prometheus metrics and Grafana Loki logs, dynamically generating dashboards that are displayed directly in the web interface. The Research Agent retrieves pre-loaded NVIDIA Mission Control manuals and DGX SuperPOD reference architecture documents, and also supports users uploading their own runbooks. The Kubernetes Agent provides read-only cluster state awareness, covering pods, services, deployments, GPU resources, node status, and scheduling status. The Cluster Management Agent interfaces with the NVIDIA Base Command Manager API through diagnostic MCP tools for cluster management advice and troubleshooting.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The center of the architecture is the user-side agent runtime provided by the NemoClaw blueprint, which receives SRE queries and routes them to the appropriate domain agents. OpenShell is responsible for governance and control. Each shell execution takes place in an independent Kubernetes sandbox rather than running inside the main gateway pod; the sandbox lifecycle does not modify existing services or production workloads. Security is layered: Kubernetes checks invoke a fixed version of the kubectl binary via the server-side run_kubectl tool using registered read-only credentials, and OpenShell itself does not receive kubectl or Kubernetes credentials. RBAC explicitly limits the scope of resources agents can view—the article shows actual permission verification results: it can get pods and configmaps, but cannot get secrets, create pods, or update pods.

From production usage, NVIDIA distilled five lessons. First, clear agent boundaries: overlapping responsibility ownership creates noise rather than signal, so each agent is only responsible for its own domain. Second, read-only first: accurate investigation capabilities build SRE trust before automation, allowing them to see what the agent can and cannot do first. Third, cross-layer correlation: the framework connects workload, hardware, and network signals to find root causes missed by isolated tools—this is exactly the core pain point of multi-dashboard troubleshooting. Fourth, precise MCP tool definitions: clear tool descriptions improve routing and reduce wrong tool selection. Fifth, transparent reasoning: SREs need to see tool calls, evidence, and conclusions before taking action.

NVIDIA continues to refine this framework by measuring root cause analysis accuracy, tracking SRE agreement, and expanding the agent ecosystem. For large-scale AI factories, the recommended deployment model is to place agent orchestration and reasoning on a dedicated GPU-accelerated control plane isolated from the production cluster, while the production cluster only provides telemetry data, does not consume training GPUs, and does not interfere with workloads. Internal deployments run model inference on dedicated GPU capacity, using existing control-plane CPU Kubernetes admin space to host runtime services. Smaller scales can use RTX PRO Workstations or DGX Spark.

In multi-tenant scenarios, scoped agents, tools, sandboxes, and Kubernetes permissions control the cluster data, tools, and nodes accessible to each tenant. Tenant SREs can only see their own namespaces, jobs, and nodes, while central operations retain broader cluster visibility. Each tenant can use its own domain agents and dedicated inference hardware. OpenShell enforces these boundaries by running each tenant's agents and tools in isolated sandboxes with kernel-level restrictions.

This orchestration framework is currently released as experimental open source software on GitHub; APIs, configuration, and deployment requirements may change between versions. NVIDIA customers can contact their account managers to obtain internally developed cluster operations agents.

Sources

  • developer.nvidia.comNVIDIA Technical Blog↗
Continue researching →