ReadingResearchRadarInvestment frameworkAI Company Primary Sources
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

NVIDIA Open-Sources Distributed AI Workload Failure Attribution Method: Separating Root Causes from Cascading Symptoms

NVIDIA has open-sourced its internal distributed AI workload failure attribution system, named Attribution. The core approach treats the workload rather than individual nodes as the unit of analysis, using deterministic rules to separate cascading cross-rank errors triggered by a single hardware fault into root causes and supporting symptoms. The article uses a synthetic example of NVLink signal integrity issues causing multi-node NCCL errors to demonstrate the four-stage attribution process and its dependency boundaries regarding telemetry completeness.

2026-11-03 07:39 GMT+8

In large-scale distributed AI workloads, a hard fault in a single GPU or an NVLink link often leaves traces not just on one node. It can cause collective NCCL communication failures across other ranks within the same workload, trigger state transitions recorded by the scheduler, and prompt monitoring systems to report network metric anomalies. Operations teams face error messages scattered across dozens or even hundreds of nodes, while the actual issue requiring repair might be just a single port on one machine. NVIDIA is open-sourcing its long-used internal failure attribution system under the name NVIDIA Attribution, aiming to transform this troubleshooting process from manual, per-node log reading into reproducible, rule-based diagnostics.

The core design decision of Attribution is workload-centricity rather than node-centricity. It does not merely answer what is wrong with this specific GPU, but answers why this entire workload failed. The scope of analysis is determined by scheduler identifiers or explicit execution windows and node allocations, after which observations related to that workload are collected from application logs, system logs, GPU driver messages, Fabric Manager outputs, scheduler history, and node network metrics.

The attribution process consists of four stages. Stage one defines the workload boundary. Stage two collects observations. Stage three involves the attribution engine extracting events and evaluating their timing and interaction relationships through deterministic rules, separating downstream symptoms from the initial event that caused the failure. Stage four returns structured results, including the workload terminal state, attribution domain, root cause node, list of supporting symptoms, and collector coverage records.

Sign in / Sign up
ReadingResearchRadarInvestment frameworkAI Company Primary Sources

The article provides a synthetic example to demonstrate this workflow. In a multi-rank training workload, application logs contain three distinct NCCL errors, while the system log on node-02 contains XID 149, with its error word decoded as an NVLink signal integrity issue. During the extraction phase, raw logs are converted into typed events with source provenance. In the filtering stage, an EARLY rule named HandleXidBurstEvents reads the XID burst evidence, decodes the error words, and emits an attribution event for signal integrity issues on node-02, leaving the three NCCL events unchanged. In the final refinement stage, the Xid149AndNvlinkFiltering rule retains the signal integrity symptom while demoting symptoms from other nodes. The Gb200PriorityRootCause rule further reduces the active symptom set, resulting in a hardware failure attribution mapping. The NCCL errors are not deleted but preserved as interesting evidence, remaining available for inspection.

A key boundary of this method lies in telemetry quality. The article explicitly states that when multiple causes fit the existing evidence, the attribution domain is marked as ambiguous; diagnostic quality depends on the completeness of underlying telemetry, and the system does not force incomplete evidence into definitive conclusions. Production-scale data—analyzing millions of workloads weekly, covering hundreds of thousands of GPUs, and identifying failure patterns around specific host and switch ports in large RDMA clusters—are self-reported by NVIDIA, with no independent reproduction observed.

Architecturally, the system comprises two deployable components: the Attribution Service and the Attribution Blueprint. The Service is an HTTP wrapper around the core attribution engine, receiving standardized observations and returning a single verdict. The Blueprint is a Python reference orchestrator responsible for discovering workloads, parsing execution context, running collectors, calling the Service, enriching verdict results, and writing output. The Blueprint includes built-in execution context connectors for Slurm, Kueue, and static inputs, as well as log and metric connectors for Loki, local files, Kubernetes pod logs, and PromQL metrics. Rules are checked into the code repository in declarative configuration and Python formats, released alongside the Service, and are not dynamically uploaded via HTTP API.

For infrastructure teams, the significance of this open-source release lies in the possibility of encoding failure patterns once and reusing them across workloads and clusters. Developers can add rules for new failure modes, implement connectors for their own schedulers and telemetry backends, and route attribution results into incident management, recovery, and fleet management systems. Downstream systems consume the terminal_state and sub_domain fields, eliminating the need to parse product-specific log formats.

Sources

  • developer.nvidia.comNVIDIA Technical Blog↗
Continue researching →