ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

AWS Proposes Three Code-Level Checks to Detect GPU Silent Data Corruption Bypassing Hardware Monitoring

Silent data corruption returns erroneous but plausible values without triggering any hardware alerts. ECC, Xid, DCGM, and NVLink checksums each guard different boundaries and cannot capture errors in the compute path. AWS embeds three checks within the PyTorch DistributedDataParallel (DDP) training loop, covering forward/backward passes, optimizer steps, and weight storage as failure points. The measured overhead is approximately 7%–10%, with an upper bound on the false-positive rate of about one in five thousand.

2026-11-12 07:30 GMT+8

The fundamental reason silent data corruption (SDC) is difficult to detect is that it occurs internally within the compute path. A GPU's multiply-accumulate unit returns a wrong but plausible value—for example, 4.0001 instead of 4.0000—without triggering any error signal. Every existing integrity check guards a different boundary: ECC protects memory, Xid captures driver and hardware faults, DCGM deep health checks run on idle nodes, and NVLink and PCIe checksums verify whether transmission bytes arrived intact. Checkpoint checksums confirm storage round-trip consistency. None of these checks whether the output of the arithmetic logic path is correct, because there is no ECC or parity on that path.

This means detection must come from the training task itself. AWS adds three checks to the PyTorch DistributedDataParallel training loop, arranged by where faults are likely to land. The first is a non-finite gradient check: before gradient aggregation, GPUs are flagged if their local gradient norm is NaN or Inf; after aggregation, the reduced gradients are checked for NaN or Inf. The second is a ring replay audit: before the optimizer step, each GPU recomputes a micro-batch from another node's GPU, comparing loss values bit-for-bit and, under deterministic kernels, comparing gradient fingerprints. This can catch finite but incorrect computed values. The third is weight fingerprint comparison: after the optimizer step, fingerprints of all replicas' weights are compared, catching corruption in the update step or in stored weights.

Sign in / Sign up
ReadingResearchRadarInvestment framework

There is strict logic governing the execution order and timing of the three checks. Faults in the compute path must be caught before the optimizer step, because once erroneous gradients propagate through aggregation to all replicas, all weights remain consistent, making it impossible to distinguish afterwards. Conversely, faults in the optimizer step can only be seen after it occurs. Ring replay is feasible because between the backward pass and the optimizer step, all replicas hold identical weights, allowing any GPU to recompute another GPU's micro-batch and expect identical bits.

In tests training a 953-million-parameter model on 32 H100 GPUs, running only the gradient hooks and NaN/Inf checks (excluding audits) increased step time by approximately 7.4%; running deterministic audits every 100 steps added about 10.1%; running audits every 10 steps added about 34.7%. Across 15,320 fault-free audit steps and 3,080 weight checks, no alarms were triggered, yielding an upper bound on the false-positive rate of approximately one in five thousand (at 95% confidence) using the rule of three.

This solution has clear boundaries. It was tested only on DDP layouts; tensor parallelism, pipeline parallelism, and FSDP sharding require separate adaptation. One-time finite faults that happen to fall on non-audit steps will not be captured by any check. Deterministic audits use math attention kernels, whose cost grows with the square of sequence length, meaning overhead in long-sequence training may far exceed the measurements in this article. AWS explicitly states that synthetic tests do not establish precision in production environments and recommends retaining manual approval before replacing nodes.

From a practical deployment perspective, this scheme links detection, localization, and recovery into a closed loop. Once a check triggers an alarm in the training loop, the alarm carries a suspect_window recording the range of steps since that check last passed, indicating the fault was active during one of those steps. On SageMaker HyperPod clusters orchestrated by Amazon Elastic Kubernetes Service (Amazon EKS), Kubernetes node names contain instance IDs, so mapping a suspect GPU's rank to a specific instance requires only a single lookup. Operations personnel can use this to stop the task, isolate the node, check DCGM metrics, and upon confirmation, complete hardware replacement via node health labels or the BatchReplaceClusterNodes interface. The training script then resumes from a checkpoint saved before the suspect_window.

Sources

  • aws.amazon.comAmazon Web Services↗
Continue researching →