LectureRechercheRadarCadre d'investissement
Connexion / Inscription
← Retour à la lecture
ActualitésSujet · AWSMatériel3 min de lectureRédaction DisconfirmAI

AWS Proposes Three Code-Level Checks to Detect GPU Silent Data Corruption Bypassing Hardware Monitoring

Silent data corruption returns erroneous but plausible values without triggering any hardware alerts. ECC, Xid, DCGM, and NVLink checksums each guard different boundaries and cannot capture errors in the compute path. AWS embeds three checks within the PyTorch DistributedDataParallel (DDP) training loop, covering forward/backward passes, optimizer steps, and weight storage as failure points. The measured overhead is approximately 7%–10%, with an upper bound on the false-positive rate of about one in five thousand.

2026-11-12 07:30 GMT+8

The fundamental reason silent data corruption (SDC) is difficult to detect is that it occurs internally within the compute path. A GPU's multiply-accumulate unit returns a wrong but plausible value—for example, 4.0001 instead of 4.0000—without triggering any error signal. Every existing integrity check guards a different boundary: ECC protects memory, Xid captures driver and hardware faults, DCGM deep health checks run on idle nodes, and NVLink and PCIe checksums verify whether transmission bytes arrived intact. Checkpoint checksums confirm storage round-trip consistency. None of these checks whether the output of the arithmetic logic path is correct, because there is no ECC or parity on that path.

This means detection must come from the training task itself. AWS adds three checks to the PyTorch DistributedDataParallel training loop, arranged by where faults are likely to land. The first is a non-finite gradient check: before gradient aggregation, GPUs are flagged if their local gradient norm is NaN or Inf; after aggregation, the reduced gradients are checked for NaN or Inf. The second is a ring replay audit: before the optimizer step, each GPU recomputes a micro-batch from another node's GPU, comparing loss values bit-for-bit and, under deterministic kernels, comparing gradient fingerprints. This can catch finite but incorrect computed values. The third is weight fingerprint comparison: after the optimizer step, fingerprints of all replicas' weights are compared, catching corruption in the update step or in stored weights.

Connexion / Inscription
LectureRechercheRadarCadre d'investissement

There is strict logic governing the execution order and timing of the three checks. Faults in the compute path must be caught before the optimizer step, because once erroneous gradients propagate through aggregation to all replicas, all weights remain consistent, making it impossible to distinguish afterwards. Conversely, faults in the optimizer step can only be seen after it occurs. Ring replay is feasible because between the backward pass and the optimizer step, all replicas hold identical weights, allowing any GPU to recompute another GPU's micro-batch and expect identical bits.

In tests training a 953-million-parameter model on 32 H100 GPUs, running only the gradient hooks and NaN/Inf checks (excluding audits) increased step time by approximately 7.4%; running deterministic audits every 100 steps added about 10.1%; running audits every 10 steps added about 34.7%. Across 15,320 fault-free audit steps and 3,080 weight checks, no alarms were triggered, yielding an upper bound on the false-positive rate of approximately one in five thousand (at 95% confidence) using the rule of three.

This solution has clear boundaries. It was tested only on DDP layouts; tensor parallelism, pipeline parallelism, and FSDP sharding require separate adaptation. One-time finite faults that happen to fall on non-audit steps will not be captured by any check. Deterministic audits use math attention kernels, whose cost grows with the square of sequence length, meaning overhead in long-sequence training may far exceed the measurements in this article. AWS explicitly states that synthetic tests do not establish precision in production environments and recommends retaining manual approval before replacing nodes.

From a practical deployment perspective, this scheme links detection, localization, and recovery into a closed loop. Once a check triggers an alarm in the training loop, the alarm carries a suspect_window recording the range of steps since that check last passed, indicating the fault was active during one of those steps. On SageMaker HyperPod clusters orchestrated by Amazon Elastic Kubernetes Service (Amazon EKS), Kubernetes node names contain instance IDs, so mapping a suspect GPU's rank to a specific instance requires only a single lookup. Operations personnel can use this to stop the task, isolate the node, check DCGM metrics, and upon confirmation, complete hardware replacement via node health labels or the BatchReplaceClusterNodes interface. The training script then resumes from a checkpoint saved before the suspect_window.

Sources

  • aws.amazon.comAmazon Web Services↗
Sujet

AWS

亚马逊云科技(AWS)针对大规模分布式训练中难以察觉的“静默数据损坏”(SDC)问题,在 Amazon SageMaker HyperPod 平台上推出了一套基于代码层的检测方案。SDC 指硬件返回错误但合理的数值且不触发任何告警,导致 E…

2 étapes1 piècesMis à jour 2026-11-17 01:25

Comment on en est arrivé là

  1. 2026-11-12AWS 发布技术博客,介绍在 SageMaker HyperPod 上通过三项代码级检查检测 SDC 的方案,包括非有限梯度检查、环形重放审计和权重指纹比对。aws.amazon.com ↗
  2. 2026-11-12第三方媒体 AINewsTrend 报道该方案,指出其要求至少三个 GPU 节点,并在 4 台 ml.p5.48xlarge 实例(32 H100 GPUs)集群上完成测试。ainewstrend.com ↗

Ce qui reste non résolu

  • 该方案对非 PyTorch 框架(如 JAX/TPU)或其他云环境的适用性如何?
  • 在实际生产环境中,频繁替换节点对训练任务连续性和成本的具体影响量化数据尚缺。
  • 除了 AWS 提出的软件层检查,硬件厂商(如 NVIDIA)是否有计划提供原生支持以捕获计算路径中的 SDC?
Lire le sujet complet →
Continuer la recherche →