Meta Engineers T. Rice & A. Afzali Detail Fault Tolerance for 100k-GPU Clusters: Open-Source Tools torchcomms and MCCL Keep Training Alive
Meta's infrastructure team shared their practices for fault-tolerant large-scale GPU training on the PyTorch official channel. The talk introduces fault tolerance APIs in the open-source library torchcomms, highlighting MCCL, a collective communication library used in Meta's production environment. It also covers using new NCCL primitives (ncclShrink, ncclGrow) to dynamically adjust communicator membership without deadlocking, and fast recovery mechanisms using torchstore for checkpointing to peer memory or local disk.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
T. Rice and A. Afzali from Meta's infrastructure team published a talk on the PyTorch official channel regarding fault tolerance in large-scale GPU training. This resource addresses the engineering challenge of maintaining training continuity when hardware faults become daily occurrences in clusters spanning tens of thousands of GPUs.
According to the publisher's information, the core of the talk introduces fault tolerance APIs in the open-source project torchcomms, covering NCCL, MCCL, and Gloo backends. MCCL is described as Meta's production fault-tolerance collective communication library. The talk delves into its design and fault tolerance capabilities.
The content also demonstrates how torchcomms uses new NCCL primitives—ncclShrink, ncclGrow, and ncclCommRevoke—to gracefully shrink or grow communicator membership without deadlocking the cluster. Additionally, it covers novel checkpointing approaches, such as using torchstore to save checkpoints to peer memory or local disk, leveraging fast RDMA transfers between GPU peers to replace slow distributed filesystem writes and enable live recovery.