Meta Details Prometheus Cluster Architecture: Over 11% GPUs Held in Reserve, Training Interruptions Cut by 99%

US tech giant Meta disclosed operational details of its "Prometheus" supercomputing cluster located in New Albany, Ohio, on October 20, 2026. The cluster, designed to support large-scale artificial intelligence model training, is expanding toward a scale of 1 million GPUs. Meta stated that to maintain training continuity, the cluster incurs significant redundancy costs, with more than 11% of GPUs currently idle, either held as hot spares or awaiting replacement due to failure.

In terms of hardware configuration, Prometheus utilizes Nvidia GB200/GB300 systems, where each NVLink domain contains 72 GPUs, but only 64 are actively used for training, while the remaining 8 serve as immediate hot spares. Meta explained that a single GPU failure could force an entire rack out of the cluster; the hot spare mechanism prevents such cascading effects, thereby reducing downtime. Additionally, the cluster employs distributed checkpointing, which asynchronously saves model states to multi-tier storage when larger failures occur, preventing the need to restart training from scratch.

Meta acknowledged that signal integrity and power management posed severe challenges as the cluster scaled. With over 600,000 NVLink connections within the system, even minor signal degradation could trigger recovery processes. To address this, Meta introduced continuous telemetry and L11 diagnostic tests, and used software called Dynamo Dimmer to gradually adjust GPU power consumption to handle power spikes. After a year of engineering optimization, training interruptions in Prometheus decreased by 99% compared to mid-2025, dropping from nearly 1,000 per day per 8,000 GPUs to about 11. Meta aims to further reduce the interruption rate to approximately two per day per 8,000 GPUs.