MaxText Streaming DiLoCo: Fragmenting Models for Step-by-Step Synchronization to Eliminate Network Wait in Cross-Datacenter LLM Training
Google engineers explain the core mechanism of Streaming DiLoCo: dividing model parameters into fragments by layer, synchronizing only one fragment per step rather than the entire model, and replacing periodic large traffic bursts with uniform small flows to maintain near line-speed training throughput on narrowband WAN links. The article provides roofline bandwidth calculations, self-test comparisons for Qwen3-32B at 10/30 Gbps, and convergence verification on MoE architectures.
Traditional large-scale model training relies on a tightly coupled accelerator cluster where all devices synchronize gradients at every step via high-bandwidth interconnects. When physical and cost constraints make it difficult to continue scaling a single cluster, deploying compute slices across geographically distributed data centers is the natural next step. However, standard data parallelism degrades severely over wide-area networks (WAN): every step requires transmitting the gradients for the entire model, and because WAN bandwidth is far lower than internal data center interconnects, transmission cannot be fully overlapped with computation, causing training to stall while waiting.
DiLoCo (Distributed Low-Communication), proposed by teams from Google DeepMind and Google Research, focuses on reducing synchronization frequency. It divides training into M independent learner groups, each running H steps of local optimization (typically using AdamW) on its own data without communicating between groups. After every H steps, each group calculates weight deltas, averages them via an all-reduce operation, and updates global weights using an outer optimizer (SGD with Nesterov momentum). This reduces communication volume by a factor of 1/H, with papers reporting convergence quality comparable to standard data parallelism.
However, Vanilla DiLoCo has a lingering issue: although total communication is reduced by a factor of H, each synchronization event still transmits the weight deltas for the entire model. On narrowband links, this creates large traffic spikes once every H steps. Taking Qwen3-32B (approximately 32 billion parameters, bfloat16) as an example, the full model is about 65.5 GB. On a 10 Gbps link, transferring 65.5 GB takes over 50 seconds, whereas a normal calculation step takes about 6 seconds—resulting in a 50-second pause every 65 steps.