ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

MaxText Streaming DiLoCo: Fragmenting Models for Step-by-Step Synchronization to Eliminate Network Wait in Cross-Datacenter LLM Training

Google engineers explain the core mechanism of Streaming DiLoCo: dividing model parameters into fragments by layer, synchronizing only one fragment per step rather than the entire model, and replacing periodic large traffic bursts with uniform small flows to maintain near line-speed training throughput on narrowband WAN links. The article provides roofline bandwidth calculations, self-test comparisons for Qwen3-32B at 10/30 Gbps, and convergence verification on MoE architectures.

2026-11-19

Traditional large-scale model training relies on a tightly coupled accelerator cluster where all devices synchronize gradients at every step via high-bandwidth interconnects. When physical and cost constraints make it difficult to continue scaling a single cluster, deploying compute slices across geographically distributed data centers is the natural next step. However, standard data parallelism degrades severely over wide-area networks (WAN): every step requires transmitting the gradients for the entire model, and because WAN bandwidth is far lower than internal data center interconnects, transmission cannot be fully overlapped with computation, causing training to stall while waiting.

DiLoCo (Distributed Low-Communication), proposed by teams from Google DeepMind and Google Research, focuses on reducing synchronization frequency. It divides training into M independent learner groups, each running H steps of local optimization (typically using AdamW) on its own data without communicating between groups. After every H steps, each group calculates weight deltas, averages them via an all-reduce operation, and updates global weights using an outer optimizer (SGD with Nesterov momentum). This reduces communication volume by a factor of 1/H, with papers reporting convergence quality comparable to standard data parallelism.

However, Vanilla DiLoCo has a lingering issue: although total communication is reduced by a factor of H, each synchronization event still transmits the weight deltas for the entire model. On narrowband links, this creates large traffic spikes once every H steps. Taking Qwen3-32B (approximately 32 billion parameters, bfloat16) as an example, the full model is about 65.5 GB. On a 10 Gbps link, transferring 65.5 GB takes over 50 seconds, whereas a normal calculation step takes about 6 seconds—resulting in a 50-second pause every 65 steps.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The solution offered by Streaming DiLoCo is to cut model parameters into P disjoint fragments, synchronizing exactly one fragment per step within the H-step window. When P = H, each step synchronizes 1/P of the model, presenting the network with a nearly uniform stream of small traffic rather than periodic large spikes. MaxText's fragmentation method sets P = N_layers + 1: each Transformer decoder layer constitutes one fragment, while all non-scanned parameters (token embeddings, final layer normalization, and output head) form another fragment.

There is an easily overlooked engineering detail here: the size of the embedding fragment depends on the vocabulary dimension and can be much larger than that of a single-layer fragment. If unaddressed, the step synchronizing this fragment would still generate a traffic spike. MaxText addresses this by evenly splitting and merging large matrices from the embedding fragment into other layer fragments, ensuring all fragments are roughly equal in size and truly achieving a uniform flow.

The synchronization order also matters. MaxText does not synchronize adjacent layers sequentially but instead interleaves fragment selection (e.g., synchronizing fragment 0, then layer k, then layer 2k...). This ensures that every round of synchronization touches different depths of the network, preventing any section of the model from remaining stale for too long while other sections update.

Roofline modeling allows for precise calculation of bandwidth requirements. Standard data parallelism transfers 2|Θ|b bytes per step (where |Θ| is the parameter count and b is bytes per parameter), requiring completion within the backward propagation time t_bwd, resulting in a peak bandwidth requirement of approximately 2|Θ|b / t_bwd. For Qwen3-32B under bfloat16, this is about 85 Gbps. Streaming DiLoCo transfers only 1/P of the model per step, lowering the bandwidth requirement to approximately 85/P Gbps. When P = 65, this drops to about 1.3 Gbps—more than sufficient for a 10 Gbps link.

The Google team simulated three network conditions on two TPU v5p-32 slices using Linux tc traffic shaping. At 30 Gbps, data parallelism degraded to 19.41 seconds/step, while Streaming DiLoCo maintained 5.76 seconds/step (a 3.37x speedup). At 10 Gbps, data parallelism degraded to 55.11 seconds/step, whereas Streaming DiLoCo remained at 5.76 seconds/step (a 9.57x speedup). Vanilla DiLoCo at 10 Gbps ran at 7.55 seconds/step—much better than data parallelism but still suffering from periodic pauses every 65 steps.

Regarding convergence quality, the team ran 4,000 steps of pre-training on two MoE architectures (Qwen3-30B-A3B and Gemma4-26B-A4B). After a brief adjustment period in the first ~800 steps, the loss curve for Streaming DiLoCo smoothly tracked and slightly outperformed the synchronous data parallelism baseline. Stable convergence relies on two design choices: the outer loop uses Nesterov momentum (β=0.9, learning rate 0.3–0.7) rather than simple averaging; and the inner loop's AdamW first- and second-moment buffers remain unchanged after fragment synchronization, ensuring learners do not lose optimizer state.

It must be emphasized that all the above performance figures come from self-tests on Google's proprietary TPU clusters, using Linux tc to simulate network constraints rather than real cross-domain deployments. The 9.57x speedup appears in the extremely constrained scenario of 10 Gbps. Actual production network conditions, topologies, and traffic patterns may differ. However, its core mechanism—breaking large synchronizations into uniform small flows to match narrowband links—is a transferable distributed systems pattern that does not depend on a single high-bandwidth cluster.

Sources

  • developers.googleblog.com↗
Continue researching →