Traditional large-scale model training relies on a tightly coupled accelerator cluster where all devices synchronize gradients at every step via high-bandwidth interconnects. When physical and cost constraints make it difficult to continue scaling a single cluster, deploying compute slices across geographically distributed data centers is the natural next step. However, standard data parallelism degrades severely over wide-area networks (WAN): every step requires transmitting the gradients for the entire model, and because WAN bandwidth is far lower than internal data center interconnects, transmission cannot be fully overlapped with computation, causing training to stall while waiting.
DiLoCo (Distributed Low-Communication), proposed by teams from Google DeepMind and Google Research, focuses on reducing synchronization frequency. It divides training into M independent learner groups, each running H steps of local optimization (typically using AdamW) on its own data without communicating between groups. After every H steps, each group calculates weight deltas, averages them via an all-reduce operation, and updates global weights using an outer optimizer (SGD with Nesterov momentum). This reduces communication volume by a factor of 1/H, with papers reporting convergence quality comparable to standard data parallelism.
However, Vanilla DiLoCo has a lingering issue: although total communication is reduced by a factor of H, each synchronization event still transmits the weight deltas for the entire model. On narrowband links, this creates large traffic spikes once every H steps. Taking Qwen3-32B (approximately 32 billion parameters, bfloat16) as an example, the full model is about 65.5 GB. On a 10 Gbps link, transferring 65.5 GB takes over 50 seconds, whereas a normal calculation step takes about 6 seconds—resulting in a 50-second pause every 65 steps.
The solution offered by Streaming DiLoCo is to cut model parameters into P disjoint fragments, synchronizing exactly one fragment per step within the H-step window. When P = H, each step synchronizes 1/P of the model, presenting the network with a nearly uniform stream of small traffic rather than periodic large spikes. MaxText's fragmentation method sets P = N_layers + 1: each Transformer decoder layer constitutes one fragment, while all non-scanned parameters (token embeddings, final layer normalization, and output head) form another fragment.
There is an easily overlooked engineering detail here: the size of the embedding fragment depends on the vocabulary dimension and can be much larger than that of a single-layer fragment. If unaddressed, the step synchronizing this fragment would still generate a traffic spike. MaxText addresses this by evenly splitting and merging large matrices from the embedding fragment into other layer fragments, ensuring all fragments are roughly equal in size and truly achieving a uniform flow.
The synchronization order also matters. MaxText does not synchronize adjacent layers sequentially but instead interleaves fragment selection (e.g., synchronizing fragment 0, then layer k, then layer 2k...). This ensures that every round of synchronization touches different depths of the network, preventing any section of the model from remaining stale for too long while other sections update.