Streaming DiLoCo
3 steps1 pieces3 Sources
Google's MaxText team has detailed Streaming DiLoCo, a distributed training method that fragments model parameters to eliminate network stalls in cross-datacenter LLM training. Traditional data parallelism requires high-bandwidth, low-latency links because all devices must synchronize gradients at every step. When scaling across geographically separated datacenters over Wide Area Networks (WAN), this synchronization causes severe compute idle time.
How it works:
- DiLoCo Foundation: Introduced by Google DeepMind, DiLoCo allows independent "learners" (compute slices) to perform many local optimization steps (H steps) before synchronizing. This reduces communication frequency by a factor of H.
- The Bottleneck: Vanilla DiLoCo still transmits the entire model's weight deltas at once during synchronization. On narrow WAN links, this creates large periodic traffic spikes and blocking stalls.
- Streaming Solution: Streaming DiLoCo partitions the model into P disjoint fragments (typically one per layer plus embeddings). Instead of sending the whole model every H steps, it synchronizes one fragment every Δh steps. This spreads the same total communication volume across all steps, creating a uniform, low-bandwidth stream rather than a massive burst.
Performance Evidence: In tests with Qwen3-32B on two TPU v5p-32 slices throttled to 10 Gbps:
- Standard Data Parallelism: Step time rose to 55.11 seconds.
- Vanilla DiLoCo: Step time was 7.55 seconds.
- Streaming DiLoCo: Step time remained at 5.76 seconds, matching the unthrottled baseline speed. Traffic between groups dropped from 4,258.8 GB to 67 GB for the same training span. The method is available in MaxText v0.2.5+.
How it got here
2023-11-13: Google DeepMind introduces DiLoCo, reducing communication by 500x while maintaining convergence quality.