ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
TopicDisconfirmAI EditorialUpdated 2026-11-22 16:54

Streaming DiLoCo

3 steps1 pieces3 Sources

Google's MaxText team has detailed Streaming DiLoCo, a distributed training method that fragments model parameters to eliminate network stalls in cross-datacenter LLM training. Traditional data parallelism requires high-bandwidth, low-latency links because all devices must synchronize gradients at every step. When scaling across geographically separated datacenters over Wide Area Networks (WAN), this synchronization causes severe compute idle time.

How it works:

  1. DiLoCo Foundation: Introduced by Google DeepMind, DiLoCo allows independent "learners" (compute slices) to perform many local optimization steps (H steps) before synchronizing. This reduces communication frequency by a factor of H.
  2. The Bottleneck: Vanilla DiLoCo still transmits the entire model's weight deltas at once during synchronization. On narrow WAN links, this creates large periodic traffic spikes and blocking stalls.
  3. Streaming Solution: Streaming DiLoCo partitions the model into P disjoint fragments (typically one per layer plus embeddings). Instead of sending the whole model every H steps, it synchronizes one fragment every Δh steps. This spreads the same total communication volume across all steps, creating a uniform, low-bandwidth stream rather than a massive burst.

Performance Evidence: In tests with Qwen3-32B on two TPU v5p-32 slices throttled to 10 Gbps:

  • Standard Data Parallelism: Step time rose to 55.11 seconds.
  • Vanilla DiLoCo: Step time was 7.55 seconds.
  • Streaming DiLoCo: Step time remained at 5.76 seconds, matching the unthrottled baseline speed. Traffic between groups dropped from 4,258.8 GB to 67 GB for the same training span. The method is available in MaxText v0.2.5+.

How it got here

  1. 2023-11-13

    2023-11-13: Google DeepMind introduces DiLoCo, reducing communication by 500x while maintaining convergence quality.

Sign in / Sign up
ReadingResearchRadarInvestment framework
arxiv.org ↗
  • 2025-01-30

    2025-01-30: 'Streaming DiLoCo' paper published, detailing parameter fragmentation and overlapping communication to reduce peak bandwidth.

    arxiv.org ↗
  • 2026-11-19

    2026-11-19: Google Developers Blog explains the implementation in MaxText, providing roofline calculations and Qwen3-32B benchmarks.

    developers.googleblog.com ↗
  • What is still unsettled

    • How does Streaming DiLoCo scale to models larger than 32B parameters in real-world multi-region deployments?
    • What are the specific trade-offs in memory usage for the outer optimizer state compared to standard FSDP?

    Sources

    • developers.googleblog.comGoogle Developers Blog↗
    • arxiv.orgarXiv: Streaming DiLoCo Paper↗
    • arxiv.orgarXiv: DiLoCo Original Paper↗

    Coverage in this topic

    • MaxText Streaming DiLoCo: Fragmenting Models for Step-by-Step Synchronization to Eliminate Network Wait in Cross-Datacenter LLM Training2026-11-20News

      Google engineers explain the core mechanism of Streaming DiLoCo: dividing model parameters into fragments by layer, synchronizing only one fragment per step rather than the entire model, and replacing periodic large traffic bursts with uniform small flows to maintain near line-speed training throughput on narrowband WAN links. The article provides roofline bandwidth calculations, self-test comparisons for Qwen3-32B at 10/30 Gbps, and convergence verification on MoE architectures.