Google Releases Open-Source TPU Sync, Claims Over 90% Reduction in Inference Latency

U.S.-based technology company Google announced TPU Sync on November 9, 2026, an open-source system designed to address a growing bottleneck in AI inference: moving data between different types of memory as context windows expand. Written in C++, the framework facilitates efficient data transfer between accelerator memory, host DRAM, storage, and the network.

TPU Sync utilizes zero-copy transfers and native PjRtBuffer hardware descriptors to avoid unnecessary copies. Google claims the system can cut KV-cache transfer latency and time-to-first-token (TTFT) by more than 90% in some tests. Specific benchmarks show that for the 235-billion-parameter Qwen3 model, KV cache movement time dropped from several seconds to about 230 milliseconds. For a larger 480-billion-parameter model, the reduction reached 90%. Additionally, combined host-to-device and device-to-host throughput hit 223 GB/s per Ironwood TPU host, approaching the theoretical PCIe Gen5 limit of 256 GB/s.

This release complements Google's previous announcements, such as TorchTPU (enabling native PyTorch execution on TPUs) and llm-d (a distributed inference framework). TPU Sync focuses on low-level memory management, allowing developers using popular inference engines like vLLM and SGLang to handle disaggregated workloads without rebuilding their entire serving stack. Analysis firm SemiAnalysis noted that external TPU serving previously lacked the infrastructure to effectively separate prefill and decode workloads, a gap TPU Sync aims to fill.

However, Google warns that the software is not yet production-ready, with APIs still changing, and recommends sticking to verified commits or tagged releases. Support is currently limited to Cloud TPU v7x and v6e, with specific JAX and PyTorch version requirements. While the performance gains are significant, these figures are vendor-self-tested and have not been independently verified.