독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · 谷歌TPU Sync중대2 분 소요DisconfirmAI 편집부

Google Releases Open-Source TPU Sync, Claims Over 90% Reduction in Inference Latency

Google launches TPU Sync to accelerate AI inference by optimizing data transfer between memory tiers, reporting significant reductions in time-to-first-token in self-tests.

2026-11-10 02:40 GMT+8

U.S.-based technology company Google announced TPU Sync on November 9, 2026, an open-source system designed to address a growing bottleneck in AI inference: moving data between different types of memory as context windows expand. Written in C++, the framework facilitates efficient data transfer between accelerator memory, host DRAM, storage, and the network.

TPU Sync utilizes zero-copy transfers and native PjRtBuffer hardware descriptors to avoid unnecessary copies. Google claims the system can cut KV-cache transfer latency and time-to-first-token (TTFT) by more than 90% in some tests. Specific benchmarks show that for the 235-billion-parameter Qwen3 model, KV cache movement time dropped from several seconds to about 230 milliseconds. For a larger 480-billion-parameter model, the reduction reached 90%. Additionally, combined host-to-device and device-to-host throughput hit 223 GB/s per Ironwood TPU host, approaching the theoretical PCIe Gen5 limit of 256 GB/s.

This release complements Google's previous announcements, such as TorchTPU (enabling native PyTorch execution on TPUs) and llm-d (a distributed inference framework). TPU Sync focuses on low-level memory management, allowing developers using popular inference engines like vLLM and SGLang to handle disaggregated workloads without rebuilding their entire serving stack. Analysis firm SemiAnalysis noted that external TPU serving previously lacked the infrastructure to effectively separate prefill and decode workloads, a gap TPU Sync aims to fill.

However, Google warns that the software is not yet production-ready, with APIs still changing, and recommends sticking to verified commits or tagged releases. Support is currently limited to Cloud TPU v7x and v6e, with specific JAX and PyTorch version requirements. While the performance gains are significant, these figures are vendor-self-tested and have not been independently verified.

로그인 / 회원가입
독서리서치레이더투자 프레임워크

출처

  • thenewstack.ioThe New Stack↗
주제

谷歌TPU Sync

谷歌于2026年11月发布开源软件TPU Sync,旨在解决大模型推理中因上下文窗口扩大而导致的内存数据传输瓶颈。该工具通过优化键值缓存(KV Cache)在加速器内存、主机DRAM及网络间的传输效率,宣称在特定测试场景下可将首字生成时间(…

3 단계1 건업데이트됨 2026-11-10 23:27

현재까지의 과정

  1. 2026-11-09谷歌宣布第八代TPU组合扩展,包括TPU 8tx和TPU 8i Max,并预告相关软件更新。blog.google ↗
  2. 2026-11-10谷歌正式发布开源软件TPU Sync,用于加速AI推理中的KV缓存传输。cloud.google.com ↗
  3. 2026-11-10媒体报道称TPU Sync在部分测试中可将大模型推理延迟降低超90%,但指出其尚非生产就绪。thenewstack.io ↗

아직 해결되지 않은 문제

  • TPU Sync在实际生产环境中的稳定性如何?谷歌何时会提供稳定的API版本?
  • 与英伟达GB200/GB300系统相比,TPU Sync优化后的TPU集群在综合性价比上是否具有明确优势?
  • 开发者社区对TPU Sync的接受度如何?是否会显著推动更多企业从CUDA生态迁移至TPU?
전체 주제 읽기 →
리서치 계속하기 →