読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · llm-d推理框架重大2 分で読めるDisconfirmAI 編集部

Google, Red Hat and IBM Release llm-d 1.0: An Open Source Distributed Inference Framework for Agent Workloads

llm-d 1.0 launches with multi-tenant QoS and zero-downtime upgrades, reporting a 2.9x throughput increase on large-scale clusters.

2026-10-20

At the PyTorch Conference North America on October 20, 2026, llm-d, an open-source project co-maintained by Google, Red Hat, and IBM, released its first stable version, llm-d 1.0. This Kubernetes-native distributed inference serving stack is designed to turn model servers like vLLM or SGLang into a coordinated pool, specifically addressing load balancing challenges in agent and reinforcement learning (RL) workloads.

As AI applications shift from simple single-turn chats to multi-step agentic tasks, traditional round-robin load balancing often leads to redundant computation and memory waste. llm-d 1.0 introduces cache-aware routing, tiered KV cache offloading, and prefill/decode disaggregation. The new release focuses on production-grade operational capabilities, including multi-tenant quality of service (QoS) control, zero-downtime upgrades, and agent-aware session routing.

According to benchmark data published by Google and IBM, llm-d demonstrates significant performance improvements in specific scenarios. In a 17,000-node multi-region Google Kubernetes Engine (GKE) deployment, throughput increased by 2.9 times compared to standard methods. On Ironwood TPUs running coding agent workloads, median time-to-first-token (TTFT) dropped by more than 85%, while throughput rose by 2.4 times. Additionally, by introducing an async inference processor, the system can handle background jobs during idle cycles of real-time traffic, raising accelerator duty cycles from approximately 40% to over 70%.

It is important to note that these performance metrics are self-reported by the project maintainers and have not yet been fully reproduced by independent third parties. Currently operating as a CNCF Sandbox project under the Apache 2.0 license, llm-d supports various hardware platforms, including NVIDIA, AMD, and Intel GPUs, as well as Google TPUs. For enterprises seeking to build their own large-scale AI inference infrastructure, this framework offers a technical path to reduce dependency on single proprietary cloud services.

ログイン / 新規登録
読書リサーチレーダー投資フレームワーク

ソース

  • cloud.google.com↗
  • research.ibm.com↗
トピック

llm-d推理框架

【背景】llm-d是由Google、Red Hat、IBM等联合发起的开源分布式推理框架,旨在解决大规模AI模型自托管部署中的效率与运维难题。2026年10月20日,该项目正式发布1.0版本,标志着其从技术验证阶段进入具备生产级运维能力的成…

3 ステップ3 件更新日 2026-10-26 18:06

経緯

  1. 2026-10-20llm-d项目贡献者在PyTorch Conference North America前夕宣布发布llm-d 1.0版本,强调其具备多租户隔离、零停机升级等生产级运维能力。research.ibm.com ↗
  2. 2026-10-21Google官方博客详细介绍llm-d 1.0的技术架构,展示其在Google Cloud TPU/GPU上的集成情况,以及缓存感知路由如何将中位首token时间(TTFT)降低超过85%。cloud.google.com ↗
  3. 2026-10-23eeNews Europe报道指出,测试显示llm-d在特定高并发代码智能体负载下,推理成本比商业API低五到十倍,凸显其经济优势。eenewseurope.com ↗

未解決の問題

  • llm-d 1.0在非Google Cloud环境下的长期稳定性表现如何?
  • 对于中小型企业而言,部署和维护llm-d所需的技术门槛是否显著低于直接使用商业API?
  • 随着更多组织加入CNCF Sandbox项目,社区对llm-d未来路线图的影响力将如何演变?

このスレッド内の他の記事

  • Open-source AI inference framework llm-d releases v1.0: tests claim 5-10x cost reduction vs commercial APIs under specific loads2026-10-23
  • llm-d Releases Version 1.0: Open-Source AI Inference Framework Achieves Production Readiness2026-10-20
トピック全文を読む →
リサーチを続ける →