読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · llm-d推理框架重大1 分で読めるDisconfirmAI 編集部

Open-source AI inference framework llm-d releases v1.0: tests claim 5-10x cost reduction vs commercial APIs under specific loads

The open-source project llm-d, founded by Red Hat and IBM, released version 1.0 featuring cache-aware routing to boost GPU utilization. Tests indicate inference costs are five to ten times lower than commercial APIs for specific high-concurrency coding agent workloads.

2026-10-23 16:00 GMT+8

The open-source project llm-d, founded by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, released version 1.0 on October 23, 2026, aiming to tackle resource contention in large-scale AI inference. Now governed by the Cloud Native Computing Foundation (CNCF), the project introduces techniques such as cache-aware routing, memory offloading, and separating input prompt processing from response generation to improve GPU utilization and reduce running costs.

According to eeNews Europe, in a benchmark test targeting GLM 5.2, an open-weight model with approximately 753 billion parameters, researchers used llm-d on NVIDIA GPUs to achieve peak throughput of more than 6.6 million output tokens per minute while supporting up to 3,000 concurrent coding agents without preemptions. Based on current cloud rental rates, the researchers estimated that inference costs per token were five to ten times lower than comparable commercial API pricing for the tested workloads. Another study at IBM Research reported that using llm-d for quantum error correction hardware investigation resulted in inference costs 11.5 times lower than an entirely proprietary configuration.

It is important to note that these cost savings figures come from specific workloads and configurations rather than general performance guarantees. Version 1.0 also introduces multi-tenant quality of service, session-aware routing, and support for updates without planned downtime, with over 500 contributors from more than 25 organizations currently involved in development.

ソース

ログイン / 新規登録
読書リサーチレーダー投資フレームワーク
  • eenewseurope.comeeNews Europe↗
トピック

llm-d推理框架

【背景】llm-d是由Google、Red Hat、IBM等联合发起的开源分布式推理框架,旨在解决大规模AI模型自托管部署中的效率与运维难题。2026年10月20日,该项目正式发布1.0版本,标志着其从技术验证阶段进入具备生产级运维能力的成…

3 ステップ3 件更新日 2026-10-26 18:06

経緯

  1. 2026-10-20llm-d项目贡献者在PyTorch Conference North America前夕宣布发布llm-d 1.0版本,强调其具备多租户隔离、零停机升级等生产级运维能力。research.ibm.com ↗
  2. 2026-10-21Google官方博客详细介绍llm-d 1.0的技术架构,展示其在Google Cloud TPU/GPU上的集成情况,以及缓存感知路由如何将中位首token时间(TTFT)降低超过85%。cloud.google.com ↗
  3. 2026-10-23eeNews Europe报道指出,测试显示llm-d在特定高并发代码智能体负载下,推理成本比商业API低五到十倍,凸显其经济优势。eenewseurope.com ↗

未解決の問題

  • llm-d 1.0在非Google Cloud环境下的长期稳定性表现如何?
  • 对于中小型企业而言,部署和维护llm-d所需的技术门槛是否显著低于直接使用商业API?
  • 随着更多组织加入CNCF Sandbox项目,社区对llm-d未来路线图的影响力将如何演变?

このスレッド内の他の記事

  • Google, Red Hat and IBM Release llm-d 1.0: An Open Source Distributed Inference Framework for Agent Workloads2026-10-21
  • llm-d Releases Version 1.0: Open-Source AI Inference Framework Achieves Production Readiness2026-10-20
トピック全文を読む →
リサーチを続ける →