Open-source AI inference framework llm-d releases v1.0: tests claim 5-10x cost reduction vs commercial APIs under specific loads

The open-source project llm-d, founded by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, released version 1.0 on October 23, 2026, aiming to tackle resource contention in large-scale AI inference. Now governed by the Cloud Native Computing Foundation (CNCF), the project introduces techniques such as cache-aware routing, memory offloading, and separating input prompt processing from response generation to improve GPU utilization and reduce running costs.

According to eeNews Europe, in a benchmark test targeting GLM 5.2, an open-weight model with approximately 753 billion parameters, researchers used llm-d on NVIDIA GPUs to achieve peak throughput of more than 6.6 million output tokens per minute while supporting up to 3,000 concurrent coding agents without preemptions. Based on current cloud rental rates, the researchers estimated that inference costs per token were five to ten times lower than comparable commercial API pricing for the tested workloads. Another study at IBM Research reported that using llm-d for quantum error correction hardware investigation resulted in inference costs 11.5 times lower than an entirely proprietary configuration.

It is important to note that these cost savings figures come from specific workloads and configurations rather than general performance guarantees. Version 1.0 also introduces multi-tenant quality of service, session-aware routing, and support for updates without planned downtime, with over 500 contributors from more than 25 organizations currently involved in development.