Google, Red Hat and IBM Release llm-d 1.0: An Open Source Distributed Inference Framework for Agent Workloads

At the PyTorch Conference North America on October 20, 2026, llm-d, an open-source project co-maintained by Google, Red Hat, and IBM, released its first stable version, llm-d 1.0. This Kubernetes-native distributed inference serving stack is designed to turn model servers like vLLM or SGLang into a coordinated pool, specifically addressing load balancing challenges in agent and reinforcement learning (RL) workloads.

As AI applications shift from simple single-turn chats to multi-step agentic tasks, traditional round-robin load balancing often leads to redundant computation and memory waste. llm-d 1.0 introduces cache-aware routing, tiered KV cache offloading, and prefill/decode disaggregation. The new release focuses on production-grade operational capabilities, including multi-tenant quality of service (QoS) control, zero-downtime upgrades, and agent-aware session routing.

According to benchmark data published by Google and IBM, llm-d demonstrates significant performance improvements in specific scenarios. In a 17,000-node multi-region Google Kubernetes Engine (GKE) deployment, throughput increased by 2.9 times compared to standard methods. On Ironwood TPUs running coding agent workloads, median time-to-first-token (TTFT) dropped by more than 85%, while throughput rose by 2.4 times. Additionally, by introducing an async inference processor, the system can handle background jobs during idle cycles of real-time traffic, raising accelerator duty cycles from approximately 40% to over 70%.