NVIDIA Engineers Xianglong Lu and Bo Chu Discuss Agentic AI Inference Architecture on Kubernetes

NVIDIA engineers Xianglong Lu and Bo Chu published a video on the PyTorch official channel sharing production lessons from running Agentic AI inference on Kubernetes. As Agentic AI redefines inference workloads, long tool-call workflows create sustained KV-cache pressure; parallel subagents need shared prefixes and consistent routing; and sessions must survive pod/GPU failures without losing context.

The talk highlights the use of NVIDIA Dynamo for cache-aware routing and disaggregated serving, and RoleBasedGroup (RBG) for stateful continuity, discovery, and role/unit coordination across pipeline stages. The speakers compared deployment patterns (Dynamo-native vs RBG), discussed what broke under cache contention and partial failures, and explained how they modeled routing, session continuity, recovery, and observability in Kubernetes-native ways.