SenseTime AI Infrastructure CTO Xuan Shanming: Agentic Inference Competition Hinges on Architectural Efficiency, Not Chip Count
Xuan Shanming, Vice President and CTO of SenseTime's Large Device AI infrastructure platform, presented the architectural approach for Agentic-era inference at the CSDI Summit in Shenzhen, China: layering inference by hardware requirements — compute-intensive Prefill on domestic Chinese chips, bandwidth-intensive Decode on high-end GPUs (PD separation); further exploring cross-datacenter KV Cache transfer for cross-domain PD separation, and splitting Attention and FFN within Decode onto different hardware (AF separation). Xuan claimed 5x throughput improvement under specific test conditions and projected 150%-200% single-card throughput gains from AF separation, but these figures are vendor self-reported with undisclosed test configurations.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
Agentic applications have pushed token consumption to thousand-fold levels while high-end chip supply cannot keep pace. Xuan Shanming, Vice President and CTO of SenseTime's Large Device platform, outlined the response at the October 16 CSDI Summit in Shenzhen, China: rather than simply stacking more compute, the approach decomposes the inference process by hardware requirements, letting different chips handle what they do best.
Large model inference has two phases: Prefill processes input tokens and is compute-intensive; Decode generates output tokens step by step, requiring continuous reads from context cache with high memory bandwidth demands. PD separation decouples these two steps — using compatible domestic Chinese compute for Prefill, high-bandwidth high-end GPUs for Decode, connected via KV Cache transfer. This isn't simply running different chips together; it's reorganizing compute resources around their characteristics. In some heterogeneous configurations, 30 to 50 mid-to-low-tier Prefill nodes may be needed to match one Decode node's processing capacity. Once a Decode node fails, large amounts of Prefill resources sit idle, necessitating dynamic failover and role-switching mechanisms.