Horizon Robotics' J. Sun & S. Tian Present Multi-Tenant Data Platform Re-architecture at PyTorch Channel
A video published by the official PyTorch channel features J. Sun and S. Tian from Horizon Robotics discussing the re-architecture of their multi-tenant data platform. The presentation covers strategies for handling 8 million daily tasks in the context of autonomous driving's shift to end-to-end models, focusing on centralized Argo Workflows agents, Informer bottleneck optimization, and elastic quotas using Kueue and Volcano.
ImportanceLocalEvidenceE2 unreplicatedWrite-upQuick
The official PyTorch channel published a technical talk video on September 24, 2026, presented by J. Sun and S. Tian from Horizon Robotics. This resource documents the company's infrastructure re-architecture practices as they faced thousands of concurrent, sub-minute workflows during the transition to end-to-end autonomous driving models.
According to the detailed summary provided by the publisher, the talk focuses on resolving stability issues and resource waste caused by traditional orchestration and "Informer Choke." Key topics include replacing per-workflow agents with a centralized Argo Workflows Agent Service for an "Agentless" approach; overcoming Informer bottlenecks by refactoring metadata, reducing VolcanoJob size by 30%, and cutting sync latency from 4 minutes to 10 seconds; and leveraging Kueue and Volcano to implement elastic quotas and resolve preemption conflicts in multi-tenant environments.
For engineering teams focused on large-scale AI infrastructure efficiency, this video offers specific quantitative references, such as claimed monthly savings of 2.4 million CNY in infrastructure costs and over 1,000 CPU nodes. Readers can use this information to evaluate the applicability of similar architectural solutions in their own business scenarios.