Silicon Valley 101 Interviews Alibaba International President Zhang Kuo: Why High-Scoring Models Have Only ~60% Pass Rate in Real Business Tasks?
On October 8, 2026, Silicon Valley 101 released Episode 255, featuring a dialogue with Zhang Kuo, President of Alibaba International Station. The episode addresses the industry pain point of 'stronger models but weaker user perception' by sharing insights from Accio Work's deployment in e-commerce and daily operations. Zhang revealed that despite near-perfect scores on general benchmarks, frontier models achieve only about a 61% task completion rate without human intervention when tested against 107 real-world business tasks. The discussion explores the fundamental differences between coding and business agents, proposes the formula 'Agent = Model × Harness × Context', and examines the survival logic of vertical SaaS in the AI era, cost optimization via multi-model routing, and the irreplaceable role of human judgment in final business decisions.
BedeutungWesentlichBeweisE2 nicht repliziertAufbereitungSchnell
Silicon Valley 101 released Episode 255, titled 'Models Are Getting Stronger, So Why Don't Users Feel It? Revisiting Alibaba International Station President Zhang Kuo,' on October 8, 2026. Hosted by Yiwen, the episode features an in-depth conversation with Zhang Kuo, President of Alibaba International Station, focusing on the challenges and opportunities of deploying AI Agents from general capabilities to vertical business scenarios. Following a previous interview half a year ago about the Accio Work product, this revisit highlights real-world cases of merchants using the tool over the past six months, aiming to answer a core question: How much of the improvement in large model capabilities has actually translated into progress in merchants' work?
In the episode, Zhang Kuo shares a critical observation: although many large models have achieved near-perfect scores on general evaluation leaderboards, his team has not felt a corresponding level of capability improvement in actual business operations. To quantify this gap, the Alibaba International Station team distilled 107 specific tasks from millions of real operational data points on Alibaba.com and Accio Q&A logs. These tasks cover high-frequency commercial activities such as comparing supplier quotes, identifying phishing emails, booking shipping schedules, and handling transaction disputes. According to Zhang, even the most frontier models achieve a task pass rate of only around 61% under conditions of zero human intervention in this specific open-source benchmark. This data intuitively reveals the huge chasm between 'leaderboard scores' and 'actual delivery,' indicating that long-horizon, open-ended business tasks remain a shortcoming of current AI capabilities.