TimeWarp Benchmark Reveals Web Agent Vulnerability to UI Changes; Self-Reported Gains Up to 37.7% for Qwen-3
New benchmark shows vision-based web agents struggle with UI evolution; proposed TimeTraj method boosts Qwen-3 success rates by up to 37.7% in self-tests.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
The TimeWarp benchmark reveals that current web agents are vulnerable to changes in web interfaces across different eras, particularly affecting vision-based agents, while text-based agents become brittle when fine-tuned on a single version.
Proposed by the Spark Lab team at the University of Utah, TimeWarp aims to evaluate whether agents perform well on 'tomorrow's web.' It includes Wiki, News, and Shop environments, each with six UI versions spanning different internet eras.
To address this, the authors propose TimeTraj, an annotation method using plan distillation to collect trajectories across multiple versions. Training with this method improved Qwen-3 4B success rates by 20.4% to 37.7%, and Llama-3.1 8B by 0% to 27.0%.
These results are from first-party benchmarks (E2) and have not yet been independently reproduced. Code and data are open-sourced on GitHub and Hugging Face for further validation of generalization capabilities.