Real-Time Interaction Benchmark: Frontier Models Achieve Only 36% Pass Rate Under Time Pressure
Realtime-Bench tests the real-time decision-making capabilities of frontier large language models using 50 dynamic tasks. Under Wall Clock conditions simulating real-world inference latency, the best pass rate is only 36%; after removing delay (Paused Clock), high-reasoning models reach success rates of 70% to 80%. The paper notes that increased reasoning effort yields mixed results under time pressure, reflecting a trade-off between decision quality and timely execution.
Jack Clark, author of the Import AI newsletter, introduced Realtime-Bench in his latest issue—a benchmark suite specifically designed to test the real-time interaction capabilities of large language models. Developed by researchers from the University of Texas at Austin, Ohio State University, the University of Illinois at Urbana-Champaign, NVIDIA, Northwestern University, and Steadyworks, the benchmark comprises 50 dynamic tasks: 30 web-based tasks (covering travel, e-commerce, services, and security) and 20 game tasks (arcade, puzzle, strategy, and simulation). A common feature across all tasks is that delayed actions directly lead to failure—for example, even if reasoning is correct, it is futile once a ticket purchasing window has closed.
The core findings stem from a comparison of three clock conditions. The Wall Clock condition advances the environment based on actual inference latency, simulating real-world deployment scenarios; the Paused Clock halts all delays, providing an ideal upper bound for model capability; and the Token Clock advances simulated time based on the number of generated tokens, isolating physical service speed from the model's temporal perception. Under Wall Clock conditions, OpenAI’s GPT-6 Astra (in high-reasoning mode) achieved the highest pass rate of 36%. In contrast, under Paused Clock conditions, high-reasoning models approached success rates of nearly 70% to 80%. The arXiv paper abstract further distinguishes two metrics: the best success rate under Wall Clock is 44%, while the best Pass^3 rate is 36%.