arXiv Preprint: System1-Arena Benchmark Shows Large System-1 Models Match Fast Closed LLMs in Non-Agentic Decisions at Quarter Latency

The arXiv preprint "System1-Arena: Which Decisions Can Be Left to System 1?" introduces System1-Arena, a new benchmark designed to evaluate large language models (LLMs) on discrete decision tasks and distinguish which decisions can be handled by System-1 models returning probabilities versus System-2 methods generating reasoning steps. The benchmark contains 92k questions from 268 public sources across four categories: text classification, applied decisions, evaluation/control, and agent/action, plus 138 closed-loop environments.

The team evaluated 30 models, including 7 closed LLMs, 13 open LLMs, and 10 System-1 models. Results show that the largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. In contrast, reasoning LLMs are the most accurate but cost about 70x more than fast LLMs. In closed loops, large open LLMs succeed most often (up to 32.9%), while small System-1 models rarely succeed.