독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스중대1 분 소요DisconfirmAI 편집부

arXiv Preprint: System1-Arena Benchmark Shows Large System-1 Models Match Fast Closed LLMs in Non-Agentic Decisions at Quarter Latency

New benchmark System1-Arena evaluates 30 models on discrete decisions, finding large System-1 models match fast closed LLMs in accuracy with lower latency.

2026-10-16 12:00 GMT+8

The arXiv preprint "System1-Arena: Which Decisions Can Be Left to System 1?" introduces System1-Arena, a new benchmark designed to evaluate large language models (LLMs) on discrete decision tasks and distinguish which decisions can be handled by System-1 models returning probabilities versus System-2 methods generating reasoning steps. The benchmark contains 92k questions from 268 public sources across four categories: text classification, applied decisions, evaluation/control, and agent/action, plus 138 closed-loop environments.

The team evaluated 30 models, including 7 closed LLMs, 13 open LLMs, and 10 System-1 models. Results show that the largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. In contrast, reasoning LLMs are the most accurate but cost about 70x more than fast LLMs. In closed loops, large open LLMs succeed most often (up to 32.9%), while small System-1 models rarely succeed.

출처

  • arxiv.org↗
리서치 계속하기 →
로그인 / 회원가입
독서리서치레이더투자 프레임워크