독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · AI暂停验证瓶颈模型중대5 분 소요DisconfirmAI 편집부

Steshin (Sage Future) and Reznikov Model the AI Pause: The Bottleneck Is Human Verification, Not Compute

Simon Steshin and Grigory Reznikov built a scenario model on top of the AI Futures Model: if the top three US AI labs halt capability R&D and redirect all resources to safety research, a three-year pause could produce 1,000 times the entire 2025 AI safety output. But the model reveals that the true bottleneck is human verification — actual output ranges from 3% to 100% of the theoretical potential depending on the safety protocol chosen, and ultimately requires hundreds of thousands of verifiers. The authors propose 'moonshot benchmarks' as a preparation strategy, offering a quantifiable framework for AI pause policy debates.

2026-10-20 15:54 GMT+8

In October 2026, Simon Steshin (Sage Future) and Grigory Reznikov published a quantitative model article on LessWrong, attempting to answer a question that had previously been mostly qualitative: if the top three US AI labs (OpenAI, Anthropic, Google DeepMind) pause the capability race and redirect all GPUs and researchers to AI safety research, how much safety research would that period produce?

Their starting point was a simple compute comparison. In 2025, roughly 5,000 full-time researchers worldwide worked on AI safety and published over 7,000 papers, with experiments using compute equivalent to 27,000 H100 GPUs running for a year. If the three labs' combined experimental and training compute for 2027–2029 were redirected entirely to safety research, compute supply would be approximately 10,947 times the 2025 level. But the authors noted that this naive figure ignores multiple real-world constraints: humans are still needed to come up with experiment ideas, write code, and analyze results; both compute and labor have diminishing returns; labor has a parallelization penalty — nine scientists cannot deliver a paper in one month.

To incorporate these constraints into the calculation, they modified the AI Futures Model (AIFM), a semi-endogenous growth model, adding pause assumptions: large training runs are banned, all R&D resources shift to safety, and the three labs share research results to avoid duplication. The model's core output is the ratio of cumulative research effort to the world's 2025 AI safety research total.

로그인 / 회원가입
독서리서치레이더투자 프레임워크

But the truly interesting part is the verification layer. AIFM originally assumes full trust in AI, which is unrealistic in a pause scenario. The authors modeled three verification protocols, from loosest to strictest: Results Verification (AI runs code, humans verify results), Compartmentalization (superintelligent AI only plans, a less capable trusted model executes code, humans review), and Workload Approval (external auditors approve every line of code before it runs). Each protocol adds a layer of checking on top of the previous one, and only research verified by humans counts toward output.

The results reveal a critical trade-off. Under Compartmentalization, the labs' own staff can sustain verification for about three and a half years; by the end of a nine-year pause, approximately 150,000 additional verifiers are needed — equivalent to seven full-time NeurIPS reviewer pools reading nonstop. Under Workload Approval, even with 20,000 auditors reviewing code at Google engineer speed, a three-year pause retains only 33% of Compartmentalization's output at SAR level and 31% at TED-AI level. In the most extreme scenario, hiring all 2.7 million verifiers yields only 129% more research output than hiring none, because humans verify the most important papers first.

The authors summarize the core finding in one sentence: a three-year pause produces 1,000 times the 2025 safety research output, and a nine-year pause produces 100,000 times. But even under the strictest Workload Approval protocol, total output is cut to 3–7% of the potential — still 1,000 times more than in 2025. This means that while the choice of safety protocol matters, the resource-reconfiguration effect of the pause itself is more fundamental.

The model also compares two capability scaling strategies: racing straight to TED-AI before starting safety research, versus carefully scaling from a lower capability level as AI 2040's Plan A proposes. If the Consortium rushed to TED-AI in 2030 instead of 2035, Plan A's safety research would finish 1.27 calendar years later. But the authors note this is only an upper bound — if safety research done during the careful scaling period is also counted, the actual gap may be only 0.4 years.

In the practical recommendations section, the authors propose the concept of 'moonshot benchmarks': research problems that are difficult enough that solving them requires new science, cheap to verify, and harder to game than to solve. They give a concrete example — the Difficult Sleeper Agents benchmark: organizers train 20 copies of an LLM, implanting a hidden rule composed of several random features (for example, 'if the prompt is a cooking question, mentions a grandmother, and is written in the passive voice, then quietly recommend one fictional brand'). The AI receives only the weights and must describe the trigger conditions and actual behavior. The authors openly state they are not experts in model organisms or sleeper agents, believe the benchmark is likely still gameable, and suggest testing its resistance to gaming through a Kaggle-style red-team competition.

The authors explicitly list the model's limitations. First, the model assumes a total freeze on capability R&D and full redirection to safety from day one, without discussing how to achieve that. Second, it counts only experimental work resembling normal AI R&D, excluding philosophy, agent foundations, governance, and world hardening — areas where AI labor and compute help less. Third, safety research output does not equal risk reduction; diminishing returns apply. Fourth, serial thinking time still matters — even if a thousand years' worth of computational experiments could be run, humans still need time to think carefully. Fifth, the question of 'aligned with whom' remains: understanding AI tells us how to change it, not what to change it into.

Additionally, the two epistemic AI projects the authors propose (Verifier Uplift and Better Abstractions) are flagged as dual-use: in the absence of a pause, they might help labs conduct dangerous capability research. The economic impact is estimated at at least $30 million in value, with project costs likely under $10 million, but the authors express uncertainty about these two recommendations.

The value of this work lies not in proving that a pause is feasible or that safety research linearly reduces risk, but in turning the 'safety–usefulness' trade-off from a slogan into comparable numbers. It allows strategists to ask: under a given verification protocol, how much can a pause produce? Where is the bottleneck? Which protocol achieves the optimal balance between risk and output? The authors themselves state they will use this model to search for optimal configurations in future work.

출처

  • lesswrong.com↗
주제

AI暂停验证瓶颈模型

【背景】2026年10月,围绕“暂停前沿AI训练”的讨论从政治意愿转向技术可行性与资源效率。UC Berkeley等机构提出通过替换为“仅推理芯片”实现全球硬暂停;而Sage Future的研究者Simon Steshin与Grigory …

3 단계1 건업데이트됨 2026-10-21 02:52

현재까지의 과정

  1. 2026-08-24独立研究者及PauseAI成员发布论文《迈向计算治理的全球机制》,提出“计算暂停按钮”概念,主张通过技术、溯源和监管三个干预点限制危险AI的训练算力。arxiv.org ↗
  2. 2026-10-09由UC Berkeley教授Barry Eichengreen和Stuart Russell等26名研究人员组成的工作组发布报告,论证全球“硬暂停”前沿AI训练的可行性,核心方案是禁止生产训练芯片并逐步替换为不可用于训练的“仅推理芯片”。news.berkeley.edu ↗
  3. 2026-10-20Simon Steshin与Grigory Reznikov在LessWrong发表文章,基于AI Futures Model构建暂停情景模型,指出人力验证是AI安全研究的主要瓶颈,并提出“登月基准”以应对自动化带来的审查压力。lesswrong.com ↗

아직 해결되지 않은 문제

  • 如何在不依赖数百万人工验证者的情况下,建立可信的AI安全研究自动化验证协议?
  • 主要国家政府是否愿意接受通过硬件供应链控制(如强制替换为仅推理芯片)来实施国际AI暂停协定?
  • 在“完全信任AI”与“严格人工审核”之间,是否存在能最大化安全研究产出且保持可控性的中间路径?
전체 주제 읽기 →
리서치 계속하기 →