ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsTopic · NBER AI复现论文1 min read·2026-10-03

AI produces 36 science manuscripts in three months, expert review still essential

A Harvard team used an open-source harness to run Claude across 18 fields; output was large but every result needed expert verification.

ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick

2026-10-01 22:02 GMT+8

Harvard physicist Matthew Schwartz used the open-source BootLoops harness to drive Claude, producing 36 manuscripts with 19 co-authors in three months, spanning 18 fields from particle physics to linguistics.

Concrete results include 30 elliptic integrals computed by Claude, fifteen of them for the first time, and a solution to a 20-year-old equation in neutral biodiversity theory that had been impossible to compute at scale; applied to data, it showed tree species composition on Barro Colorado Island in Panama changing 4.5 times faster than the theory allows.

In his guest post on Anthropic's site, Schwartz warns the model declares victory early, automated checks are unreliable, and conclusions can be wrong even when calculations are correct — each result became scientifically valuable only after domain experts stepped in. The manuscripts have not been peer reviewed.

Sources

  • 原始出处anthropic.com↗
Topic

NBER AI复现论文

NBER工作论文W35782揭示,大语言模型(LLM)在批量复现经济学研究时,对近八成论文标记出结果出入。Matthew Schwartz、Isaiah Andrews与Jesse M. Shapiro开发的开源工作流,利用已发表的复现包对…

Sign in / Sign up
ReadingResearchRadarInvestment framework

3 steps2 piecesUpdated 2026-10-06 23:11

How it got here

  1. 2026-09-29NBER发布工作论文W35782,介绍一种能复现、改进和扩展已发表经济学研究的LLM开源工作流。
  2. 2026-10-01经济学家Tyler Cowen在Marginal Revolution撰文,详细披露该研究数据:4,452个复现包中3,460篇被标记结果出入。
  3. 2026-10-03Anthropic发布博客,作者Matthew Schwartz阐述“Claude-shaped science”理念,解释为何LLM在特定定量科学任务上高效但需专家把关。

What is still unsettled

  • 被LLM标记为“结果出入”的3,460篇论文中,有多少比例经人工复核确认为真实错误?
  • 该开源工作流在非经济学领域的通用性及误报率如何?
  • 学术界是否会建立针对LLM辅助复现的标准验证协议?

Also in this thread

  • LLM reproduces economics papers at scale, with discrepancies flagged in nearly 80%2026-10-03
Read the full topic →
Continue researching →