LeituraPesquisaRadarFramework de investimento
Entrar / Cadastrar
← Voltar para a leitura
NotíciasTópico · NBER AI复现论文Material1 min de leituraEditorial DisconfirmAI

LLM reproduces economics papers at scale, with discrepancies flagged in nearly 80%

Economists used an LLM to reproduce economics research in bulk, flagging discrepancies in 3,460 of 4,452 replication packages; flagged results are automated detections, not proof of error.

Verificado 2026-10-03 10:51 GMT+8

Readers can now reproduce published economics research in bulk with a large language model: of 4,452 published replication packages, 3,460 were flagged for discrepancies, while calculations in 496 papers were sped up by more than a factor of 10 and 923 received extensions consistent with the original aims.

Previously, such reproduction relied on manual, paper-by-paper checks, which were costly and limited in coverage.

The workflow was built by economists Matthew Schwartz, Isaiah Andrews and Jesse M. Shapiro. It first reproduces the original calculations, then checks them against published findings and runs sensitivity analysis; the flagged counts and speedups are author-reported.

The flagged "discrepancies" are the workflow's automated detections, not proof the original papers were wrong; the paper has not been peer reviewed, is NBER working paper w35782, was covered by Tyler Cowen on Marginal Revolution on October 1, and has no third-party replication yet.

Fontes

  • nber.org↗
Entrar / Cadastrar
LeituraPesquisaRadarFramework de investimento
marginalrevolution.com
↗
Tópico

NBER AI复现论文

NBER工作论文W35782揭示,大语言模型(LLM)在批量复现经济学研究时,对近八成论文标记出结果出入。Matthew Schwartz、Isaiah Andrews与Jesse M. Shapiro开发的开源工作流,利用已发表的复现包对…

3 etapas2 peçasAtualizado 2026-10-06 23:11

Como chegamos aqui

  1. 2026-09-29NBER发布工作论文W35782,介绍一种能复现、改进和扩展已发表经济学研究的LLM开源工作流。wesearch.press ↗
  2. 2026-10-01经济学家Tyler Cowen在Marginal Revolution撰文,详细披露该研究数据:4,452个复现包中3,460篇被标记结果出入。marginalrevolution.com ↗
  3. 2026-10-03Anthropic发布博客,作者Matthew Schwartz阐述“Claude-shaped science”理念,解释为何LLM在特定定量科学任务上高效但需专家把关。anthropic.com ↗

O que ainda está indefinido

  • 被LLM标记为“结果出入”的3,460篇论文中,有多少比例经人工复核确认为真实错误?
  • 该开源工作流在非经济学领域的通用性及误报率如何?
  • 学术界是否会建立针对LLM辅助复现的标准验证协议?

Também nesta thread

  • AI produces 36 science manuscripts in three months, expert review still essential2026-10-03
Ler o tópico completo →
Continuar pesquisando →