ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Stanford HAI Postdoctoral Fellow Jiacheng Miao Proposes: Whether a Paper Can Become an Executable Agent Is Itself a Reproducibility Metric

Jiacheng Miao (Stanford Institute for Human-Centered AI) introduced the Paper2Agent framework at a Snorkel AI Reading Group session: packaging papers and their codebases as MCP servers, enabling agents to learn to execute paper methods through a build-and-test loop. The team converted 100 computational biology papers into 74 usable agents, achieving a score of 91.2% on 300 benchmark questions, compared to 80.3% for Claude Code directly accessing the code repositories (both using Sonnet 4). Miao’s core argument is that whether a paper can be successfully executed by an agent serves as a new metric for reproducibility.

Verified 2026-11-20 04:14 GMT+8

Traditional research papers are passive, fragmented carriers of knowledge. Jiacheng Miao used AlphaGenome as an example: a roughly 200-page PDF where key details are hidden in supplementary materials; reproducing results requires figuring out the code repository and Python environment from scratch, while applying the method requires understanding the training data. He stated that this is currently how scientific knowledge is communicated.

His proposal is that papers should not only be published as PDFs but also released alongside an interactive agent. The specific approach involves packaging the paper and its codebase as an MCP server, which includes three types of components—tools that wrap computational methods, resources that expose links to the manuscript and code, and step-by-step instructions written according to the paper’s analytical chain. Users ask natural language questions, and the agent calls upon the paper’s methods, writes analysis code, and returns results.

Reliability comes from a multi-agent build-and-test loop: one set of agents identifies the codebase, installs dependencies, and determines applicable tasks; the team extracts question-and-answer pairs from the paper and repository as benchmarks; another set of testing agents repeatedly runs these Q&As, iteratively hardening the server until it can reproduce all results reported in the paper and every notebook in the repository.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Of 100 computational biology papers, 74 were successfully converted into usable agents. On 300 benchmark questions, Paper2Agent scored 91.2%, while Claude Code directly accessing GitHub repositories scored 80.3% (both using Sonnet 4). Miao noted that zero-shot coding agents sometimes misunderstand paper content and write incorrect code, whereas paper agents trained through the loop complete tasks faster, and he estimates they are also less costly.

However, Miao considers the more interesting inference to be the reproducibility metric. He said that judging whether a paper is reproducible has always been difficult—some papers take other researchers three months to replicate. If a paper is very hard to convert into a robust agent, it may lack key information and is not yet something people can use, only something they can read. This turns the success or failure of agent execution into a proxy indicator for paper completeness.

This metric has clear boundaries. Among the papers that failed conversion, many had incomplete codebases, poor documentation, or unresolved errors, making it impossible for the team to reliably expose them as working tools. The benchmarks were run by the team themselves, and Miao acknowledged that when using newer models, the gap between paper agents and Claude Code is narrowing, though it has not yet disappeared.

The Paper2Agent paper was published in Nature, the code is open-sourced on GitHub, and hosted agents are available for free use at paper2agent.ai. Miao envisions a future where every paper comes with an agent availability statement, similar to today’s data availability statements.

Sources

  • snorkel.aiSnorkel AI↗
Continue researching →