Stanford HAI Postdoctoral Fellow Jiacheng Miao Proposes: Whether a Paper Can Become an Executable Agent Is Itself a Reproducibility Metric
Jiacheng Miao (Stanford Institute for Human-Centered AI) introduced the Paper2Agent framework at a Snorkel AI Reading Group session: packaging papers and their codebases as MCP servers, enabling agents to learn to execute paper methods through a build-and-test loop. The team converted 100 computational biology papers into 74 usable agents, achieving a score of 91.2% on 300 benchmark questions, compared to 80.3% for Claude Code directly accessing the code repositories (both using Sonnet 4). Miao’s core argument is that whether a paper can be successfully executed by an agent serves as a new metric for reproducibility.
Traditional research papers are passive, fragmented carriers of knowledge. Jiacheng Miao used AlphaGenome as an example: a roughly 200-page PDF where key details are hidden in supplementary materials; reproducing results requires figuring out the code repository and Python environment from scratch, while applying the method requires understanding the training data. He stated that this is currently how scientific knowledge is communicated.
His proposal is that papers should not only be published as PDFs but also released alongside an interactive agent. The specific approach involves packaging the paper and its codebase as an MCP server, which includes three types of components—tools that wrap computational methods, resources that expose links to the manuscript and code, and step-by-step instructions written according to the paper’s analytical chain. Users ask natural language questions, and the agent calls upon the paper’s methods, writes analysis code, and returns results.
Reliability comes from a multi-agent build-and-test loop: one set of agents identifies the codebase, installs dependencies, and determines applicable tasks; the team extracts question-and-answer pairs from the paper and repository as benchmarks; another set of testing agents repeatedly runs these Q&As, iteratively hardening the server until it can reproduce all results reported in the paper and every notebook in the repository.