TheoPhysEval Benchmark: Top AI Agents Solve Unpublished Theoretical Physics Problems with 98% Accuracy
New benchmark shows GPT-6-Astra and other AI agents can accurately solve most unpublished research-level theoretical physics calculation problems at low cost, but rely heavily on literature memorization rather than original breakthroughs.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
The newly released TheoPhysEval benchmark indicates that when equipped with realistic tools such as literature search and script writing, the best-performing AI agent, GPT-6-Astra (harnessed via Codex), achieved a 98% accuracy rate on final answers for 50 unpublished research-level theoretical physics problems, at an average cost of under USD 3 per problem.
This benchmark aims to assess AI's ability to autonomously solve professional scientific problems without human hints. The problems were designed by domain experts; typically, a world expert would take about a day to solve one, while experts in adjacent specialties would require considerably longer. This result fills a gap in systematic evaluation for 'unpublished' and 'verifiable' advanced scientific reasoning tasks.
Notably, when AI access to web search and background papers was denied, accuracy showed only small changes. This suggests that models rely significantly on memorization of existing literature rather than logical derivation from scratch. Despite excellent performance in concrete calculations, researchers note that this has not translated into substantive breakthroughs in theoretical physics, implying that current AI agents still struggle to generate entirely new scientific insights.