ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial2 min readDisconfirmAI Editorial

METR report: AI models excel at coding but lag in judgment tasks

A new METR report shows AI capabilities are improving exponentially on verifiable tasks like coding, but models still fall short of human performance on open-ended tasks requiring judgment.

2026-11-21 02:15 GMT+8

The AI evaluation nonprofit METR has released an update to its widely discussed "time horizon" plot. The report finds that while frontier AI models continue to improve rapidly on tasks with clear success criteria—such as software engineering and mathematics—they struggle significantly with open-ended tasks that require human-like judgment, such as designing research evaluations or scoring philosophical arguments.

METR's original time horizon plot measured task difficulty by the time it took humans to complete them, showing that the length of tasks AI could successfully complete was doubling every four months. The updated report expands this suite to include more complex, long-duration, and judgment-heavy tasks. On these new metrics, even the most advanced models failed to produce evaluations deemed useful by METR researchers, and their performance on philosophical argument scoring remained below human levels.

This divergence highlights a key bottleneck in automating AI research and development (AI R&D). Progress on "verifiable" tasks is driven by reinforcement learning, where systems can be trained through repeated attempts and automated feedback. However, applying this technique to tasks where success is ambiguous or takes longer to ascertain remains difficult. Experts note that while models are getting better at these fuzzy tasks—with some recent models achieving "borderline above the bar" performance compared to "clearly below" a year ago—the lack of robust metrics makes predicting future progress challenging.

The report also addresses practical difficulties in evaluation. For many of the new tasks, human completion times are estimated rather than measured due to cost and expertise constraints. Additionally, subjective tasks often rely on limited human raters, raising questions about the generalizability of the scores. Despite these limitations, the inclusion of judgment-based tasks is seen as a necessary step toward understanding whether AI can autonomously drive scientific discovery.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Sources

  • technologyreview.comMIT Technology Review↗
Continue researching →