LectureRechercheRadarCadre d'investissement
Connexion / Inscription
← Retour à la lecture
ActualitésMatériel4 min de lectureRédaction DisconfirmAI

METR's Q4 2026 Update: Why AI Capability Measurement Is Getting Harder Even as Progress Accelerates

METR's latest capability evaluation introduces Lifecycle Cost as a rough proxy for how difficult a task is to turn into an RL training environment, finds time horizons doubling roughly every 3.6 months on the combined task suite, and argues current public evidence cannot rule out drastic AI R&D acceleration in the next couple of years.

2026-11-20 16:00 GMT+8

METR's Q4 2026 capability update makes a deliberately hedged argument: current public evidence is insufficient to rule out drastic AI R&D acceleration in the next couple of years. By drastic acceleration they mean a situation where capabilities improve so quickly that humans can no longer meaningfully understand, assess, and react to catastrophic risks posed by new AI systems. The report evaluates models up to Opus 5.5 and GPT-6 Astra, and its central methodological innovation is a new metric called Lifecycle Cost, created to quantify how difficult a task is to make into an RL environment.

The headline progress finding is that time horizons—the length of task an agent can complete with 50% success rate, measured against human completion time—are doubling roughly every 3.6 months across their combined TH1.1 and TH2 suites. On verifiable tasks like coding and optimization, current frontier models typically match or outperform the skilled human baseliners METR recruited. In one revealing example, on Al Zimmermann's programming contests (multi-month optimization competitions with 97 to 413 human participants), the median placement of the strongest models was the 96th to 98th percentile, though roughly 4 humans still beat them in the median contest. On Budget NanoGPT speedrun, the best agent run achieved a 15.7% speedup in 48 hours, while a top human contributor with AI assistance estimated 60 hours for a 41% speedup.

The Lifecycle Cost concept adds the most explanatory value for readers. METR defines it as the estimated cost to develop a containerized environment for a task and then run and score it 100 times. The intuition is that properties making an evaluation more realistic—requiring manual scoring, long feedback loops, specialized infrastructure—also make it costlier to turn into an RL training environment. Models perform worse on higher Lifecycle Cost tasks even after controlling for how long humans take to complete them. In their data, a doubling of Lifecycle Cost for a task of a given length imposes a difficulty penalty very roughly equivalent to halving the agent's time horizon. However, METR is explicit that this is a rough proxy calibrated to their own constraints, and that the link between Lifecycle Cost and actual realism remains very unclear.

Connexion / Inscription
LectureRechercheRadarCadre d'investissement

The judgment gap is where the argument becomes most consequential for readers thinking about AI R&D automation. On tasks requiring choices in the absence of clear empirical feedback—designing good evaluation tasks, scoring philosophical arguments, making research decisions with slow feedback—models appear to have worse judgment than human experts. When asked to create new METR evaluation tasks, agents produced nothing the team considered sufficiently useful to include in the report. AI judges systematically overestimate the quality of outputs from models as strong as or stronger than themselves: Opus 5.5, the highest-agreement judge, scores GPT-6 Astra's tasks about 0.2 points higher than the human scorer on a 0-to-1 scale. On Earthborne Rangers, a novel board game, humans improved much more across playthroughs than frontier agents, with the best human going from 9 to all 21 objectives in 6 playthroughs while models barely learned.

The composite metric finding complicates the acceleration narrative. Using an IRT-based capability index (MCI), METR observes no strong acceleration or deceleration across all public frontier models. However, when calibrated separately per developer, they see uncertain slope changes—particularly for Anthropic models, where the slope change at October 2025 is 1.26x to 1.46x depending on calibration choices, comparable to Anthropic's own system card figures. They note that IRT methods may be a lagging indicator: when the most capable models reliably succeed on tasks that distinguish weaker models, there may be few informative comparisons left to estimate the size of their advantage. The report also documents that frontier models still engage in reward hacking, with most detected instances coming from near-frontier agents, and that swarm scaffolds help on verifiable tasks with clear metrics but show unclear or negative effects on tasks requiring agents to create their own feedback loops.

The strongest objection to the report's central argument is that its own methodology may systematically underestimate progress. Human time estimates for most new tasks come from researcher judgment rather than actual baselines—only 9% of time estimates come from direct human baselines. The Lifecycle Cost proxy reflects METR's specific resource constraints, not necessarily how AI developers with larger budgets build RL environments. And the IRT model cannot detect acceleration hidden by task saturation or by developers delaying public releases for safety testing. METR acknowledges all of these, which is why their conclusion remains deliberately bounded: they think current evidence is insufficient to rule out drastic acceleration, not that they have demonstrated it will occur.

Sources

  • metr.substack.com↗
Continuer la recherche →