ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Snorkel AI Review: Fable 5.2 Coding Failures Shift from Terminal Crashes to Strategic Misjudgments

Snorkel AI compared Fable 5.2 and 5.1 across 26 matched frontier coding tasks, finding that Fable 5.2’s pass@1 rose from 53% to 71%. The primary gain stems from improved reliability—terminal crashes and formatting errors have nearly disappeared—but remaining failures are concentrated in strategy selection and task comprehension, and these errors do not improve with multiple sampling attempts.

Verified 2026-11-13 08:15 GMT+8

AI Advocate Jonathan Schlosser and Senior Research Scientist Ankit Aich of Snorkel AI compared Fable 5.2 against 5.1 across 26 matched frontier coding tasks. Fable 5.2 achieved a pass@1 rate of 71.4%, an increase of 18 percentage points over Fable 5.1’s 53.1%. However, the numbers themselves are not the most useful part of this review; what truly matters is the structural shift in the causes of failure.

Failures in version 5.1 were mechanical. Of the 97 failed trajectories, 42% were due to premature termination—the model had written most of the correct solution, attempted a large file write via the terminal, crashed the session, and could not recover the state it had built. 25% were output format errors, and 26% were tool call argument errors. The common characteristic was that the solution existed, but the step required to deliver it failed.

Version 5.2 has almost entirely eliminated this class of problems. Across 121 trials, there were no premature terminations, no format failures, and only one tool argument error. The improvement in reliability is significant: while Fable 5.1 passed every trial on only 6 of the 26 tasks, Fable 5.2 achieved this consistency on 15 tasks. When Fable 5.2 could solve a task, it typically succeeded on the first attempt.

However, the remaining failures present a completely different profile. Of the 40 failed trials with root cause analysis, 35% were attributed to wrong strategy selection, 28% to reasoning defects, and 23% to misreading task requirements. These are not execution-level mistakes, but judgment issues at the planning level.

More critically, these reasoning errors are stable rather than random. The review found that reasoning defects were concentrated on two tasks, where every attempt independently reached the same incorrect conclusion. This means the model is not making occasional errors, but systematically misinterpreting expected behavior. It correctly executes a flawed plan, even verifying its own answer internally, so internal checks pass while external verifiers fail.

Sign in / Sign up
ReadingResearchRadarInvestment framework

This finding has direct implications for practical use. In the era of Fable 5.1, users could compensate for failures through multiple sampling—crashes were random, and retries often bypassed them. But Fable 5.2’s remaining failures are deterministic: a stable misinterpretation fails five times out of five, and additional sampling does not recover success. As the review authors noted, this class of failure belongs to the same category as issues exposed by Opus 5 on the same task set, and is harder to fix with scaffolding engineering because the failing runs execute cleanly, self-verify successfully, but verify against an incorrect premise.

It should be noted that this is Snorkel AI’s own evaluation; the task set was created by the team, and the 26 matched tasks come from a subset of 30 original tasks where both models had at least three clean runs. The authors also acknowledge whether this pattern persists in future model generations remains untested. But for this current batch of frontier coding agents, the bottleneck has shifted from random crashes at the execution level to stable misjudgments at the planning level.

Sources

  • snorkel.aiSnorkel AI↗
Continue researching →