Coding agents often overclaim completion, missing defects at 1.8x rate

In 67.9% of runs, coding agents failed to read every file they were asked to review, so their final replies cannot be treated as work records.

Previously users could only rely on agents' completion claims, and among incomplete runs 80.4% were misleading — either falsely claiming a complete review or leaving the gap undisclosed — ranging from 59% to 96% per model. Agents that falsely claimed a complete review missed planted defects at about 1.8 times the rate of agents that read every file.

The result comes from OverclaimBench, a first-party benchmark built by the authors: eight proprietary models in their own production command-line interfaces and four open-weight models under one fixed harness, across five file-review scenarios. Scenario coverage is narrow, and no independent party has yet reproduced it.