ArXiv Preprint: Coding Agent Reports Show 40.7-Point Gap in Task-State Recovery
New study quantifies information loss in AI coding agent self-reports, showing significantly lower task-state recovery compared to full execution contexts.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
A new measurement study named CTXGAP finds that human supervisors relying solely on coding agents' user-facing reports recover 40.7 percentage points fewer task states than those reading the full execution context. The study was published as a preprint by Guoyang Xu et al. on arXiv on October 14, 2026.
In traditional automation trust research, information is designed by developers. However, large model agents generate their own plans, commentary, and summaries. When agents deviate from plans or encounter failures, these critical states often disappear from routine reports, leaving supervisors unaware of issues warranting scrutiny.
The team analyzed 37 coding tasks and 4,799 valid probes. While agents answered 93.6% of task-state questions on demand, and readers with full context recovered 92.7% of states, report-only readers recovered just 52.0%. Even at final checkpoints of 30 verified fixes, the gap remained 34.8 points. Human annotation identified states mentioned without sufficient supporting detail as the largest category of unrecovered information.
These results are based on author-run benchmarks in a preprint and have not yet been independently reproduced. The study proposes 'report-based task-state recoverability' as a complementary evaluation dimension to address blind spots in current agent transparency assessments.