Self-run test finds AI monitors miss over half of agent-simulation details

A research team used a self-authored murder mystery to test whether AI can reconstruct the underlying story from multi-agent interaction logs. The conclusion: even the strongest monitor, GPT-6 Astra at high reasoning effort, fully recovered only 48.1% of the seeded facts and relationships.

The authors first hand-built an event graph with 184 nodes and 229 edges as ground truth, then had eight agents run the investigation, and finally asked models to reconstruct the history from roughly 58,000 words of trajectory. Omissions far outnumbered errors: Astra high left 42.6% of rubric items unreported, while Gemini 3.1 Pro recovered just 16.5%.

This is a first-party result: scoring was done by a GPT-5.6 Sol judge pipeline, not yet validated against independent human grading, and all ten simulations came from a single authored mystery, so generalization is unknown — the authors list human validation as their next step. For readers doing agent-incident investigations, the transferable piece is the method: fixing the reference account in advance is what lets you measure what a monitor report leaves out, not just what it gets wrong.