CMU Study: AI Agent Decision Failures Stem from 'Reading Right, Acting Wrong'
A Carnegie Mellon University preprint finds that most errors by fast System One models occur after correctly reading state facts, pointing to composition failures.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
Researchers at Carnegie Mellon University have found that 'System One' models—fast AI systems used in agents that choose actions by scoring options in a single pass—primarily fail not because they misread environmental information, but because they cannot combine correctly read facts into the right action.
In a frozen benchmark covering simulated environments, customer-service policies, and tool use, the team paired every next-action question with independent factual probes about the same state. The results showed that for the six most accurate systems, 73% to 85% of decision errors occurred on states where every measured fact was answered correctly. This indicates that the models 'understood' the situation but still chose the wrong next step.
The authors note that when probed facts provably determine the action, models fail to combine them themselves; however, composing the action via code using the models' own fact answers recovers success rates to above 0.96. This finding offers a new direction for debugging agents: focusing on enhancing the ability to derive actions from known facts, rather than solely improving context perception.