New Study Claims Theory of Mind Benchmarks Test Algorithmic State Tracking, Not Belief Reasoning
PDDL-Mind research suggests high scores on ToM benchmarks often reflect fitting state transition algorithms rather than genuine psychological understanding.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
A recent arXiv preprint titled "Algorithmic Primitives of Theory of Mind Benchmarks" proposes that existing Theory of Mind (ToM) benchmarks may not truly measure model comprehension but instead test parsing and algorithmic state tracking.
The research team developed PDDL-Mind, a neuro-symbolic method that parses narratives into the Planning Domain Definition Language (PDDL) and explicitly executes state update algorithms. Experiments showed that across mainstream benchmarks like MMToM, MuMA, and FanToM, outsourcing parsing to programs alone brought performance close to strong agentic pipelines. Outsourcing state tracking to PDDL moved GPT-4o's performance to near-human levels.
These results suggest that complex ToM benchmarks primarily penalize Large Language Models for their unreliable implicit implementation of state tracking and parsing algorithms, rather than inherent limitations in belief or counterfactual reasoning. This finding calls for a re-evaluation of current AI assessment validity in social cognition domains.