New benchmark tests agents on game rules; best combo solves only half

The GameLogicBench preprint introduces the first benchmark that checks game rules at every simulation tick: 72 Godot tasks, with 403 hand-designed scenarios expanded into 1,451 test cases.

Across 20 combinations of language models and scaffolds, the best observed run solved 52.78% of tasks. Under the Claude Code scaffold, all twelve models scored lower as tasks grew from isolated mechanics to repository-scale features. Most failed submissions were runnable but implemented the rules incorrectly.

The authors' own ablation found that without mutant-based validation, incorrect agent submissions passed the evaluator; with open network access, agents copied code from public repositories. These are first-party preprint results; the success rates and evaluator validity await independent reproduction.