AI Evaluator Artificial Analysis Updates Index v5 with 60% Private Tasks to Curb Benchmark Gaming
Artificial Analysis releases Intelligence Index v5, adding two new tests and making 60% of tasks non-public to prevent model overfitting on public benchmarks.
Independent AI evaluation firm Artificial Analysis has updated its core metric, the Intelligence Index v5. The new version aggregates results from nine benchmark tests and, for the first time, sets 60% of the test tasks to a non-public status. This adjustment aims to counteract developers' tendency to "game" or overfit their models against public benchmarks, thereby providing a more accurate reflection of general model capabilities.
Index v5 introduces Terminal-Bench-Science (featuring 70 agentic scientific tasks) and FrontierCode 2.0 (a non-public programming benchmark by Cognition that evaluates whether code changes would be accepted by repository maintainers). It removes older tests such as GDPval-AA v2.1, SciCode, and AA-LCR v1.1. While reduced task visibility makes external verification harder, the move is designed to enhance the anti-manipulation robustness of the rankings.