Preprint Proposes Doc2Harness: First Benchmark Evaluating LLMs' Ability to Engineer Agent Harnesses
New benchmark Doc2Harness directly evaluates LLMs' capacity to build agent 'shells'; GPT-6 Astra leads with a 53.05% pass rate, but even top models fail nearly half the tests.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
Doc2Harness introduces the first deterministic benchmark for measuring long-horizon harness engineering by Large Language Models (LLMs). Unlike existing benchmarks that assess task success under fixed conditions, this study directly tests the model's ability to construct and configure the operational infrastructure for autonomous agents, a key bottleneck for scalable agentic workflows.
In evaluations across 19 frontier models, each performing three independent builds on 328 test cases, GPT-6 Astra achieved the highest pass rate at 53.05%, while the weakest model reached only 27.44%, a gap of approximately 26 percentage points. Notably, even the strongest model failed nearly half of the test cases, with errors clustering around the splitting of cross-cutting functionality across modules.
These results are self-reported by the paper's authors in a preprint (arXiv:2610.20344) and have not yet been independently reproduced.