Benchmark: Coding Agent Harness Choice Can Cause 5x Cost Variance for Same Model
New FrontierHarness Eval v1.1 data shows that selecting different coding agent harnesses can lead to up to fivefold cost differences for the same AI model, with 65% of inference spending wasted on failed or timed-out runs.
According to the newly released FrontierHarness Eval v1.1 benchmark by Runta and Baseten, the same AI model can cost up to five times more to run depending on the coding agent harness it is paired with. The study tested 22 configurations across five open-weight models, executing 27,280 runs and consuming approximately 83 billion tokens.
While the model itself accounted for 92.8% of cost differences, the harness choice remained significant. Rankings shifted by a median of 11 positions when the model changed, and costs varied by as much as fivefold even when the model stayed the same. For instance, with the DeepSeek model, Cline CLI cost five times as much as OpenCode v1 but delivered only a two-percentage-point higher pass rate.
A major finding concerns resource waste: nearly 65% of estimated inference spending went to runs that either failed verification or timed out. Specifically, 52.7% was spent on failed runs and 11.8% on timeouts, leaving only 35.5% for successful executions. Among the top five best-value harnesses, 68% to 82% of unsuccessful runs ended with the agent submitting a final answer that failed verification rather than timing out.
The research also identified step count as the primary driver of cost, showing a statistically significant correlation (0.51) between steps and cost per successful task. Mini-SWE-agent, despite having the smallest initial prompt, averaged 38 steps per successful task, ranking 17th in cost efficiency. In contrast, the most cost-effective harnesses relied heavily on shell commands and rarely used subagents or context summarization.