AI Coding Agent Cost Benchmarks
4 steps1 pieces7 Sources
New benchmarks reveal that the choice of coding agent harness significantly impacts AI costs, with the same model potentially costing up to five times more depending on the tool configuration. FrontierHarness Eval v1.1 by Runta and Baseten tested 22 configurations across five open-weight models, finding that 65% of inference spending was wasted on failed or timed-out runs. While the model itself accounts for 92.8% of cost differences, the harness still causes considerable variance; rankings flip dramatically between models, meaning a cheap harness for one model may be expensive for another. Databricks' internal benchmark corroborates this, showing that simple harnesses like Pi often outperform complex ones in cost-efficiency without sacrificing quality. The core disagreement lies in whether public leaderboards (like Artificial Analysis) accurately reflect real-world repository costs, with AWS and Databricks arguing that private, code-specific benchmarks are necessary due to varying token efficiency and context management strategies.
How it got here
2026-11-17: Runta and Baseten publish FrontierHarness Eval v1.1 results, highlighting up to 5x cost variance per task based on harness choice.
news.sulat.com ↗2026-11-18: The New Stack reports on the benchmark, noting 65% of spend goes to failed runs and step count drives cost.
thenewstack.io ↗2026-07-08: Databricks releases internal benchmark data showing simple harnesses reduce costs while maintaining quality on multi-million line codebases.
databricks.com ↗