ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
TopicDisconfirmAI EditorialUpdated 2026-11-20 17:35

AI Coding Agent Cost Benchmarks

4 steps1 pieces7 Sources

New benchmarks reveal that the choice of coding agent harness significantly impacts AI costs, with the same model potentially costing up to five times more depending on the tool configuration. FrontierHarness Eval v1.1 by Runta and Baseten tested 22 configurations across five open-weight models, finding that 65% of inference spending was wasted on failed or timed-out runs. While the model itself accounts for 92.8% of cost differences, the harness still causes considerable variance; rankings flip dramatically between models, meaning a cheap harness for one model may be expensive for another. Databricks' internal benchmark corroborates this, showing that simple harnesses like Pi often outperform complex ones in cost-efficiency without sacrificing quality. The core disagreement lies in whether public leaderboards (like Artificial Analysis) accurately reflect real-world repository costs, with AWS and Databricks arguing that private, code-specific benchmarks are necessary due to varying token efficiency and context management strategies.

How it got here

  1. 2026-11-17

    2026-11-17: Runta and Baseten publish FrontierHarness Eval v1.1 results, highlighting up to 5x cost variance per task based on harness choice.

    news.sulat.com ↗
  2. 2026-11-18

    2026-11-18: The New Stack reports on the benchmark, noting 65% of spend goes to failed runs and step count drives cost.

    thenewstack.io ↗
  3. 2026-07-08

    2026-07-08: Databricks releases internal benchmark data showing simple harnesses reduce costs while maintaining quality on multi-million line codebases.

    databricks.com ↗
Sign in / Sign up
ReadingResearchRadarInvestment framework
  • 2026-11-18

    2026-11-18: LLMtrack updates cost tracking guides, emphasizing per-engineer and per-PR metrics over seat pricing.

    llm-track.com ↗
  • What is still unsettled

    • How do subscription-based pricing models compare to API-billed agents in terms of total cost of ownership?
    • Can public benchmarks like SWE-Bench be adjusted to better reflect private repository complexities?
    • What is the long-term impact of reward hacking instances found in agent logs on overall cost efficiency?

    Sources

    • thenewstack.ioThe New Stack↗
    • databricks.comDatabricks Blog↗
    • news.sulat.comSulat News↗
    • artificialanalysis.aiArtificial Analysis↗
    • llm-track.comLLMtrack↗
    • aws.amazon.comAWS Cloud Financial Management↗
    • Dev.to↗

    Coverage in this topic

    • Benchmark: Coding Agent Harness Choice Can Cause 5x Cost Variance for Same Model2026-11-18News

      New FrontierHarness Eval v1.1 data shows that selecting different coding agent harnesses can lead to up to fivefold cost differences for the same AI model, with 65% of inference spending wasted on failed or timed-out runs.