LeituraPesquisaRadarFramework de investimento
Entrar / Cadastrar
← Voltar para a leitura
NotíciasTópico · AI Coding Agent Cost BenchmarksMaterial2 min de leituraEditorial DisconfirmAI

Benchmark: Coding Agent Harness Choice Can Cause 5x Cost Variance for Same Model

New FrontierHarness Eval v1.1 data shows that selecting different coding agent harnesses can lead to up to fivefold cost differences for the same AI model, with 65% of inference spending wasted on failed or timed-out runs.

2026-11-18 06:13 GMT+8

According to the newly released FrontierHarness Eval v1.1 benchmark by Runta and Baseten, the same AI model can cost up to five times more to run depending on the coding agent harness it is paired with. The study tested 22 configurations across five open-weight models, executing 27,280 runs and consuming approximately 83 billion tokens.

While the model itself accounted for 92.8% of cost differences, the harness choice remained significant. Rankings shifted by a median of 11 positions when the model changed, and costs varied by as much as fivefold even when the model stayed the same. For instance, with the DeepSeek model, Cline CLI cost five times as much as OpenCode v1 but delivered only a two-percentage-point higher pass rate.

A major finding concerns resource waste: nearly 65% of estimated inference spending went to runs that either failed verification or timed out. Specifically, 52.7% was spent on failed runs and 11.8% on timeouts, leaving only 35.5% for successful executions. Among the top five best-value harnesses, 68% to 82% of unsuccessful runs ended with the agent submitting a final answer that failed verification rather than timing out.

The research also identified step count as the primary driver of cost, showing a statistically significant correlation (0.51) between steps and cost per successful task. Mini-SWE-agent, despite having the smallest initial prompt, averaged 38 steps per successful task, ranking 17th in cost efficiency. In contrast, the most cost-effective harnesses relied heavily on shell commands and rarely used subagents or context summarization.

Entrar / Cadastrar
LeituraPesquisaRadarFramework de investimento

Fontes

  • thenewstack.ioThe New Stack↗
Tópico

AI Coding Agent Cost Benchmarks

New benchmarks reveal that the choice of coding agent harness significantly impacts AI costs, with the same model potent…

4 etapas1 peçasAtualizado 2026-11-20 17:35

Como chegamos aqui

  1. 2026-11-172026-11-17: Runta and Baseten publish FrontierHarness Eval v1.1 results, highlighting up to 5x cost variance per task based on harness choice.news.sulat.com ↗
  2. 2026-11-182026-11-18: The New Stack reports on the benchmark, noting 65% of spend goes to failed runs and step count drives cost.thenewstack.io ↗
  3. 2026-07-082026-07-08: Databricks releases internal benchmark data showing simple harnesses reduce costs while maintaining quality on multi-million line codebases.databricks.com ↗

O que ainda está indefinido

  • How do subscription-based pricing models compare to API-billed agents in terms of total cost of ownership?
  • Can public benchmarks like SWE-Bench be adjusted to better reflect private repository complexities?
  • What is the long-term impact of reward hacking instances found in agent logs on overall cost efficiency?
Ler o tópico completo →
Continuar pesquisando →