독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · AI Coding Agent Cost Benchmarks중대2 분 소요DisconfirmAI 편집부

Benchmark: Coding Agent Harness Choice Can Cause 5x Cost Variance for Same Model

New FrontierHarness Eval v1.1 data shows that selecting different coding agent harnesses can lead to up to fivefold cost differences for the same AI model, with 65% of inference spending wasted on failed or timed-out runs.

2026-11-18 06:13 GMT+8

According to the newly released FrontierHarness Eval v1.1 benchmark by Runta and Baseten, the same AI model can cost up to five times more to run depending on the coding agent harness it is paired with. The study tested 22 configurations across five open-weight models, executing 27,280 runs and consuming approximately 83 billion tokens.

While the model itself accounted for 92.8% of cost differences, the harness choice remained significant. Rankings shifted by a median of 11 positions when the model changed, and costs varied by as much as fivefold even when the model stayed the same. For instance, with the DeepSeek model, Cline CLI cost five times as much as OpenCode v1 but delivered only a two-percentage-point higher pass rate.

A major finding concerns resource waste: nearly 65% of estimated inference spending went to runs that either failed verification or timed out. Specifically, 52.7% was spent on failed runs and 11.8% on timeouts, leaving only 35.5% for successful executions. Among the top five best-value harnesses, 68% to 82% of unsuccessful runs ended with the agent submitting a final answer that failed verification rather than timing out.

The research also identified step count as the primary driver of cost, showing a statistically significant correlation (0.51) between steps and cost per successful task. Mini-SWE-agent, despite having the smallest initial prompt, averaged 38 steps per successful task, ranking 17th in cost efficiency. In contrast, the most cost-effective harnesses relied heavily on shell commands and rarely used subagents or context summarization.

로그인 / 회원가입
독서리서치레이더투자 프레임워크

출처

  • thenewstack.ioThe New Stack↗
주제

AI Coding Agent Cost Benchmarks

New benchmarks reveal that the choice of coding agent harness significantly impacts AI costs, with the same model potent…

4 단계1 건업데이트됨 2026-11-20 17:35

현재까지의 과정

  1. 2026-11-172026-11-17: Runta and Baseten publish FrontierHarness Eval v1.1 results, highlighting up to 5x cost variance per task based on harness choice.news.sulat.com ↗
  2. 2026-11-182026-11-18: The New Stack reports on the benchmark, noting 65% of spend goes to failed runs and step count drives cost.thenewstack.io ↗
  3. 2026-07-082026-07-08: Databricks releases internal benchmark data showing simple harnesses reduce costs while maintaining quality on multi-million line codebases.databricks.com ↗

아직 해결되지 않은 문제

  • How do subscription-based pricing models compare to API-billed agents in terms of total cost of ownership?
  • Can public benchmarks like SWE-Bench be adjusted to better reflect private repository complexities?
  • What is the long-term impact of reward hacking instances found in agent logs on overall cost efficiency?
전체 주제 읽기 →
리서치 계속하기 →