독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
주제DisconfirmAI 편집부업데이트됨 2026-11-20 17:35

AI Coding Agent Cost Benchmarks

4 단계1 건7 출처

New benchmarks reveal that the choice of coding agent harness significantly impacts AI costs, with the same model potentially costing up to five times more depending on the tool configuration. FrontierHarness Eval v1.1 by Runta and Baseten tested 22 configurations across five open-weight models, finding that 65% of inference spending was wasted on failed or timed-out runs. While the model itself accounts for 92.8% of cost differences, the harness still causes considerable variance; rankings flip dramatically between models, meaning a cheap harness for one model may be expensive for another. Databricks' internal benchmark corroborates this, showing that simple harnesses like Pi often outperform complex ones in cost-efficiency without sacrificing quality. The core disagreement lies in whether public leaderboards (like Artificial Analysis) accurately reflect real-world repository costs, with AWS and Databricks arguing that private, code-specific benchmarks are necessary due to varying token efficiency and context management strategies.

현재까지의 과정

  1. 2026-11-17

    2026-11-17: Runta and Baseten publish FrontierHarness Eval v1.1 results, highlighting up to 5x cost variance per task based on harness choice.

    news.sulat.com ↗
  2. 2026-11-18

    2026-11-18: The New Stack reports on the benchmark, noting 65% of spend goes to failed runs and step count drives cost.

    thenewstack.io ↗
  3. 2026-07-08

    2026-07-08: Databricks releases internal benchmark data showing simple harnesses reduce costs while maintaining quality on multi-million line codebases.

    databricks.com ↗
로그인 / 회원가입
독서리서치레이더투자 프레임워크
  • 2026-11-18

    2026-11-18: LLMtrack updates cost tracking guides, emphasizing per-engineer and per-PR metrics over seat pricing.

    llm-track.com ↗
  • 아직 해결되지 않은 문제

    • How do subscription-based pricing models compare to API-billed agents in terms of total cost of ownership?
    • Can public benchmarks like SWE-Bench be adjusted to better reflect private repository complexities?
    • What is the long-term impact of reward hacking instances found in agent logs on overall cost efficiency?

    출처

    • thenewstack.ioThe New Stack↗
    • databricks.comDatabricks Blog↗
    • news.sulat.comSulat News↗
    • artificialanalysis.aiArtificial Analysis↗
    • llm-track.comLLMtrack↗
    • aws.amazon.comAWS Cloud Financial Management↗
    • Dev.to↗

    이 주제 커버리지

    • Benchmark: Coding Agent Harness Choice Can Cause 5x Cost Variance for Same Model2026-11-18뉴스

      New FrontierHarness Eval v1.1 data shows that selecting different coding agent harnesses can lead to up to fivefold cost differences for the same AI model, with 65% of inference spending wasted on failed or timed-out runs.