LectureRechercheRadarCadre d'investissement
Connexion / Inscription
← Retour à la lecture
SujetRédaction DisconfirmAIMis à jour 2026-11-20 17:35

AI Coding Agent Cost Benchmarks

4 étapes1 pièces7 Sources

New benchmarks reveal that the choice of coding agent harness significantly impacts AI costs, with the same model potentially costing up to five times more depending on the tool configuration. FrontierHarness Eval v1.1 by Runta and Baseten tested 22 configurations across five open-weight models, finding that 65% of inference spending was wasted on failed or timed-out runs. While the model itself accounts for 92.8% of cost differences, the harness still causes considerable variance; rankings flip dramatically between models, meaning a cheap harness for one model may be expensive for another. Databricks' internal benchmark corroborates this, showing that simple harnesses like Pi often outperform complex ones in cost-efficiency without sacrificing quality. The core disagreement lies in whether public leaderboards (like Artificial Analysis) accurately reflect real-world repository costs, with AWS and Databricks arguing that private, code-specific benchmarks are necessary due to varying token efficiency and context management strategies.

Comment on en est arrivé là

  1. 2026-11-17

    2026-11-17: Runta and Baseten publish FrontierHarness Eval v1.1 results, highlighting up to 5x cost variance per task based on harness choice.

    news.sulat.com ↗
  2. 2026-11-18

    2026-11-18: The New Stack reports on the benchmark, noting 65% of spend goes to failed runs and step count drives cost.

    thenewstack.io ↗
  3. 2026-07-08

    2026-07-08: Databricks releases internal benchmark data showing simple harnesses reduce costs while maintaining quality on multi-million line codebases.

    databricks.com ↗
Connexion / Inscription
LectureRechercheRadarCadre d'investissement
  • 2026-11-18

    2026-11-18: LLMtrack updates cost tracking guides, emphasizing per-engineer and per-PR metrics over seat pricing.

    llm-track.com ↗
  • Ce qui reste non résolu

    • How do subscription-based pricing models compare to API-billed agents in terms of total cost of ownership?
    • Can public benchmarks like SWE-Bench be adjusted to better reflect private repository complexities?
    • What is the long-term impact of reward hacking instances found in agent logs on overall cost efficiency?

    Sources

    • thenewstack.ioThe New Stack↗
    • databricks.comDatabricks Blog↗
    • news.sulat.comSulat News↗
    • artificialanalysis.aiArtificial Analysis↗
    • llm-track.comLLMtrack↗
    • aws.amazon.comAWS Cloud Financial Management↗
    • Dev.to↗

    Couverture dans ce sujet

    • Benchmark: Coding Agent Harness Choice Can Cause 5x Cost Variance for Same Model2026-11-18Actualités

      New FrontierHarness Eval v1.1 data shows that selecting different coding agent harnesses can lead to up to fivefold cost differences for the same AI model, with 65% of inference spending wasted on failed or timed-out runs.