LecturaInvestigaciónRadarMarco de inversión
Iniciar sesión / Registrarse
← Volver a la lectura
TemaEditorial de DisconfirmAIActualizado 2026-11-20 17:35

AI Coding Agent Cost Benchmarks

4 pasos1 piezas7 Fuentes

New benchmarks reveal that the choice of coding agent harness significantly impacts AI costs, with the same model potentially costing up to five times more depending on the tool configuration. FrontierHarness Eval v1.1 by Runta and Baseten tested 22 configurations across five open-weight models, finding that 65% of inference spending was wasted on failed or timed-out runs. While the model itself accounts for 92.8% of cost differences, the harness still causes considerable variance; rankings flip dramatically between models, meaning a cheap harness for one model may be expensive for another. Databricks' internal benchmark corroborates this, showing that simple harnesses like Pi often outperform complex ones in cost-efficiency without sacrificing quality. The core disagreement lies in whether public leaderboards (like Artificial Analysis) accurately reflect real-world repository costs, with AWS and Databricks arguing that private, code-specific benchmarks are necessary due to varying token efficiency and context management strategies.

Cómo llegó hasta aquí

  1. 2026-11-17

    2026-11-17: Runta and Baseten publish FrontierHarness Eval v1.1 results, highlighting up to 5x cost variance per task based on harness choice.

    news.sulat.com ↗
  2. 2026-11-18

    2026-11-18: The New Stack reports on the benchmark, noting 65% of spend goes to failed runs and step count drives cost.

    thenewstack.io ↗
  3. 2026-07-08

    2026-07-08: Databricks releases internal benchmark data showing simple harnesses reduce costs while maintaining quality on multi-million line codebases.

    databricks.com ↗
Iniciar sesión / Registrarse
LecturaInvestigaciónRadarMarco de inversión
  • 2026-11-18

    2026-11-18: LLMtrack updates cost tracking guides, emphasizing per-engineer and per-PR metrics over seat pricing.

    llm-track.com ↗
  • Lo que aún está pendiente

    • How do subscription-based pricing models compare to API-billed agents in terms of total cost of ownership?
    • Can public benchmarks like SWE-Bench be adjusted to better reflect private repository complexities?
    • What is the long-term impact of reward hacking instances found in agent logs on overall cost efficiency?

    Fuentes

    • thenewstack.ioThe New Stack↗
    • databricks.comDatabricks Blog↗
    • news.sulat.comSulat News↗
    • artificialanalysis.aiArtificial Analysis↗
    • llm-track.comLLMtrack↗
    • aws.amazon.comAWS Cloud Financial Management↗
    • Dev.to↗

    Cobertura en este tema

    • Benchmark: Coding Agent Harness Choice Can Cause 5x Cost Variance for Same Model2026-11-18Noticias

      New FrontierHarness Eval v1.1 data shows that selecting different coding agent harnesses can lead to up to fivefold cost differences for the same AI model, with 65% of inference spending wasted on failed or timed-out runs.