読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · 智能体上下文重发成本重大7 分で読めるDisconfirmAI 編集部

CoreWeave Blog: The True Driver of Agent Production Costs is Context Resend in Loops; Architecture Choices Matter More Than Model Selection

Shreyan Basu Ray, author for the Weights & Biases blog under CoreWeave, explains that agentic workflow costs are driven by input tokens rather than output tokens—each loop iteration resends all accumulated context, resulting in a long-tailed cost distribution. The article identifies model-tier orchestration (expensive models for planning, cheap models for execution) as the strongest architecture-level cost lever among published evidence. τ-bench's pass^k consistency data indicates that single-attempt success rates severely underestimate the true cost calculated per successful task.

2026-09-29 08:31 GMT+8

Most teams can get an agent to complete a task once. Getting the same agent to complete that task on the 100th identical request, at a cost you can forecast, with a trace you can read when it fails, is a different engineering problem. Weights & Biases by CoreWeave blog author Shreyan Basu Ray attributed the core mechanism of this gap in a long-form piece published in July 2026 to one sentence: the cost driver is the loop resending accumulated context each iteration.

Specifically, a ten-step agent does not make ten small calls. It makes ten increasingly large ones. System prompts, tool schemas, conversation history, and every prior tool result are resent in each iteration. Data reported by Anthropic gives the shape: agent interactions use roughly 4x the tokens of chat interactions, and multi-agent systems roughly 15x. Agentic workloads routinely run input-to-output token ratios of 100:1 or worse, which means agentic cost is an input-token problem, not an output-token problem—intuitions built on chat pricing are off by an order of magnitude.

The article illustrates the mechanism with a concrete example of a customer service ticket triage agent: assume a 6,000-token system prompt with tool schemas, 8 steps, roughly 1,500 tokens of tool results and reasoning added per step, and 300 output tokens per step. A single run totals about 90,000 input tokens and 2,400 output tokens. At representative frontier pricing ($3 per million input tokens, $15 per million output tokens), that is approximately $0.31 per task. Compare that to a single classification call using the same system prompt, which costs about $0.02. The agent is roughly 14 times more expensive, and the entire difference comes from context resent across turns.

ログイン / 新規登録
読書リサーチレーダー投資フレームワーク

If the stable system prompt is cached (with a typical cache-read rate of one-tenth of base input), the same run drops to about $0.18, a reduction of about 42%. At a scale of 100,000 tasks per month, this single change is worth about $13,000. A systematic evaluation across three major providers on a multi-turn research benchmark found that prompt caching reduced API costs by 41% to 80%, improved time to first token by 13% to 31%, and savings scaled with prompt size, reaching nearly 90% at 50,000-token prompts.

But the article emphasizes a counterintuitive finding: naive full-context caching is not the optimal strategy. Caching only the stable system prompt was most consistent across both cost and latency, because full-context caching triggers cache writes for dynamic tool calls and results that will never be reused, adding overhead without corresponding reads. On one model, full-context caching produced a latency regression of about 9%, while system-prompt-only caching improved latency by about 31%. Practical rules are therefore very specific: keep the cacheable prefix stable, do not place timestamps, session identifiers, or user-specific strings at the beginning of the system prompt, because one differing token at the start invalidates the entire prefix.

Architecture choices impact cost far more than model selection. The article notes that the strongest published evidence for architecture-driven savings is model-tier orchestration: expensive models plan and delegate, cheap models execute. Anthropic reported in its multi-agent documentation that on the BrowseComp benchmark, an orchestrator model delegating to cheaper worker sub-agents reached 96% of the all-frontier configuration's performance at 46% of the cost—specifically, 86.8% accuracy at $18.53 per problem versus 90.8% accuracy at $40.56 per problem. The mechanism is simple: the orchestrator only sees plans and summaries, while workers absorb the bulk of the token-heavy reading and writing at the lower rate.

This saving holds under two conditions: workers must not genuinely need frontier reasoning for their subtasks, or you pay twice through retries; and repeat calls should land on the same worker so each keeps its own warm cache instead of re-paying full price for shared context.

A more subtle issue than cost is the honest measurement of success rates. τ-bench introduced pass^k—the fraction of k independent attempts at the same task that all succeed. The results are sobering: an agent resolving over 60% of retail tasks on a single attempt dropped to about 25% when required to succeed across eight attempts. A system that looks like a 60% performer is closer to one in four measured honestly. If your agent faces real users, the hundredth identical request matters more than the first.

The MAST Multi-Agent System Failure Taxonomy, based on analysis of over 1,600 annotated execution traces across seven frameworks, identified 14 failure modes. About 40% belong to specification and system design issues, about a third to inter-agent alignment failures, and the rest to verification failures. The authors explicitly note that improvements in base model capability are insufficient to cover the entire taxonomy—waiting for the next generation of models to fix reliability suggests a long wait, because the defects lie in the system design itself.

This cost structure directly rewrites priorities for inference infrastructure. In workloads with an input-to-output ratio of about 100:1, the majority of GPU time is spent processing inputs rather than generating outputs—the bottleneck is context, not generation. KV cache hit rate thus elevates from a tuning detail to a first-class operational metric, because a high hit rate directly eliminates the dominant cost.

The optimization order given by the article is: fix context first, then caching, then routing, and finally consider training, measuring after each step. Skipping intermediate steps to switch to a larger model usually reveals that you paid more to solve a problem that belonged in the context layer.

Gartner predicted in June 2025 that over 40% of agentic AI projects would be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. Model capability is not on that list—the failures are economic and managerial, not capability-based.

Cost per call flatters agents. If a triage agent has a 70% success rate and retries once upon failure, the effective cost per successful task is much higher than the per-run number, because failed runs consume the full token budget before failing. Benchmarks or dashboards that ignore failed runs always make agents look cheaper than they are.

The same effect appears in published benchmark data: on the SWE-bench verified subset, multiple configurations reached identical 54.0% accuracy, but costs ranged from about $259 to about $1,790—same accuracy, roughly seven times the spend. Without a cost axis, leaderboards cannot tell you which to deploy.

There is a documented cliff when budgets tighten. In TestEvo-Bench, capping per-task spend at $1 dropped one agent's success rate on test generation from 70.6% to 44.2%, and another from 59.5% to 18.8%. Budget caps are a real control mechanism, but they are not free; different agents degrade in very different ways under budget pressure.

Token spend is just the visible part. Production budgets also bear tool and API call costs, retrieval and vector database infrastructure, trace storage and observability, evaluation overhead, human escalation handling—which is often the largest true unit cost when an agent routes to a person—and the engineering time to maintain tool contracts.

The tail matters more than the mean. Runs generated by agents that recursively spawn subagents or hit retry storms can cost orders of magnitude more than the median, and without per-run caps those runs are unbounded. Only by tagging cost per task in traces from day one and classifying by outcome can you calculate cost per successful task, cost per escalation, and the cost distribution.

The article simultaneously draws boundaries: when deterministic rules, SQL queries, classifiers, search indexes, or RPA scripts can reliably solve the task, agents are the wrong tool. Agents add latency, cost, nondeterminism, and operational surface area, requiring genuine uncertainty or adaptation problems to justify them. Choosing a deterministic alternative is not a step down; it is matching the tool to the problem.

There is an instructive divergence in the multi-agent field. Anthropic reported that its multi-agent research system outperformed single-agent configurations by 90.2% on internal evaluations, while also reporting a roughly 15x token multiplier and noting that token usage explained most of the performance variance. Cognition held a nearly opposite view in "Don't Build Multi-Agents": agents acting on partial context make conflicting decisions, so a single thread produces more coherent results. Both are right about different task shapes—the former is breadth-first research where subtasks are truly independent; the latter is coding, where each decision constrains the next.

The advice running through all choices is: choose the least autonomous architecture that satisfies the task, then add agentic flexibility only where the workflow genuinely needs runtime decisions. Architecture determines cost; task shape determines architecture.

The question in production environments is not whether an agent can complete a task once, but whether the workflow can fail safely, improve continuously under evaluation, and stay within latency and cost budgets. Every item in the current design consensus—from explicit termination conditions to typed tools to trajectory evaluation—exists because someone has already paid the tuition for not doing so. Compare against a deterministic baseline before scaling: if the agent cannot beat the baseline on measurable dimensions, that is also a useful result, and the cost learned now is far smaller than after launch.

ソース

  • wf.coreweave.com↗
トピック

智能体上下文重发成本

AI智能体在生产环境中的成本失控并非源于模型推理本身,而是“上下文重发”机制导致的隐性开销。当智能体执行多步任务时,每次调用都会将累积的历史对话、系统提示词和工具定义重新发送给大模型。这种模式使得输入Token消耗随步骤数呈二次方增长,而非…

6 ステップ1 件更新日 2026-10-24 15:41

経緯

  1. 2026-05-14云成本咨询公司LeanOps发布审计报告,指出AI智能体因上下文累积导致成本是聊天机器人的10-100倍,其中62%的账单来自重复发送的输入Token。leanopstech.com ↗
  2. 2026-05-26GitHub仓库Bolt的研究文档详细拆解了智能体单次调用的成本构成,量化了固定系统开销(约1.4万Token/次)对月度成本的显著影响。github.com ↗
  3. 2026-07-22CoreWeave旗下Weights & Biases博客发表文章,提出模型分层编排(贵模型规划、便宜模型执行)是比模型选择更强的架构级成本杠杆。wf.coreweave.com ↗

未解決の問題

  • 记忆系统(如Mem0, Hindsight)在长周期对话中何时能真正突破盈亏平衡点,实现比全量重发更低的服务成本?
  • 不同提供商的前缀缓存机制(如Anthropic的显式标记 vs OpenAI的自动缓存)在混合工作负载下的实际ROI差异有多大?
  • 是否存在标准化的智能体成本可观测性指标,以便企业跨平台比较不同框架(如LangChain vs AutoGen)的上下文管理效率?
トピック全文を読む →
リサーチを続ける →