ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Airbnb Selects Coding Models by Cost Per Task, but Why Evaluation Scores Fail to Predict Production Spend

The Airbnb engineering team has publicly disclosed its model selection methodology for coding agents: running evaluations on proprietary tasks, recording scores and cost per task for each model and reasoning tier, and using these metrics to set default models and inference ceilings. However, after one deployment, a model that performed well in evaluations incurred weekly production costs several times higher than its predecessor—because users began engaging in longer autonomous sessions, making cached context the primary cost driver. This case reveals the behavioral gap between evaluation and production, along with the diagnostic path to address it.

2026-11-19 05:48 GMT+8

Internally, Airbnb operates two coding agent tools: AirChat runs on engineers' personal cloud development machines, while AirChat Remote executes cloud-based agents in independent cloud workspaces, allowing engineers to hand off a task and move on to other work. Both paths route through Airbnb's large language model gateway to Amazon Bedrock. Thousands of engineers and engineering-adjacent staff use these tools weekly. The challenge arose when eight new models were released over approximately five weeks; by mid-October, about two-thirds of coding agent requests were directed to these new models. Each release can simultaneously alter output token volume, cost, latency, and capacity, yet the benchmark charts published alongside a launch do not demonstrate how a model handles the actual work of Airbnb engineers.

Their approach involves running two systems in parallel. Prior to launch, they run evaluations on proprietary tasks, measuring the score and cost per task for each model and each reasoning tier. Cost per task is defined as the list-price cost of all tokens consumed by the agent to complete one task. Bedrock reports four types of tokens per request: uncached input, cache reads, cache writes, and output. In evaluations, they price all four token types for every request and sum them across all requests within a task. Reasoning tiers simultaneously affect both score and cost: on one model, moving from a low tier to a high tier increased the score by 0.8 points but raised the cost per task by a factor of 2.5. They used these results to set the default tier for each model and an inference ceiling, which is enforced by the gateway.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Post-launch, production observability revealed what actually occurred. A coding model referred to as Model A scored more than three points higher than the previous best model in evaluations, with a similar cost per task, all scored by the same grader. They deployed it, setting the inference ceiling to high and instructing engineers to use low or medium tiers for daily work. Weeks later, when calculating Model A's production usage at list price, they found that the average weekly spend per user was several times higher than that of the predecessor model. The cost per request was approximately triple, and the average number of requests sent per user was also roughly triple.

The mechanism behind this deviation lies in behavioral change. Output tokens account for only one-tenth of a request's cost; the majority of costs stem from cached context—the content the model re-reads in each request. Because Model A could work autonomously for extended periods, engineers began using longer, self-directed sessions. Evaluations measured the model's performance on a single task, whereas production measured how people chose to collaborate with it. Later, a newer model scored higher than Model A on the same tasks and grader, at about one-sixth of Model A's cost per task. They migrated Model A's users to this newer model via a configuration change. In the first full week post-migration, this group of heavy users sent roughly the same number of requests, but their spend on the primary model dropped to about one-seventh.

This case has direct operational implications for teams managing coding agents. While cost per task in evaluations is a necessary filtering signal, it assumes fixed task boundaries. When model capabilities change how users work—through longer autonomous sessions, more requests, and larger cached contexts—production costs deviate from evaluation predictions. The diagnostic path involves comparing CloudWatch runtime metrics (latency, token counts, throttling) with gateway usage records (aggregated by model, client, team, and user), using the evaluated cost per task as an expected baseline, and observing which dimension drives actual deviations. Airbnb also emphasizes retaining at least one alternative model for each major model that performs comparably in proprietary evaluations, enabling load shifting; without such alternatives, choices under capacity pressure become guesswork.

Sources

  • aws.amazon.comAmazon Web Services↗
Continue researching →