Airbnb Selects Coding Models by Cost Per Task, but Why Evaluation Scores Fail to Predict Production Spend
The Airbnb engineering team has publicly disclosed its model selection methodology for coding agents: running evaluations on proprietary tasks, recording scores and cost per task for each model and reasoning tier, and using these metrics to set default models and inference ceilings. However, after one deployment, a model that performed well in evaluations incurred weekly production costs several times higher than its predecessor—because users began engaging in longer autonomous sessions, making cached context the primary cost driver. This case reveals the behavioral gap between evaluation and production, along with the diagnostic path to address it.
Internally, Airbnb operates two coding agent tools: AirChat runs on engineers' personal cloud development machines, while AirChat Remote executes cloud-based agents in independent cloud workspaces, allowing engineers to hand off a task and move on to other work. Both paths route through Airbnb's large language model gateway to Amazon Bedrock. Thousands of engineers and engineering-adjacent staff use these tools weekly. The challenge arose when eight new models were released over approximately five weeks; by mid-October, about two-thirds of coding agent requests were directed to these new models. Each release can simultaneously alter output token volume, cost, latency, and capacity, yet the benchmark charts published alongside a launch do not demonstrate how a model handles the actual work of Airbnb engineers.
Their approach involves running two systems in parallel. Prior to launch, they run evaluations on proprietary tasks, measuring the score and cost per task for each model and each reasoning tier. Cost per task is defined as the list-price cost of all tokens consumed by the agent to complete one task. Bedrock reports four types of tokens per request: uncached input, cache reads, cache writes, and output. In evaluations, they price all four token types for every request and sum them across all requests within a task. Reasoning tiers simultaneously affect both score and cost: on one model, moving from a low tier to a high tier increased the score by 0.8 points but raised the cost per task by a factor of 2.5. They used these results to set the default tier for each model and an inference ceiling, which is enforced by the gateway.