ReadingResearchRadarInvestment frameworkAI Company Primary Sources
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Pydantic Engineer Michael Pfaffenberger: Code Mode Halves Costs on Long Tasks, But Is More Expensive with Just 5 Records

Pydantic engineer Michael Pfaffenberger ran tests scoring 8 models across 5 small applications using 5,276 evaluations. He found that code mode (having the model write programs to call tools) reduced average costs by 58% and time by 51% on long tasks, but increased costs by 21% to 38% when handling just 5 records.

2026-10-28 17:00 GMT+8

A popular saying among AI Agent developers is that rather than having the model call tools one by one, it is better to have it write a program that wraps tool calls in a loop, making everything cheaper, faster, and more accurate. Pydantic engineer Michael Pfaffenberger believes in this direction, but he noticed that almost all numbers supporting this argument come from complex Agent scenarios with hundreds of tools or massive payloads. In reality, many Agents are quite simple: a few CRUD tools and a specific task. So he designed an intentionally simple benchmark test to verify under what conditions this argument actually holds true.

The test covered five small Pydantic AI applications—shopping cart, calendar, chatroom, kanban board, and inventory system—each with single-record CRUD tools, paginated lists, and no shell, file system, or bulk interfaces. Three workloads were used: 50 daily short tasks (averaging 13 tool calls), 12 long multi-step tasks (60 to 500 records, averaging 68 calls), and one repeatable rule task that could scale from 5 to 200 records. Eight models were tested, with code mode both on and off, resulting in 5,276 scored runs, verified by a deterministic oracle rather than an LLM judge.

The core finding is that the longer the task, the greater the advantage of code mode. On long tasks, the eight models saw an average cost reduction of 58%, a time reduction of 51%, and a success rate increase of 16 percentage points. At a scale of 200 records, code mode was 47% to 75% cheaper than standard tool calling. However, at just 5 records, code mode was 21% to 38% more expensive in three of the five tasks. The crossover point appears to be between 20 and 50 records.

Sign in / Sign up
ReadingResearchRadarInvestment frameworkAI Company Primary Sources

Pfaffenberger's explanation of the mechanism overturns a common assumption. The usual story is that each tool call adds another round of model requests, and code mode saves money by reducing these rounds. But the tests revealed that modern models already make massive parallel calls—at a scale of 200 records, claude-haiku-5-5 packed all 200 calls into a single response. With code mode enabled, the number of model requests dropped by only 1% to 22% on average, far less dramatic than advertised. What was actually cut was the content accumulated per request: the call instructions written by the model and the tool results read back. In standard tool calling, results from 205 calls all entered the context, with gpt-6.1-sol ultimately reading 183,723 input tokens; in code mode, the model wrote only 5 program snippets and read 34,461 tokens. Output tokens fell by 66%, and input tokens by 53%.

The cost structure of code mode is therefore clear: it charges a fixed fee—a sandbox usage guide of about 2,600 characters attached to each request—and then gains decreasing marginal costs as task size grows. Standard tool calling has no fixed fee, but every record makes the model write one more call and read one more result. For a single edit, the fixed fee wins; for hundreds of records, the per-item fee wins. Pfaffenberger writes: 'Models are better at writing loops than at being the loop itself, and this gap grows with task length.'

Changes in correctness are equally noteworthy. Without code mode, success rates slid from 92% to 100% at 5 records down to 62% to 79% at 200 records; with code mode, they remained at 92% to 100% even at 200 records. The weakest model, gpt-6-luna, failed all 15 runs at 200 records without code mode, with 7 times doing nothing but explaining why it couldn't; with code mode, it succeeded 8 times. This wasn't a crash or timeout, but the model choosing to give up when faced with 200 calls. In long tasks, 16 out of 24 runs without code mode failed, whereas with code mode, only 1 out of 24 failed.

However, Pfaffenberger honestly listed where code mode pays a price. Small models write buggy code, adding another round trip for each failure; some models print data back into the conversation, offsetting token savings; a code snippet that fails midway may duplicate writes upon retry, with gpt-6-luna creating duplicate records three times in long tasks due to this. Regarding debugging, failed programs hide intermediate values in the sandbox, forcing developers to read code rather than JSON.

This test suite ran within Pydantic's own Monty sandbox—an interpreter written in Rust for Python, with default host file access disabled, no network, a 30-second execution limit, and a 256MiB heap memory cap. Pfaffenberger emphasized that sandbox design is as important as model capability: generated code can only reach the tools you provide it, and those tools still require their own validation and authorization.

As a first-party vendor benchmark, these data have not yet been independently reproduced. The test environment uses in-memory application state, involving no real network latency, concurrent load, or production-grade fault recovery. However, Pfaffenberger's explanation of the mechanism—that code mode cuts not the number of request rounds but the context accumulated per request—is a structural argument that can be independently verified, offering practical reference value for all developers considering introducing code execution capabilities into Agents.

Sources

  • pydantic.dev↗
Continue researching →