ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Datadog Troubleshooting Prompt Cache Misses: Four Silent Code-Level Traps

Datadog engineers documented the troubleshooting process of locating individual request discrepancies from aggregate cost anomalies, discovering four specific causes—Go map iteration order, missing request body control parameters, unpassed SDK options, and API route incompatibility—that prevented prompt caching from actually taking effect despite appearing enabled.

2026-11-06 08:00 GMT+8

The AI applications team at Datadog encountered a counterintuitive phenomenon while investigating LLM costs: multiple internal applications had enabled prompt caching options, yet there were zero cache hits in actual requests. Each request contained over 180,000 input tokens, with single-request costs ranging from $0.74 to $0.82. Once caching took effect, the cost for the same task could drop to between $0.02 and $0.14. The gap was so significant that finding the cause seemed impossible.

Staff Engineer Bruno Volpato and his team recorded the complete troubleshooting process. Their methodology was to first use Cloud Cost Management to view overall spending, then use Agent Observability to trace down to individual model calls, and finally compare the differences between the first call and subsequent calls within the same task. An agent might call the model multiple times to complete one task, repeating system instructions, tool definitions, and conversation history each time. If caching worked correctly, subsequent calls should read these repeated contents from the cache.

The investigation revealed four categories of silent failure modes. The first category involved mutable content mixed into reusable prefixes. A chat service concatenated system instructions with current time, organization details, feature flags, and incident context. Prompt caching relies on byte-by-byte matching of request prefixes—if a timestamp appears before the instructions, even if the subsequent instructions are identical, the cache cannot hit. Their fix was to place stable instructions first and request-specific details last, using cache boundary markers to denote the end of the reusable portion where supported.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The second category was non-deterministic ordering of tool lists. That chat service built its tool list in Go, and Go's map iteration does not guarantee order. One request might send tools in the order A, B, C, while the next sends them as B, A, C. Since tool descriptions are part of the cached prefix, a change in sequence breaks the match. The patch was simple: sort by name before building the list. The team also added a test that inserts the same tools into maps in different orders to verify the resulting list is consistent.

The third category was caching options not being passed to the actual request. One application enabled an option called WithPromptCaching, but inspection of requests sent to Anthropic revealed that this option only added a beta header; the request body lacked the control parameters telling the provider what to cache. Another application was more direct: the SDK's caching option was simply not passed to the component preparing the provider request.

The fourth category was API routes that did not support caching. Datadog's gateway can route through Anthropic's native Messages API or a compatibility endpoint formatted for OpenAI. The native route supports cache reads, while the compatibility route has none. Anthropic's documentation confirms that the compatibility endpoint does not support prompt caching. Migrating applications to the native API became a potential fix path, but the team needed to verify that message roles, tool calling, and streaming responses still functioned correctly under the new route before recommending it.

Volpato emphasized a troubleshooting trap in the article: you cannot rely solely on aggregate cache hit rates. They found that applications and gateways might both record the same model call, leading to double-counting of costs; conversely, if records lack cache fields, caching appears unused. Some records omit system instructions or tool definitions, making it impossible to judge why the provider didn't reuse previous inputs. When evaluating fix opportunities, the team also pointed out an easily overlooked cost dimension: some providers charge extra fees for cache writes. If you repeatedly write content that is rarely reused, costs may actually increase.

The value of this article lies not in discovering a brand-new caching principle—the mechanisms of prefix matching and tool order consistency are not new discoveries. Its contribution is demonstrating a real troubleshooting process: starting from aggregate cost anomalies, narrowing down to discrepancies at the individual request level, and ultimately pinpointing four specific code-level causes. For teams running LLM applications with poor caching results, this troubleshooting path and these four patterns are far more directly usable than any generic principles.

Sources

  • datadoghq.comDatadog↗
Continue researching →