Ollama and Together AI Joint Article: Cache Invalidation in Long-Horizon Agent Sessions Almost All Stem from Violating a Single Prefix-Matching Rule
Parth Sareen of Ollama and Zain Hasan of Together AI explain, from the opposing perspectives of harnesses and inference engines, why prefix caching is both critical and fragile in agent workloads, and provide specific repair paths for failure modes such as tool lazy-loading, reasoning effort changes, and sub-agent inheritance.
Every step an agent takes, the harness resends the entire conversation to the model: the system prompt, tool definitions, user requests, every file the agent has read, and every command it has executed. What typically returns is only a few dozen tokens of tool calls. This is the typical traffic pattern for agent workloads—extremely long input sequences with very short outputs. If the GPU has to process the entire history from scratch on every turn, the longer the session, the slower and more expensive it becomes.
Prefix caching is the mechanism that prevents this degradation, but its rules are extremely simple: the cache matches from the beginning of the prompt and stops at the first differing token, after which all content must be reprocessed. Appending new content to the end of the prompt incurs no extra cost; changing any character earlier, even just one, means everything after it must be recalculated. If the change occurs at the very top, it effectively invalidates the entire session. Parth Sareen of Ollama and Zain Hasan of Together AI point out in their joint article that almost all agent cache issues derive from this single rule.
The easiest mistake on the harness side is placing content that changes every turn into the system prompt. Timestamps, git status, or the working directory the agent is currently switching through—if the harness re-renders the system prompt with fresh values on every turn, every turn becomes a full re-read. The fix is to place such information in the task message, write it once, and keep the system prompt completely identical across all sessions.