ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial5 min readDisconfirmAI Editorial

Ollama and Together AI Joint Article: Cache Invalidation in Long-Horizon Agent Sessions Almost All Stem from Violating a Single Prefix-Matching Rule

Parth Sareen of Ollama and Zain Hasan of Together AI explain, from the opposing perspectives of harnesses and inference engines, why prefix caching is both critical and fragile in agent workloads, and provide specific repair paths for failure modes such as tool lazy-loading, reasoning effort changes, and sub-agent inheritance.

2026-11-16

Every step an agent takes, the harness resends the entire conversation to the model: the system prompt, tool definitions, user requests, every file the agent has read, and every command it has executed. What typically returns is only a few dozen tokens of tool calls. This is the typical traffic pattern for agent workloads—extremely long input sequences with very short outputs. If the GPU has to process the entire history from scratch on every turn, the longer the session, the slower and more expensive it becomes.

Prefix caching is the mechanism that prevents this degradation, but its rules are extremely simple: the cache matches from the beginning of the prompt and stops at the first differing token, after which all content must be reprocessed. Appending new content to the end of the prompt incurs no extra cost; changing any character earlier, even just one, means everything after it must be recalculated. If the change occurs at the very top, it effectively invalidates the entire session. Parth Sareen of Ollama and Zain Hasan of Together AI point out in their joint article that almost all agent cache issues derive from this single rule.

The easiest mistake on the harness side is placing content that changes every turn into the system prompt. Timestamps, git status, or the working directory the agent is currently switching through—if the harness re-renders the system prompt with fresh values on every turn, every turn becomes a full re-read. The fix is to place such information in the task message, write it once, and keep the system prompt completely identical across all sessions.

Sign in / Sign up
ReadingResearchRadarInvestment framework

A more subtle example comes from GLM-5.3-Flash: reasoning effort is not a setting handled independently by the engine; the chat template writes it into the very first system block, ahead of tools and conversation. When a user adjusts effort from low to high, the next request differs from the previous one almost from the first token, causing the entire session to be reprocessed. In long tasks, this equates to spending hundreds of thousands of tokens just to change one setting. The authors' advice is to select an effort level and stick with it; if you truly need more thinking, starting a new session is often cheaper than continuing the old one, and you also get a clean context.

Tool lazy-loading is another decision made on the harness side. Modern harnesses support agents searching for and loading tool definitions at runtime. If new definitions are added to the tool list—the tool list sits at the top of the prompt, before the conversation—the prefix changes, and everything after it is recalculated. A better approach is to place definitions at the end, since new content is naturally appended there. However, not all models support inserting tool definitions mid-conversation; some chat templates will simply error out. Harness developers need to know this; users do not.

Large tool results also require care. An agent reads a 640-line file, but the tool returns only the first 100 lines without indicating truncation. The model does not know it is missing data and may confidently edit code it has never seen. The fix requires just one line of output: telling the model it received lines 1 to 100 of 640, and that it can search or request other ranges. Truncating saves tokens, but truncating without explanation leads to more costly errors by the model.

How sub-agents are launched determines how much of the parent session's cache they can reuse. Inheriting the entire parent session (system prompt, tools, conversation history, plus the new task) means the prefix is already cached, resulting in low processing costs; sending a brief (same system prompt and tools, but only providing necessary files and instructions) means the cache covers only the setup phase. Inheritance sounds strictly better, but a cached prefix is cheap, not free—every token generated by the sub-agent still has to read through it, and it occupies GPU memory throughout. A reviewer needing only two files and a checklist likely does not need three hundred turns of history.

The engine's responsibility is to ensure the cache persists when the next request arrives and that the request is routed to the machine holding the cache. Together AI describes two routing paths: the client sends a prompt_cache_key, and requests with the same key are routed to the same node; or the prefix is hashed, and the request is sent to the node with the longest matching cached prefix. The latter is another reason why system prompts must remain consistent across sessions—if every new session starts with exactly the same system prompt and tool definitions, a new session might directly hit an existing cache.

However, routing does not unconditionally favor warm nodes. A warm node may be overloaded, requiring a trade-off between waiting in queue and going to an idle node to reprocess the prefix. The authors explicitly state that under load-aware routing paradigms, a cache miss is sometimes the more cost-effective choice. This means that even with a perfectly stable prompt, a cache hit is not guaranteed—the quality of the backend engine implementation is also a constraint.

Cache survival time is also an issue. After an agent makes a tool call, it may wait minutes for tests to run. Keeping this idle cache in GPU memory occupies space needed by active users; discarding it means the agent must re-read everything upon return. Together AI's approach is to offload the cache of idle sessions to CPU memory and restore it for the next turn.

Viewing both sides together: the harness decides where the first change lands, and the engine decides whether the cache is still there. Getting it right on either side alone is insufficient. If the harness keeps the prompt perfectly stable but the engine sends turn 7 to the wrong machine, the cache is lost. If the engine routes perfectly but the harness changes the first line every turn, the cache is still lost. The speed an agent user sees at turn 90 depends on both sides simultaneously adhering to that prefix-matching rule.

Sources

  • together.ai↗
Continue researching →