Preprint Proposes KV-streams: Streaming Cache Speeds Up Agentic RL Training by 2.6 to 5x
New strategy streams KV cache instead of repeated prefilling, significantly boosting agentic RL training efficiency without apparent performance loss.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
KV-streams is a plug-and-play strategy designed to address the GPU memory bottleneck caused by context compaction in training agentic Large Language Models (LLMs). Traditional compaction methods require multiple re-prefills of the context, severely hindering training throughput.
Proposed by Emiliano Penaloza and 17 other authors in an arXiv preprint updated on October 15, 2026 (v3), KV-streams enables scalable compaction by streaming the KV cache forward rather than flushing it after each compaction step.
Experiments show that KV-streams supports three different compaction strategies, achieving a 2.6 to 5x wall-clock speedup in training. Beyond efficiency, the study finds that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. In controlled settings, the authors demonstrate that reinforcement learning (RL) alone is sufficient for this behavior to emerge, contrary to prior work suggesting additional mechanisms were needed.
As a preprint, these results have not yet been independently reproduced. The authors' claim that there is "no evidence of hindering performance" is based on first-party benchmarks, and real-world stability and compatibility remain to be verified.