Princeton and Microsoft Researchers Propose AbridgeKV, Reporting 3.2x Speedup on H100 for Long Reasoning Cache Compression

Researchers from Princeton University and Microsoft Research have introduced AbridgeKV, a method for compressing the KV cache during long chain-of-thought (CoT) generation. Instead of relying on traditional cache eviction strategies, AbridgeKV trains the model to compress each block of generated text into the KV entries of a few "learned pause tokens." These are extra input tokens with trained embeddings that produce no output but encode the information of the preceding block. Once encoded, the original block's cache is freed, and subsequent tokens attend to the pause tokens' KV entries instead.

In tests involving 446 competition math problems (from AIME, HMMT, etc.) with a 32K-token generation limit, AbridgeKV matched the accuracy and output length of an identically trained full-KV model while keeping only a quarter of the generated cache. In contrast, the strongest published evictor resulted in significantly longer generations at the same budget, and the closest trained compressor, LightThinker, lost 10 points in accuracy.

Regarding deployment, the study reports that served in the vLLM engine, AbridgeKV reduced the cache size per sequence by 3.5 times and achieved a 3.2 times faster completion time per problem on a saturated H100 GPU compared to full KV. These results are based on fine-tuning the Qwen3-4B-Base model on the OpenMathReasoning dataset and represent first-party benchmarks, with no independent reproduction reported yet.