vLLM introduces push-based KV transfer, benchmarks show up to 3x lower TTFT
vLLM debuts a new KV cache transfer mechanism that eliminates serial coordination overhead in pull mode, significantly improving inference latency on specific hardware.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
The vLLM project published a blog post on October 14, 2026, introducing a new "push-based" NIXL connector for disaggregated inference. This feature aims to optimize key-value (KV) cache transfer efficiency in large language model serving, thereby reducing time-to-first-token (TTFT) and time-per-output-token (TPOT).
In traditional "pull-based" modes, the decode instance must wait for the router to relay metadata, allocate destination blocks, and initiate read requests after prefill completion, creating a serial bottleneck on the critical path. The new push mode allows the decode instance to register its target memory blocks in advance while the prefill instance is still computing. Once prefill finishes, the producer writes KV blocks directly into the decoder's registered memory via RDMA, bypassing additional round-trip coordination.
According to the official vLLM blog, benchmarks using the Qwen3-32B model on Amazon EC2 P5en instance clusters showed that push mode delivered up to 3x better TTFT and up to 30% better TPOT compared to pull mode. The feature was co-developed by engineers from AWS and Mistral AI and contributed to the open-source community, supporting GPUDirect RDMA transfers between GPU and Trainium devices.