NeurIPS 2026 paper proposes Virtual Head Attention to reduce KV cache while improving model quality
VHA uses linear operations to optimize attention, reducing KV cache by 4x compared to GQA-8 with higher quality.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
Addressing the "impossible triangle" among KV cache size, training FLOPs, and model quality in long-context LLMs, the newly proposed Virtual Head Attention (VHA) mechanism aims to improve all three aspects simultaneously. The research was published at NeurIPS 2026.
VHA introduces two lightweight linear operations: Q Premix and Linear Postmix. The former recovers query head diversity, while the latter fuses inter-head features. Due to the absence of nonlinear operations, these can be folded into projection layers during inference, directly reusing existing GQA optimizations.
Author self-tests show that in 4K-context 50B-token pretraining, VHA consistently surpasses the standard GQA baseline by 0.05% to 0.81% across all scales, while reducing KV cache by 4x compared to GQA-8 at dense scales and 2x compared to GQA-4 at MoE scales. In an industrial 30B MoE model evaluation, VHA outperforms both MLA and GQA-4.
Current results are based on the author team's self-testing; independent reproduction reports have not yet emerged.