NVIDIA Engineer: The Cost of Sparse Attention Has Not Disappeared, It Has Just Moved into the Indexer
The NVIDIA team measured the design space of sparse attention indexers using production-grade FP8 kernels, pointing out that the cost has shifted from core attention to the indexer, which still must scan the entire context. They provide a set of specific design constraints derived from Tensor Core tile capacity and arithmetic intensity.
Sparse attention is often described as the cure for long-context inference: each query attends only to the k most relevant keys, so the compute cost of core attention no longer grows with context length. However, in a co-design guide published by the NVIDIA Technical Blog team in November 2026, they point out that this description obscures a key fact—the selected k keys must be found, and the indexer responsible for finding them still scans all N keys. The cost has not disappeared; it has simply moved from core attention into the indexer. In DeepSeek V3.2’s 128K prefill, the indexer and Top-k selection accounted for 47% of the time spent on an entire layer.
To understand why the indexer becomes a bottleneck, it needs to be reduced to hardware operations. An indexer is essentially a batched matrix multiplication followed by a fused post-processing step (ReLU and cross-head weighted sum). The matrix multiplication runs on Tensor Cores, while the post-processing runs on CUDA cores, finally writing one scalar logit per key back to HBM. What determines whether this operation is compute-bound or bandwidth-bound is its arithmetic intensity—the number of floating-point operations performed per byte of data moved.
The core conclusion derived by the NVIDIA team is that the indexer's arithmetic intensity is proportional to the group size G (equal to the number of query heads divided by the number of key heads), and the decode intensity grows approximately linearly with the speculative decoding draft length D+1. Intuitively, G query heads share the same key head, and their scoring results are reduced to a single logit via weighted sum, meaning that for every logit written, G times the work of a matrix multiplication is done. Speculative decoding allows D+1 query tokens to be scored in a single pass over the key cache, further amortizing the bandwidth overhead.