NVIDIA Engineer: The Cost of Sparse Attention Has Not Disappeared, It Has Just Moved into the Indexer
The NVIDIA team measured the design space of sparse attention indexers using production-grade FP8 kernels, pointing out that the cost has shifted from core attention to the indexer, which still must scan the entire context. They provide a set of specific design constraints derived from Tensor Core tile capacity and arithmetic intensity.
Sparse attention is often described as the cure for long-context inference: each query attends only to the k most relevant keys, so the compute cost of core attention no longer grows with context length. However, in a co-design guide published by the NVIDIA Technical Blog team in November 2026, they point out that this description obscures a key fact—the selected k keys must be found, and the indexer responsible for finding them still scans all N keys. The cost has not disappeared; it has simply moved from core attention into the indexer. In DeepSeek V3.2’s 128K prefill, the indexer and Top-k selection accounted for 47% of the time spent on an entire layer.
To understand why the indexer becomes a bottleneck, it needs to be reduced to hardware operations. An indexer is essentially a batched matrix multiplication followed by a fused post-processing step (ReLU and cross-head weighted sum). The matrix multiplication runs on Tensor Cores, while the post-processing runs on CUDA cores, finally writing one scalar logit per key back to HBM. What determines whether this operation is compute-bound or bandwidth-bound is its arithmetic intensity—the number of floating-point operations performed per byte of data moved.
The core conclusion derived by the NVIDIA team is that the indexer's arithmetic intensity is proportional to the group size G (equal to the number of query heads divided by the number of key heads), and the decode intensity grows approximately linearly with the speculative decoding draft length D+1. Intuitively, G query heads share the same key head, and their scoring results are reduced to a single logit via weighted sum, meaning that for every logit written, G times the work of a matrix multiplication is done. Speculative decoding allows D+1 query tokens to be scored in a single pass over the key cache, further amortizing the bandwidth overhead.
However, this intensity curve has a hardware ceiling. A Tensor Core tile can hold at most 256 query rows, but under FP32 post-processing, register pressure limits practical usage to about 200 rows. G×(D+1) must fit within this tile; otherwise, the kernel launches more CTAs, causing runtime to jump stepwise. NVIDIA’s measurements show that at G=32, up to six draft tokens remain effectively free (runtime increases only 1.37x while workload grows 6x), and at G=8, up to 26 draft tokens can be accommodated. Beyond this boundary, benefits vanish immediately. This means group size and speculation depth compete for the same tile space—you cannot push both to their limits simultaneously.
Starting from this constraint, several design choices become clear. The number of key heads KH must be 1: adding another key head requires reading the entire key cache again during decode, causing linear runtime growth (runtime increases 7.6x when KH goes from 1 to 8), while FLOPs remain unchanged. The head dimension Hsz should be at least 128: below this value, Tensor Cores wait for CUDA cores to finish post-processing, resulting in insufficient hardware utilization; above 128, the bytes per key increase during decode, raising bandwidth pressure. FP16 post-processing expands the tile from 200 rows to 256 rows, providing a 1.31x speedup in a configuration of G=64 and D+1=4.
Compression ratio CR does not change arithmetic intensity but linearly reduces total workload. CR=4 means four tokens correspond to one indexer key, reducing the workload of both the indexer and Top-k to one-quarter. The NVIDIA team emphasizes that compression, hierarchical indexing, and cross-layer index sharing are three methods to reduce the number of keys scanned by the indexer; these are orthogonal to the intensity optimizations mentioned earlier—they should be used together, not as substitutes for one another.
The cost of Top-k selection also follows the number of keys scanned, rather than the number of values retained k. NVIDIA’s measurements show that doubling the number of keys increases Top-k time by about 1.7x (prefill) and 1.5x (decode), while increasing k by 64x raises time by less than 2x. Two kernel techniques further compress this overhead: fusing Top-k into the indexer kernel eliminates the write-out and read-back of logits; in decode, using the selection result from the previous step as a guess allows verification in a single pass, running 2.1 to 2.4 times faster than standard radix select on DeepSeek V3.2 and V4.
Finally, selection granularity determines the efficiency of core attention. Token-level selection (such as DeepSeek DSA) scatters the k keys across various locations in the cache, where each query has a different access pattern, making it nearly impossible for two queries to share a fetch. Block-level selection (such as GLM 5.3 Flash, QSA) retrieves keys as contiguous blocks, serving an entire block with a single address service, allowing multiple queries to share the fetch of the same block. NVIDIA measurements indicate that with a budget of 2,048 selected keys, halving the block size from 128 to 64 almost doubles the core attention runtime—because each block introduces one query load, one partial result, and one merge step.
It is important to note that all the above measurements and conclusions come from NVIDIA’s own production-grade FP8 kernels running on NVIDIA GPUs. Tensor Core tile sizes, the scheduling relationship between CUDA cores and Tensor Cores, and HBM bandwidth characteristics are specific to NVIDIA architecture. The applicability of these guidelines to AMD MI series, Google TPUs, or other accelerators is unknown. Additionally, the acceleration ratios cited are vendor self-tests, with no independent third-party reproduction available yet.