MiniMax Researcher Olive Song: Sparse Attention Cannot Be Directly Ported Across Architectures; GPU Memory Layout Dictates Design Choices
At the 2026 AI Engineer World's Fair, MiniMax researcher Olive Song explained two core design choices of the M3 model: why MiniMax Sparse Attention (MSA) cannot adopt DeepSeek’s DSA scheme, and why text and vision data are jointly trained from the first step. She noted that in GQA architectures, sharing selection across multiple KV heads eliminates head diversity and scattered memory reads reduce the compute-to-memory ratio; thus, MSA allows each head to independently select and retrieve blocks rather than single tokens. Regarding multimodality, joint training from scratch under the same text token budget hardly affects text capability, whereas adding vision after pretraining decay leads to performance degradation.
In large language models, the computational cost of attention scales quadratically with sequence length. When context windows expand to one million tokens, full attention becomes infeasible, making sparse attention a critical choice. However, at the 2026 AI Engineer World's Fair in San Francisco, Olive Song, a reinforcement learning researcher at MiniMax, highlighted an often-overlooked fact: sparse attention designs cannot be simply ported from one model architecture to another.
Song introduced MiniMax Sparse Attention (MSA), used in the MiniMax M3 model. Its core idea is to use a lightweight index branch to determine which content the main attention branch should read. The index branch performs lightweight full-attention calculations using four query heads and one key head, training its attention scores via KL divergence loss to match the main branch. The main branch then executes attention only on the blocks selected by the index.
The real design challenge lies not in this two-branch structure itself, but in adapting it to specific attention head configurations and GPU memory access patterns. Song contrasted MSA with DeepSeek’s DSA: while DSA is an excellent design, it is highly optimized for Multi-Query Attention (MQA) and large head dimensions. MiniMax uses the more common Grouped-Query Attention (GQA), which features multiple key-value heads.
Directly applying DSA to GQA creates two problems. First, algorithmically: if the index output is aggregated into a unified selection set, multiple KV heads will attend to the exact same token set, eliminating the diversity that GQA’s multi-head design was meant to provide. Second, system-wise: GPUs are most efficient with contiguous memory reads. In MQA, a single KV head with a dimension of 512 can be laid out as a continuous 1×512 block; in GQA, the same total dimension might be split across four heads into 4×128 segments, requiring multiple independent reads and significantly lowering the compute-to-memory ratio.
MSA addresses these issues. Algorithmically, it removes the aggregation of index outputs, allowing different KV heads to select different token sets, thereby preserving GQA’s diversity. System-wise, it raises the retrieval granularity from individual tokens to block levels, reducing the number of candidates entering top-K computation, lowering selection and merging overhead, and improving memory access efficiency. Song also mentioned designing dedicated kernels for training and inference, though details were not elaborated.
Regarding performance, Song reported that MSA is approximately 9x faster during the prefill phase and 15x faster during decoding compared to full attention. These figures come from MiniMax’s internal experiments and do not specify hardware models or load conditions, so they should not be treated as universal speedups for all deployment scenarios.
The multimodal training section presented similar constraint-driven logic. Song explained that MiniMax M3 uses both text and vision data from the very first training step, rather than training a text model first and bolt-on vision capabilities later. The team compared three timing strategies for introducing vision: post-decay continued pretraining (CPT style), pre-decay but after some text training, and from-scratch native joint training.
Under the same text token budget, from-scratch joint training had almost no negative impact on text capability. Adding vision after decay caused text performance degradation because the model had already converged, and the new modality interfered with learned structures. Results from the pre-decay addition were highly sensitive to data mixture ratios and hyperparameters—a smaller learning rate mitigated performance drop in one setting, but this conclusion did not necessarily generalize to other runs. Song emphasized that during large-scale pretraining, parameters cannot be arbitrarily adjusted solely for multimodal needs.
Attention maps provided further corroboration: in CPT-style models, text tokens paid little attention to visual tokens, with visual regions appearing mostly dark on the maps; whereas native multimodal training showed no clear separation between text and visual tokens, indicating the model learned to fuse both types of information from the start.
Song also described a design choice within the vision encoder: introducing 3D attention in the Vision Transformer (ViT) allows image patches from different video frames to interact directly, rather than attending only within frames. The team found this improved visual understanding, with benefits extending to some image benchmarks. However, frame count cannot be expanded indefinitely due to system and load-balancing challenges; in their exploration, grouping four consecutive frames offered the best trade-off between performance and efficiency.
The overarching design principle throughout the talk was that simpler architectures often yield better scaling properties, but this principle must be co-designed with specific attention head configurations and GPU memory access patterns to be practical.