ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
TopicDisconfirmAI EditorialUpdated 2026-10-25 18:43

MiniMax稀疏注意力

5 steps1 pieces3 Sources

MiniMax发布并开源了名为“MiniMax Sparse Attention”（MSA）的块级稀疏注意力机制，旨在解决大模型在百万级长上下文场景下的计算瓶颈。该方案基于分组查询注意力（GQA），通过轻量级索引分支独立选择Top-k键值块，并在主分支中仅对选中块执行精确注意力计算。在109B参数模型上，MSA在保持与全注意力相当性能的同时，将每令牌注意力计算量降低28.4倍，并在H800 GPU上实现14.2倍的预填充和7.6倍的解码加速。【背景】MiniMax研究员指出，由于GPU内存布局限制，稀疏注意力无法直接跨架构移植，MSA的设计特意避免了多KV头共享选择导致的多样性消除问题，选择了各头独立检索块的路线以适配更广泛的硬件。

How it got here

  1. 2025-10-01

    MiniMax发布博客解释M2为何采用全注意力而非继承M1的闪电注意力，称当时高效注意力尚未达到生产就绪状态。

    huggingface.co ↗
  2. 2026-05-26

    MiniMax研发负责人Skyler Miao在X平台发布MSA架构图，展示其在1M token长度下预填充9.7倍、解码15.6倍的加速潜力。

    huggingface.co ↗
  3. 2026-06-11

    MiniMax在arXiv提交《MiniMax Sparse Attention》论文，正式定义MSA为基于GQA的块级稀疏注意力，并公布109B模型实验数据。

    arxiv.org ↗
  4. 2026-06-12

    论文更新版本，确认推理内核已在GitHub开源，且搭载MSA的生产级原生多模态模型MiniMax-M3已在Hugging Face发布。

    arxiv.org ↗
  5. 2026-10-25

    MiniMax研究员Olive Song在AI Engineer大会阐述MSA设计哲学，强调其不能沿用DeepSeek DSA方案的原因在于GQA架构下的内存读取效率与头多样性平衡。

    youtube.com ↗

What is still unsettled

  • MSA在端到端实际业务响应中的延迟改善比例是否与其理论FLOPs减少倍数一致？
  • 相比DeepSeek的MLA架构，MSA在显存占用上的具体权衡数据是多少？
Sign in / Sign up
ReadingResearchRadarInvestment framework

Sources

  • arxiv.orgarXiv论文: MiniMax Sparse Attention↗
  • huggingface.coHuggingFace博客: MiniMax Goes Sparse↗
  • youtube.comYouTube视频: Olive Song演讲↗

Coverage in this topic

  • MiniMax Researcher Olive Song: Sparse Attention Cannot Be Directly Ported Across Architectures; GPU Memory Layout Dictates Design Choices2026-10-25News

    At the 2026 AI Engineer World's Fair, MiniMax researcher Olive Song explained two core design choices of the M3 model: why MiniMax Sparse Attention (MSA) cannot adopt DeepSeek’s DSA scheme, and why text and vision data are jointly trained from the first step. She noted that in GQA architectures, sharing selection across multiple KV heads eliminates head diversity and scattered memory reads reduce the compute-to-memory ratio; thus, MSA allows each head to independently select and retrieve blocks rather than single tokens. Regarding multimodality, joint training from scratch under the same text token budget hardly affects text capability, whereas adding vision after pretraining decay leads to performance degradation.