LecturaInvestigaciónRadarMarco de inversión
Iniciar sesión / Registrarse
← Volver a la lectura
NoticiasTema · 滑窗与线性注意力之争Material1 min de lecturaEditorial de DisconfirmAI

Sliding window beats linear attention

Preprint shows Sliding Window Attention outperforms linear attention by 2-10x on long-context tasks without retraining.

2026-10-06 12:00 GMT+8

Sliding Window Attention (SWA) with attention sinks performs as well or better than most retrofitted Linear Attention models across multiple LLMs.

The study compares two methods for reducing LLM memory consumption: compressing the KV cache (via SWA) and retrofitting models to use Linear Attention with fixed-size states. On long-context reasoning benchmarks like Needle-in-a-Haystack and BABILong, SWA achieved 2 to 10 times higher performance than linear attention.

SWA requires no additional training, is extremely fast, and uses little memory. The authors argue that when training budgets are limited, switching to SWA is a much more effective way to reduce inference costs than retrofitting linear attention.

This is an arXiv preprint (v2 updated Oct 4, 2026); results have not yet been independently reproduced.

Fuentes

  • arxiv.org↗
Tema

滑窗与线性注意力之争

【背景】大语言模型因二次方复杂度导致显存爆炸,业界曾寄希望于“线性注意力”通过固定状态压缩来破局。然而,微软研究团队在2026年8月发布的预印本《Sliding-window beats linear attention》指出,无需重训的滑…

3 pasos1 piezasActualizado 2026-10-08 20:17

Cómo llegó hasta aquí

Iniciar sesión / Registrarse
LecturaInvestigaciónRadarMarco de inversión
  1. 2026-08-28微软应用科学组发布预印本v1,首次提出SWA with sinks优于多数后训练线性注意力模型。arxiv.org ↗
  2. 2026-09-09科技行者报道该论文,指出线性注意力此前常与“残废对手”(无sink的SWA)对比,存在基准偏差。techwalker.com ↗
  3. 2026-10-04论文更新至v2版本,进一步确认在长文本生成和推理任务中,SWA性能比线性注意力高2-10倍。arxiv.org ↗

Lo que aún está pendiente

  • 在需要全局依赖的极端长上下文场景(如百万级token),SWA的多层叠加感受野是否仍会失效?
  • 线性注意力若从头训练(而非后训练改造),其理论上限是否真的高于SWA?
  • 硬件厂商是否会针对SWA的稀疏特性推出专用加速内核,从而进一步拉大成本差距?
Leer el tema completo →
Continuar investigando →