読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · RAM-Net架构重大1 分で読めるDisconfirmAI 編集部

RAM-Net Optimizes Linear Attention with Sparse Addressing

New architecture reduces token interference via independent slot addressing, cutting per-step state access by 8x compared to Mamba2.

2026-10-07 12:00 GMT+8

RAM-Net proposes a sparse address-based access mechanism to address the degradation of long-range fine-grained recall caused by shared states in linear attention.

Traditional linear attention superimposes information from distinct tokens within a fixed-size recurrent state, creating inter-token interference. RAM-Net organizes the state as an array of independent slots and uses an Address Decoder to map each key or query to a sparse address, selecting only a small subset of slots to write to or read from at each step.

Authors report that this design achieves the lowest perplexity with competitive commonsense reasoning and outperforms strong baselines on fine-grained long-range retrieval. A key efficiency metric is accessing fewer state elements per step than all baselines, e.g., 8x fewer than Mamba2.

These are first-party benchmark results from a paper accepted at NeurIPS 2026; independent reproduction has not yet been reported.

ソース

  • arxiv.org↗
トピック

RAM-Net架构

【背景】传统线性注意力模型将无限历史压缩为固定大小状态,导致令牌间干扰和信息丢失。香港中文大学团队提出RAM-Net架构,通过“稀疏可寻址状态”解决这一瓶颈。

3 ステップ1 件更新日 2026-10-07 20:14

ログイン / 新規登録
読書リサーチレーダー投資フレームワーク

経緯

  1. 2026-02-12RAM-Net论文v1版本提交至arXiv,标题为《Expressive Linear Attention with Selectively Addressable Memory》。arxiv.org ↗
  2. 2026-10-06论文v2版本发布,标题调整为《Linear-Time Sequence Modeling with Sparsely Addressable State》,并确认被NeurIPS 2026接收。arxiv.org ↗
  3. 2026-10-07新闻摘要报道RAM-Net通过独立槽位寻址减少令牌干扰,每步状态访问比Mamba2少8倍。arxiv.org ↗

未解決の問題

  • RAM-Net在大规模工业级部署中的实际推理延迟与内存占用具体数值是多少?
  • 相较于Delta-rule等最新线性注意力变体,RAM-Net在常识推理任务上的差距是否可通过进一步调优缩小?
トピック全文を読む →
リサーチを続ける →