読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニュースクイックテイク1 分で読めるDisconfirmAI 編集部

MoRE shares expert pools across layers: self-reported lower perplexity than standard MoE at 114M–1.15B

MoRE lets adjacent layer groups share an expert pool while each layer keeps its own router plus a learnable depth embedding; authors self-report lower perplexity than standard MoE and weight sharing at 114M–1.15B under equal budgets.

検証済み 2026-09-18 15:03 GMT+8

MoRE lets adjacent layer groups share a single expert pool while each layer keeps its own router, distinguished by a lightweight learnable depth embedding; the authors self-report that at three scales from 114M to 1.15B parameters, under equal compute and parameter budgets, perplexity is lower than standard MoE and weight-sharing architectures, requiring only minor changes to existing MoE implementations.

Previously, standard MoE kept each layer's experts separate, while weight-sharing architectures struggled to distinguish layers; MoRE aims to combine parameter efficiency with layer distinction via a shared pool plus depth embeddings.

The perplexity comparisons are self-reported by Eric S. Qiu, Kilian Q. Weinberger and seven authors in total, with no third-party benchmark named, and the scales are small.

The results have not been independently reproduced; the preprint was submitted to arXiv on September 16, updated to v2 on the 17th, with the page noting acceptance to COLM 2026 (arXiv:2609.18176).

ソース

  • arxiv.org↗
リサーチを続ける →
ログイン / 新規登録
読書リサーチレーダー投資フレームワーク