독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스간단한 요점1 분 소요DisconfirmAI 편집부

MoRE shares expert pools across layers: self-reported lower perplexity than standard MoE at 114M–1.15B

MoRE lets adjacent layer groups share an expert pool while each layer keeps its own router plus a learnable depth embedding; authors self-report lower perplexity than standard MoE and weight sharing at 114M–1.15B under equal budgets.

검증됨 2026-09-18 15:03 GMT+8

MoRE lets adjacent layer groups share a single expert pool while each layer keeps its own router, distinguished by a lightweight learnable depth embedding; the authors self-report that at three scales from 114M to 1.15B parameters, under equal compute and parameter budgets, perplexity is lower than standard MoE and weight-sharing architectures, requiring only minor changes to existing MoE implementations.

Previously, standard MoE kept each layer's experts separate, while weight-sharing architectures struggled to distinguish layers; MoRE aims to combine parameter efficiency with layer distinction via a shared pool plus depth embeddings.

The perplexity comparisons are self-reported by Eric S. Qiu, Kilian Q. Weinberger and seven authors in total, with no third-party benchmark named, and the scales are small.

The results have not been independently reproduced; the preprint was submitted to arXiv on September 16, updated to v2 on the 17th, with the page noting acceptance to COLM 2026 (arXiv:2609.18176).

출처

  • arxiv.org↗
리서치 계속하기 →
로그인 / 회원가입
독서리서치레이더투자 프레임워크