독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스간단한 요점1 분 소요DisconfirmAI 편집부

Deeper models find superposition geometries beating uniform polytopes

A first-party toy-model study reports asymmetric geometries with lower error than uniform polytopes, challenging Anthropic's conjecture; unverified on real LLMs.

2026-09-28 21:06 GMT+8

Asymmetric geometries for storing neural-network features can achieve lower error than the uniform polytopes previously thought optimal — a first-party toy-model result by Bartosz Rzepkowski and colleagues.

Superposition is the phenomenon where a model packs more features than it has dimensions into a limited space. Anthropic's 2022 "Toy Models of Superposition" conjectured that for independent features of equal importance and sparsity, the best storage geometry may be uniform polytopes; previously, shallow models only found symmetric solutions.

The study redoes the same task with a deeper architecture of a linear encoder plus MLPs, reports that asymmetric geometries yield lower error, and derives two theoretical benchmark decoders explaining why shallow models only find symmetric solutions. Code is open-sourced. This is a first-party result, with experiments covering only 4 features compressed to 2 dimensions; whether it generalizes to real LLMs remains unverified, and the study was published on LessWrong.

출처

  • lesswrong.com↗
리서치 계속하기 →
로그인 / 회원가입
독서리서치레이더투자 프레임워크