LeituraPesquisaRadarFramework de investimento
Entrar / Cadastrar
← Voltar para a leitura
NotíciasTópico · Clean优化器Material1 min de leituraEditorial DisconfirmAI

New Optimizer Enables Single-GPU Training of 13B Models

The Clean optimizer reduces second-order method memory to linear cost, achieving 26% faster convergence than AdamW and enabling single-GPU pre-training of 13B models.

2026-10-06 12:00 GMT+8

The Clean optimizer reduces the memory complexity of second-order methods from quadratic to linear, enabling the pre-training of a 13B-parameter model on a single 80GB GPU.

Traditional second-order optimizers like SOAP accelerate convergence but incur prohibitive memory costs. Clean uses randomized Nyström approximation to estimate preconditioners, preserving curvature information while significantly compressing state usage.

Author-reported benchmarks show Clean reaches AdamW's final performance 26% faster in wall-clock time. Its low-precision variant, Q-Clean, reduces optimizer memory consumption by over 50% compared to Muon. These results are currently from a preprint and have not yet been independently reproduced.

Fontes

  • arxiv.org↗
Tópico

Clean优化器

Clean优化器通过引入随机Nyström近似,解决了大语言模型训练中二阶方法内存开销过高的瓶颈。该算法将SOAP类全曲率优化器的内存复杂度从二次方降至线性,使得在单张80GB GPU上预训练13B参数模型成为可能。

2 etapas1 peçasAtualizado 2026-10-07 21:12

Como chegamos aqui

Entrar / Cadastrar
LeituraPesquisaRadarFramework de investimento
  • 2026-10-03论文《Clean: Second-order LLM Training at Linear Memory Cost via Nyström Sketching》提交至arXiv,提出Clean和Q-Clean优化器。arxiv.org ↗
  • 2026-10-07媒体报道指出Clean优化器支持单卡预训练13B模型,且比AdamW快26%。arxiv.org ↗
  • O que ainda está indefinido

    • Clean优化器在更大规模模型(如70B+)上的扩展性表现如何?
    • Q-Clean的低精度压缩是否会在特定任务微调中导致显著的性能损失?
    • 该方法与现有的其他高效优化器(如Adafactor或LAMB)在实际生产环境中的对比数据尚缺。
    Ler o tópico completo →
    Continuar pesquisando →