LectureRechercheRadarCadre d'investissement
Connexion / Inscription
← Retour à la lecture
ActualitésSujet · Clean优化器Matériel1 min de lectureRédaction DisconfirmAI

New Optimizer Enables Single-GPU Training of 13B Models

The Clean optimizer reduces second-order method memory to linear cost, achieving 26% faster convergence than AdamW and enabling single-GPU pre-training of 13B models.

2026-10-06 12:00 GMT+8

The Clean optimizer reduces the memory complexity of second-order methods from quadratic to linear, enabling the pre-training of a 13B-parameter model on a single 80GB GPU.

Traditional second-order optimizers like SOAP accelerate convergence but incur prohibitive memory costs. Clean uses randomized Nyström approximation to estimate preconditioners, preserving curvature information while significantly compressing state usage.

Author-reported benchmarks show Clean reaches AdamW's final performance 26% faster in wall-clock time. Its low-precision variant, Q-Clean, reduces optimizer memory consumption by over 50% compared to Muon. These results are currently from a preprint and have not yet been independently reproduced.

Sources

  • arxiv.org↗
Sujet

Clean优化器

Clean优化器通过引入随机Nyström近似,解决了大语言模型训练中二阶方法内存开销过高的瓶颈。该算法将SOAP类全曲率优化器的内存复杂度从二次方降至线性,使得在单张80GB GPU上预训练13B参数模型成为可能。

2 étapes1 piècesMis à jour 2026-10-07 21:12

Comment on en est arrivé là

Connexion / Inscription
LectureRechercheRadarCadre d'investissement
  • 2026-10-03论文《Clean: Second-order LLM Training at Linear Memory Cost via Nyström Sketching》提交至arXiv,提出Clean和Q-Clean优化器。arxiv.org ↗
  • 2026-10-07媒体报道指出Clean优化器支持单卡预训练13B模型,且比AdamW快26%。arxiv.org ↗
  • Ce qui reste non résolu

    • Clean优化器在更大规模模型(如70B+)上的扩展性表现如何?
    • Q-Clean的低精度压缩是否会在特定任务微调中导致显著的性能损失?
    • 该方法与现有的其他高效优化器(如Adafactor或LAMB)在实际生产环境中的对比数据尚缺。
    Lire le sujet complet →
    Continuer la recherche →