読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · 扩散模型训练成本修正重大1 分で読めるDisconfirmAI 編集部

Study Revises Diffusion Language Model Training Costs: Compute Gap Drops from 16x to 3.0x-7.8x Under New Protocol

An arXiv preprint finds that using exact likelihood scoring and block diffusion training reduces the pretraining compute cost of diffusion language models far below the previously assumed 16x autoregressive model level.

2026-10-14 12:00 GMT+8

A new preprint study by Guanghan Wang et al., submitted to arXiv on October 13, 2026, reduces the estimated pretraining compute gap between diffusion language models (DLMs) and autoregressive language models (ARMs) from the widely cited 16x to between 3.0x and 7.8x.

The previous belief in prohibitive DLM training costs stemmed from older evaluation methods: scoring via evidence lower bound (ELBO) and training full-sequence decoder-only models. The new study adopts exact likelihood scoring under the model's own decoding order, block diffusion training, and an encoder-decoder architecture, sweeping over practical block sizes.

Across approximately 1,700 model variants, the researchers found that switching to exact likelihood alone reduced the compute multiple to 12.8x. Combining block diffusion training with an encoder-decoder architecture further lowered it to 7.8x at block size 32, narrowing to 3.0x at block size 4. Downstream benchmarks showed that improvements in likelihood translated to accuracy gains.

These conclusions are currently based on first-party author benchmarks and have not yet been independently reproduced.

ソース

  • arxiv.org↗
トピック
ログイン / 新規登録
読書リサーチレーダー投資フレームワーク

扩散模型训练成本修正

【背景】此前广泛引用的研究指出,扩散语言模型(DLM)达到同等似然度所需的预训练计算量是自回归模型(ARM)的16倍,这一高昂成本曾被视为阻碍DLM大规模从头预训练的主要障碍。

2 ステップ0 件更新日 2026-10-16 02:39

経緯

  1. 2025-10-15Cornell/Amazon团队发布arXiv预印本,提出新评估协议,将DLM相对ARM的预训练计算成本倍数从16x修正为3.0x-7.8x区间。arxiv.org ↗
  2. 2026-03-22独立研究者Caio Vicentino发布控制变量对比研究,发现在小规模实验中DLM训练吞吐量仅比ARM低4.7%,挑战了扩散模型训练成本极高的固有印象。arxiv.org ↗

未解決の問題

  • 在超大规模(>10B参数)预训练中,修正后的低成本协议是否能持续保持相对于ARM的优势?
  • 随着高质量公开语料库趋于枯竭,DLM在数据受限场景下的‘智能交叉点’效应是否会成为其取代ARM的主要驱动力?
  • 工业界是否会调整现有的GPU集群配置以适配DLM特有的并行解码和块扩散训练模式?
トピック全文を読む →
リサーチを続ける →