Apple's new method guides diffusion language models without an extra forward pass

Apple's machine learning research team has published a paper introducing probe guidance, which builds a guidance signal from the frozen internal states of an existing diffusion model, eliminating the extra inference forward pass that autoguidance requires.

The team reports the method sets a new state of the art on unconditional generation for continuous diffusion language models, and says that applied to a 1.7B diffusion language model it consistently improves multiple-choice question answering benchmarks. These results are first-party benchmarks and have not been independently reproduced.

The paper also reports a mechanism finding: the weak model in autoguidance must come from a low-entropy region of training. If it holds, this explains a principle autoguidance has lacked an account of, and for diffusion language model research it may be more useful than any single benchmark number.