Confidence-guided decoding sends masked diffusion models down a shortcut that magnifies addition errors by an order of magnitude

Masked diffusion models that order generation by confidence fall into a "confidence shortcut" that neglects long-range dependencies; the authors' own tests say confidence-aligned training objectives can raise addition error rates by an order of magnitude.

It was previously assumed that masked diffusion models (MDMs), which generate text by progressively unmasking tokens in any order, could in principle reveal intermediate steps along logical dependencies. Authors Dueun Kim and Albert No report that standard decoding simply prioritizes high-confidence tokens, so in multi-digit addition the models predict higher-order digits without tracking carry chains.

The authors' own controlled pretraining shows confidence-guided ordering often selects suboptimal sequences, and confidence-aligned training objectives can raise addition error rates by an order of magnitude. The experimental code is open source; the findings come from the authors' own setup and await independent reproduction.