Study finds high acceptance rate does not mean faster speed in lossy speculative decoding due to length inflation

The arXiv preprint "Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding," submitted by Chiatzen Wang et al. on October 16, 2026, raises a critical warning regarding Large Language Model (LLM) inference acceleration techniques, specifically "Speculative Decoding."

Speculative decoding typically measures speedup by the number of draft tokens accepted per verification round. In lossless settings, this metric is a valid proxy for efficiency because the target model's output distribution remains unchanged. However, researchers identified a "length inflation" phenomenon in training-free lossy speculative decoding using relaxed verification rules: while more tokens are accepted per round, the generated completion lengths increase significantly, largely offsetting the apparent speed gains.

The team evaluated existing relaxed verification rules across twelve target-drafter pairs. At the most relaxed settings, lossy verification accepted an average of 1.34 times as many tokens per round compared to lossless verification, but produced completions that were 1.54 times longer. This inflation primarily stems from extended reasoning traces, where low-probability tokens are erroneously accepted, leading to verbose or looping outputs that sometimes exhaust the budget needed to deliver a final answer.

Furthermore, rules that reduce verifier rounds typically result in substantial drops in task accuracy. The study concludes that under relaxed verification, "accepted tokens per round" alone is an unreliable measure of decoding speed; actual efficiency must be assessed by reporting both completion length and verifier rounds.