Akamai Engineers Demonstrate Inference Optimization Pitfalls: Why Speculative Decoding Slowed Down Qwen Model
Du'an Lightfoot and Khaja Omer from Akamai Technologies presented a workshop on AI inference optimization at the AI Engineer conference. They demonstrated through experiments that speculative decoding, typically an acceleration technique, reduced the generation speed of a Qwen model from approximately 58 tokens per second to 16 tokens per second in their specific configuration. The resource emphasizes the importance of establishing baselines, changing serving parameters, and rerunning load tests to evaluate the balance between speed and quality.
In AI engineering practice, blindly applying advanced optimization techniques often backfires. Du'an Lightfoot and Khaja Omer, engineers at Akamai Technologies, provided a specific case study in their "Agents That Own Their Inference" workshop video published on October 11, 2026. They demonstrated that introducing speculative decoding—a technology commonly used for acceleration—caused the generation speed of a Qwen model deployed via vLLM endpoints in a Kubernetes environment to drop significantly from approximately 58 tokens per second to 16 tokens per second.
This failure was not accidental but revealed a core principle of inference performance evaluation: any faster inference configuration must be measured against the actual model, hardware, and workload. The structure of Lightfoot and Omer's workshop clearly illustrates how to systematically diagnose this issue. First, they established baselines, including streaming measurements, model configuration, memory budgets, and the usage of prefill, decode, and KV caching. Subsequently, Omer changed the serving deployment to an FP8 model and repeated load tests and simple output quality checks before moving to speculative decoding.
When the drafter model caused performance degradation, they analyzed factors such as acceptance rate, model compatibility, and GPU contention, ultimately returning to the baseline configuration. The video also demonstrates how to find the concurrency point where throughput flattens while latency rises by adjusting parameters, modifying sequence limits, and remeasuring results. Despite minor issues like a notebook failure that changed the model used in the final comparison, the presenters acknowledged these limitations, maintaining transparency in their method.