ReadingResearchRadarInvestment frameworkAI Company Primary Sources
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

AWS: Combining Knowledge Distillation with Medusa Brings a 1.1-Billion-Parameter Model Close to 70-Billion-Parameter Quality, Accelerating Inference by 10 to 20 Times

The AWS and Booking.com teams have integrated knowledge distillation (compressing model scale) with Medusa parallel decoding (reducing sequential generation steps) into a unified training pipeline. This enables the 1.1-billion-parameter TinyLlama to approach the quality of the 70-billion-parameter Llama-3.1 across three travel product features while achieving 10 to 20 times faster inference.

2026-10-30 00:51 GMT+8

Large language models face a core contradiction in travel platforms: larger models yield more accurate answers, but inference latency and costs also rise. Booking.com's AI trip planning assistant must extract user preferences from dialogue in under one second; smart filters need to convert natural-language searches into structured filtering conditions in real time; and accommodation description generation must control costs across millions of listings. These scenarios' demands on latency and cost make deploying 70-billion-parameter-level models directly unfeasible.

In a technical blog post published in October 2026, Daniel Zagyva, Senior Machine Learning Engineer at Amazon Web Services (AWS), and Laurens van der Maas, Senior Applied Scientist at AWS, demonstrated how to integrate two orthogonally optimized techniques into a single pipeline. The first is knowledge distillation: using a large 'teacher' model to generate predictions on labeled and unlabeled domain data, then training a small 'student' model on these predictions, allowing the student to inherit the teacher's knowledge with significantly reduced parameters. The second is Medusa: attaching multiple prediction heads to a base model, where each head predicts a subsequent token in parallel, transforming sequential token-by-token decoding into multi-token parallel decoding, thereby leveraging GPU parallel computing capabilities to reduce inference steps.

These two techniques operate on different dimensions. Knowledge distillation reduces the computational cost per token (because the model is smaller), while Medusa reduces the number of sequential steps required to generate a text segment (because multiple tokens can be predicted simultaneously). Their acceleration effects are multiplicative rather than substitutive: the distilled model itself is already about 6 times faster, and adding Medusa heads on top provides an additional 2 to 3.6 times speedup, resulting in a combined 10 to 20 times acceleration.

Sign in / Sign up
ReadingResearchRadarInvestment frameworkAI Company Primary Sources

The key methodological contribution lies in integrating both into a unified training pipeline rather than processing them in stages. Specifically: first, fine-tune the teacher model (Llama-3.1, 70 billion parameters) to generate predictions on unlabeled, domain-relevant text; then, during student model training (TinyLlama, 1.1 billion parameters), attach Medusa heads directly to the student model and perform single-stage training using a mix of teacher predictions and human-labeled data. This produces a compact model that inherits both the teacher's knowledge and multi-token parallel prediction capabilities.

The team also compared the engineering trade-offs of two Medusa variants. Medusa-1 requires fine-tuning the base model first, freezing its weights, and then separately fine-tuning the Medusa heads; Medusa-2 trains all components simultaneously in a single stage. Both achieve comparable speedups with no significant quality loss, but Medusa-2's simpler pipeline is more practical for production teams managing multiple models.

Progressive experimental design clearly demonstrates the contribution of each step. TinyLlama with only domain-specific fine-tuning is already significantly better than few-shot GPT-4o, indicating that domain-specific fine-tuning can compensate for insufficient parameter count, though it still lags behind the teacher model. After adding knowledge distillation, the quality gap between student and teacher shrinks by over 90%, and inference speed becomes approximately 6 times the original. Adding Medusa-2 heads further brings the final model close to the 70-billion-parameter teacher in quality, with inference speeds reaching 10 to 20 times faster.

On Booking.com's three features, production deployment results vary in focus. Smart Filters: The KD-plus-Medusa combination improved response speed by 15 times for 99% of queries. AI Trip Planner: User engagement increased by 2.9% compared to GPT-3.5. Accommodation Description Generation: Users gave 'helpful' ratings 1.4% higher, at less than one-third the cost of GPT-4. This work previously won the Best Paper Award in the Industry Track at the Association for Computational Linguistics (ACL).

It should be noted that all performance figures above are self-reported by the team, and the blog post does not detail the specific datasets, metric definitions, or statistical significance tests used for quality evaluation. The 2.9% increase in engagement may be influenced by other product changes occurring simultaneously, making it impossible to attribute it solely to the model replacement based on this source alone. Furthermore, this pipeline targets specific domain tasks (travel dialogue, structured filtering, description generation); whether its conclusions can transfer to open-domain generation or other task types requires independent verification.

For engineering teams deploying LLM applications, the core transferable insight from this blog post is: model compression and inference parallelization are not mutually exclusive choices but orthogonal dimensions that can be stacked; integrating Medusa heads into the single-stage distillation training process is more practical than staged processing; however, the magnitude of specific benefits depends on task characteristics, hardware conditions, and baseline selections, requiring re-measurement in one's own scenario.

Sources

  • amazon.scienceAmazon Science↗
Continue researching →