AWS: Sequence-Level Knowledge Distillation Compresses Shopping Agents to 4B Parameters, Preserving Reasoning Chains Is the Key to Success
The cost structure of agent applications differs from single-turn question answering. A shopping assistant completing one recommendation may undergo multiple steps—searching, reading results, narrowing scope, verifying constraints, and submitting an answer—in a multi-step loop, with each step constituting an independent billed call to a frontier model. At scale, this per-token billing loop makes deployment unviable. The solution offered by the AWS blog is sequence-level knowledge distillation (SeqKD): have the frontier teacher model generate complete multi-step trajectories, use a benchmark grader to filter for trajectories that actually solve the task, and then perform supervised fine-tuning on the small model using these trajectories. The teacher only needs to output text, requiring no weights, logits, or reinforcement learning infrastructure, so closed-source API models can also serve as teachers.
The most practically valuable finding in this blog is an ablation experiment: given the same 426 training problems and the same filtering pipeline, the only difference was whether the teacher's reasoning text was retained in the training objective. Removing reasoning and keeping only tool calls dropped the average absolute success rate from 0.541 to 0.415, a difference of about 13 percentage points. The authors explain that reasoning is not decorative narration but a causal chain connecting observations to next actions. Agents need to decide what to search for, how to filter, and when to terminate at each step, decisions driven by reasoning. If the student learns only the sequence of tool calls without the underlying reasoning, it achieves only partial transfer—preserving reasoning is the key to bringing the student close to the teacher's level.
However, SeqKD has a clear capability ceiling. The blog notes that what determines whether distillation succeeds is not the parameter gap between student and teacher, but whether the task itself is within the student's capability range. The teacher GLM-5.3 has approximately 40 billion active parameters, while the student Qwen3.5-4B has only 4 billion—a tenfold gap—but the shopping task is simple enough for a 4B model, so the student almost fully recovers the teacher's quality. Conversely, if the task exceeds the student's original capability, distillation will only produce long reasoning chains that are correctly formatted but contain wrong answers; trajectories lengthen while accuracy remains flat. The authors' recommended response is: upon seeing this symptom, stop adding data, either switch to a larger student or reduce task difficulty. Furthermore, SeqKD plateaus at the hardest tail of multi-step loops because the student cannot see the states entered after its own mistakes; breaking through this bottleneck requires on-policy methods, making SeqKD itself merely the first stage.
From a cost structure perspective, the significance of distillation extends beyond simply shrinking the model. Frontier APIs charge per token, requiring payment for every step of every loop, with costs growing linearly with traffic. The distilled 4B student is deployed on self-hosted endpoints, billed by instance hour, where increased traffic incurs no extra charge until scaling out is required. The authors do not provide specific per-task cost figures because those depend on your request patterns, sequence lengths, and concurrency levels, requiring self-measurement. Meanwhile, IFEval shows the student's general instruction-following capability dropped by about 6 to 8 percentage points, which is the expected cost of specialization—if the drop is significantly larger, it indicates the distillation narrowed the model too severely.
Sources:Amazon Web Services