Datadog Engineers: Benchmarking Before Fine-Tuning Reveals That the Base Model’s Problem Is Not Poor Summary Quality but Inability to Finish
Before fine-tuning Qwen 3.5 9B, the Datadog Patterns team used benchmarking to diagnose issues, discovering that the base model did not produce poor summaries but exhausted its token budget before completing them. They subsequently built a training dataset from production LLM spans and trained a LoRA adapter using Baseten Loops, reducing average generation time from 44.7 seconds to 0.89 seconds.
Datadog's Patterns feature is responsible for clustering an application's production LLM traffic into similar topics. Before clustering, it uses a large model to compress each sampled interaction into a single summary sentence. The team found that this step accounts for approximately 80% of the p99 cost and 50% of the p90 latency of a Patterns run. They decided to fine-tune a smaller model to replace it, but the critical first step was not to train directly, but to first understand exactly where the base model was failing.
The team used the pup finetune benchmark command to replay held-out test cases on Baseten, employing an LLM judge to determine if each answer could serve as a substitute for the original record. The results were surprising: the issue with the base model Qwen 3.5 9B was not summary quality, but token budget. It frequently spent its entire inference budget before writing the summary. Of 83 failures, 78 were due to token exhaustion or parsing errors, rather than content errors. When it did complete the task, the summary quality was good; it was simply too slow. This diagnosis pointed directly to what fine-tuning could solve: the model already knew how to perform the task but needed to change its response style—outputting short summaries directly instead of engaging in lengthy reasoning.
Once the nature of the problem was identified, the team pulled production LLM spans from Agent Observability using pup finetune dataset pull. Each span contained the input interaction, the clustering instruction, and the summary written by the frontier model, which was then converted into a training conversation: the input messages remained unchanged, and the summary served as the target answer. To prevent the model from learning only a single instruction, they mixed various clustering instructions to build the dataset. split the data into training, validation, and test sets by trace, ensuring all rows belonging to the same trace entered the same shard to prevent data leakage.