ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Datadog Engineers: Benchmarking Before Fine-Tuning Reveals That the Base Model’s Problem Is Not Poor Summary Quality but Inability to Finish

Before fine-tuning Qwen 3.5 9B, the Datadog Patterns team used benchmarking to diagnose issues, discovering that the base model did not produce poor summaries but exhausted its token budget before completing them. They subsequently built a training dataset from production LLM spans and trained a LoRA adapter using Baseten Loops, reducing average generation time from 44.7 seconds to 0.89 seconds.

2026-11-18 08:00 GMT+8

Datadog's Patterns feature is responsible for clustering an application's production LLM traffic into similar topics. Before clustering, it uses a large model to compress each sampled interaction into a single summary sentence. The team found that this step accounts for approximately 80% of the p99 cost and 50% of the p90 latency of a Patterns run. They decided to fine-tune a smaller model to replace it, but the critical first step was not to train directly, but to first understand exactly where the base model was failing.

The team used the pup finetune benchmark command to replay held-out test cases on Baseten, employing an LLM judge to determine if each answer could serve as a substitute for the original record. The results were surprising: the issue with the base model Qwen 3.5 9B was not summary quality, but token budget. It frequently spent its entire inference budget before writing the summary. Of 83 failures, 78 were due to token exhaustion or parsing errors, rather than content errors. When it did complete the task, the summary quality was good; it was simply too slow. This diagnosis pointed directly to what fine-tuning could solve: the model already knew how to perform the task but needed to change its response style—outputting short summaries directly instead of engaging in lengthy reasoning.

Once the nature of the problem was identified, the team pulled production LLM spans from Agent Observability using pup finetune dataset pull. Each span contained the input interaction, the clustering instruction, and the summary written by the frontier model, which was then converted into a training conversation: the input messages remained unchanged, and the summary served as the target answer. To prevent the model from learning only a single instruction, they mixed various clustering instructions to build the dataset. split the data into training, validation, and test sets by trace, ensuring all rows belonging to the same trace entered the same shard to prevent data leakage.

Sign in / Sign up
ReadingResearchRadarInvestment framework
pup finetune

During the training phase, the team used the Baseten Loops SDK to train a LoRA adapter, measuring loss on the validation set after each epoch and stopping early when validation loss no longer improved, retaining the best checkpoint. In the evaluation phase, pup finetune evaluate replayed results from both the base and fine-tuned models simultaneously, scored by an LLM judge. On 150 held-out test cases, the fine-tuned model's judge pass rate increased from 44.7% for the base model to 93.3%, while average generation time dropped from 44.7 seconds to 0.89 seconds.

It should be noted that these are self-reported results from Datadog on a single summarization task. The base model's failure mode—token exhaustion during reasoning—is bound to the behavior of Qwen 3.5 9B on this specific task and may not generalize to other models or tasks. The pup finetune toolchain is deeply integrated with Datadog Agent Observability, and the specific process of converting production spans into training data would require adaptation under other monitoring systems. However, the steps themselves—diagnosing before training, building datasets from existing production traces, and splitting by trace to prevent leakage—are transferable engineering practices.

Sources

  • datadoghq.comDatadog↗
Continue researching →