ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

NVIDIA Nemotron Technical Report Reveals Actual Mechanisms of Large Model Post-Training and Multi-Teacher Distillation Solutions

AI researcher Cameron Wolfe, based on the NVIDIA Nemotron series of technical reports, outlines three core methods for large model post-training—supervised fine-tuning, reinforcement learning, and multi-teacher online distillation—and explains the optimization and infrastructure bottlenecks faced by multi-domain reinforcement learning, as well as how multi-teacher online distillation emerges as a solution.

2026-09-28 17:33 GMT+8

Open-weight models provide a window into understanding the development process of frontier large language models (LLMs), but most releases provide only the model itself, leaving training details closed. AI researcher Cameron Wolfe pointed out in Deep Learning Focus that the NVIDIA Nemotron series is one of the most transparent resources among them: its releases are often accompanied by detailed technical reports, code, training recipes, and sometimes even data. This provides a rare opportunity to directly study the actual mechanisms of LLM post-training.

Wolfe categorizes post-training methods into three core components: supervised fine-tuning (SFT), reinforcement learning (RL), and multi-teacher on-policy distillation (MOPD). The first two are well-known, while MOPD is a newer component that has recently appeared frequently in model reports; reports such as Nemotron 3 Ultra and DeepSeek-V4 have both adopted it. Although the specific recipes may vary significantly across different model families, these three underlying building blocks continue to be used over time.

SFT establishes a foundation through a generate-then-filter pipeline: prompts are collected from multiple domains, candidate trajectories are sampled using strong teacher models, and high-quality samples are filtered using domain-specific verifiers—string matching or symbolic verification for math problems, test cases for code, and LLM judges for open-ended questions. A single prompt can correspond to multiple valid trajectories; diversity selection avoids repetition, and it is common practice to refresh datasets by re-running the pipeline with updated teacher pools as new models are released.

Sign in / Sign up
ReadingResearchRadarInvestment framework

RL enables models to demonstrate capabilities beyond those shown in teacher trajectories through exploration and learning via task rewards. Wolfe emphasizes a key practical finding: the data used for RL is often more important than the algorithm itself. Specifically, multiple responses are sampled for each prompt, pass rates are calculated, and these serve as proxies for difficulty—prompts already solved 100% by the current policy are typically completely filtered out because they cannot provide meaningful learning signals; prompts with very low pass rates may be noisy or unsolvable, making them more suitable for later stages of training, which naturally leads to curriculum learning.

The true bottleneck arises when scaling RL to a large number of domains. Wolfe describes difficulties at two levels: at the optimization level, each new domain occupies part of the training batch; as the number of domains increases, sampling for individual capabilities decreases, weakening learning signals, and domains may interfere with each other—updates improving one capability may degrade another. At the infrastructure level, trajectory lengths, verification costs, and environment requirements vary drastically across domains; a math problem might generate short answers for quick verification, while software engineering agent tasks may require hundreds of interaction turns followed by executing numerous test cases. Mixing such tasks into the same batch leads to idle GPUs waiting—the so-called straggler problem.

Some models in the Nemotron series use Cascade RL to separate incompatible domains into sequential training stages, while others perform unified multi-domain RL on asynchronous infrastructure. Wolfe notes that neither fully solves the problem: unified RL struggles to balance as domains increase, while sequential RL may lead to gradual degradation of capabilities from earlier stages.

Multi-teacher on-policy distillation (MOPD) is precisely the solution to this dilemma. The idea is not to force a single policy to become an expert in all domains simultaneously, but rather to first train multiple specialized teachers—one focusing on software engineering, another on conversational applications—each independently optimizing within narrow domains without needing to balance multiple domains. These capabilities are then consolidated back into a single student model via online distillation: the student generates trajectories, which are routed to corresponding teachers; teachers assign probabilities to each token generated by the student, and the student is trained to match the teacher using a sampled reverse-KL objective. Unlike outcome-based RL, which assigns sparse rewards to entire trajectories, MOPD provides dense learning signals at every generated token, potentially making training significantly more sample-efficient.

Wolfe also observed that post-training recipes overall are evolving towards more agentic approaches: early SFT datasets consisted mainly of independent Q&A pairs, whereas now complete agent trajectories—including steps like searching repositories, calling tools, executing code, and deciding next actions—are increasingly incorporated. RL has similarly evolved from simplifying complex tasks into standalone prompts with cheap rewards, to end-to-end agentic RL—where policies interact in sandboxed environments, call tools, and traverse multi-step timelines to reach final solutions, judged by comprehensive test cases executed in the environment. At scale, this requires launching and maintaining thousands of isolated environments, sharply increasing infrastructure complexity.

Sources

  • cameronrwolfe.substack.com↗
Continue researching →