NVIDIA Releases Open Midtraining Recipe, Claims 3.8-Point Gain on SWE-bench for Nemotron Models
NVIDIA publishes an open recipe for the 'midtraining' stage between pretraining and fine-tuning, reporting significant self-tested gains in coding agent benchmarks.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
NVIDIA researchers have published an open recipe for "midtraining" on arXiv, addressing the data gap between pretraining and post-training (specifically Supervised Fine-Tuning or SFT) in frontier models. The study notes that pretraining typically uses short contexts with little agent interaction, while SFT relies mostly on synthetic data and can harm later RL trainability if too long. Midtraining fills this gap by training at 256K context length on a mixture of natural software-engineering data (repositories, PRs, commit histories) and a small share of agentic traces.
According to the paper's abstract, on the Nemotron 3.5 Lightning model, 500 billion midtraining tokens improved the SWE-bench Verified score from 60.7 to 64.5, and SWE-bench Multilingual from 55.8 to 62.8. On the larger Nemotron 3.5 Super, 280 billion tokens improved DeepSWE from 11.7 to 15.7. These results were achieved after identical full SFT runs.
The authors also report an evaluation protocol based on short fixed SFT runs, noting that base-model proxy metrics did not reliably correlate with post-training results. These are first-party benchmark claims by NVIDIA and have not yet been independently reproduced; the findings are specific to the Nemotron model family.