NVIDIA Releases Open Midtraining Recipe, Reports Gains in Coding Agent Benchmarks
NVIDIA publishes a recipe bridging pretraining and fine-tuning with real software data, reporting significant gains in coding agent benchmarks.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
NVIDIA researchers submitted an arXiv preprint on October 15 detailing an open "midtraining" recipe for large language models. The method aims to bridge the gap between short-context pretraining and synthetic-data supervised fine-tuning (SFT) by incorporating vast amounts of natural software engineering data to enhance coding agent capabilities.
The recipe trains models at a 256K context window using a mixture of packed repositories, pull requests, commit histories, build logs, and a small share of agentic traces. On NVIDIA's own Nemotron 3.5 Lightning model, 500 billion midtraining tokens improved the SWE-bench Verified score from 60.7 to 64.5, and SWE-bench Multilingual from 55.8 to 62.8. On the larger Nemotron 3.5 Super, the DeepSWE benchmark score rose from 11.7 to 15.7.
The study notes that traditional pretraining rarely sees agent interactions, while purely synthetic SFT data can hurt later RL trainability. Midtraining addresses this by mixing real-world code data. These results are currently self-reported by the authors and have not yet been independently reproduced.