NVIDIA Researchers Propose Double-Rollout RL, Boosting Small Model Accuracy to 95.4% on AIME
An NVIDIA preprint shows that estimating rewards via sampled continuations from truncated traces significantly improves reasoning model accuracy and token efficiency in few steps.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
Researchers at NVIDIA have proposed "Double-Rollout Reinforcement Learning" in an arXiv preprint to address credit assignment noise in Reinforcement Learning with Verifiable Rewards (RLVR). Instead of relying on Process Reward Models (PRMs), the method truncates each sampled reasoning trace, generates multiple continuations from the cut-off point, and replaces the original outcome reward with the mean reward of these continuations, yielding a more robust estimate of reasoning quality.
Experimental results indicate that when training the Nemotron-3-Nano-30B-A3B model using only DAPO data, this approach reaches 95.4% accuracy on the AIME 2026 benchmark within 30 reinforcement learning steps, surpassing the baseline GRPO algorithm's 91.5%. The study notes that this improvement comes with only a moderate increase in compute cost. Furthermore, when combined with length-based rewards, the algorithm improves the accuracy-token efficiency Pareto frontier compared to existing optimizations.