Preprint: RRI Improves Self-Improvement on Open-Ended Tasks via Adversarial Rubric Repair

Recursive Self-Improvement (RSI) typically relies on tasks with automatically verifiable answers, such as mathematics or coding. However, in open-ended expert domains like clinical report generation, the lack of ground truth makes simple question-answer generation unviable, and model-written rubrics are prone to exploitation where solvers list expected topics without substance.

Wei Dai et al. introduce RRI (Recursive Reward Improvement), which employs a 'reward hacking adversary' to supervise the rubric. Before training, the adversary attempts to write an answer that games the rubric while being useless, while the same model writes an honest answer without seeing the rubric. If the adversary wins, up to two repair steps revise the rubric to ensure the honest answer prevails.

On the HealthBench Professional benchmark, RRI improved the adjusted score over the Generation Only baseline by 10.9 points for 9B models and 12.4 points for 27B models. The method also showed transfer gains of 6.6 points on PRBench Hard and 8.0 points on ProfBench for the 27B model, despite these tasks not being seen during training.

These results are from first-party author testing and have not yet been independently reproduced.