Beren: Why Low-Information RL Outperforms Pretraining in LLMs via Signal-to-Noise Ratio
Addressing the puzzle that Reinforcement Learning (RL) appears informationally inefficient compared to pretraining yet achieves rapid gains, researcher Beren proposes an explanation based on Signal-to-Noise Ratio (SNR) and bias-variance tradeoffs. The article argues that while RL provides fewer bits per sample, these bits are purely signal focused on task success, whereas pretraining's dense token-level loss contains significant noise irrelevant to the final objective. This framework explains RL's high sample efficiency and its extreme sensitivity to numerical precision and policy staleness in LLM contexts.
ImportanceStructuralEvidenceE3 inspectableWrite-upQuick
In Large Language Model (LLM) training, a persistent paradox has puzzled researchers: theoretically, Reinforcement Learning (RL) seems extremely inefficient because it yields only a single scalar reward per rollout, whereas pretraining computes loss for every token, providing massive amounts of information. Yet empirically, RL often takes a model from barely capable to near-perfect performance in just hundreds or even tens of steps, far outpacing pretraining in sample efficiency. In a recent post, researcher Beren offers a resolution to this contradiction through the lens of Signal-to-Noise Ratio (SNR) and the bias-variance tradeoff.
Beren's central thesis distinguishes between the quantity of information and its quality. He argues that while pretraining delivers a firehose of bits, most of these gradients are noise relative to the specific downstream task, as they focus on predicting the next token rather than achieving task success. In contrast, RL's few bits are pure signal, directly targeting whether the task was completed successfully. Thus, RL's efficiency stems not from information volume but from its high SNR, which filters out irrelevant distractions and concentrates optimization pressure on the actual goal.
This perspective also clarifies why iterative Supervised Fine-Tuning (SFT) cannot replace RL. Although SFT uses the model's own traces and provides token-level loss, it retains the Next Token Prediction (NTP) objective. Consequently, SFT gradients remain diluted by noise related to syntax, style, and other incidental features. Only when the loss function is strictly scoped to task outcome does the model rapidly converge to the parameter region that solves the task.