ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Score Centering: Eliminating Gradient Drift in RL Training with Additive Correction

Netflix researcher Cameron Wolfe analyzes the Score Centering method proposed by Marek and Ryabinin: In disaggregated RL architectures, probability discrepancies between the sampler and trainer generate a gradient drift term that depends only on the mean reward and carries no information distinguishing success from failure. Score Centering eliminates this drift by subtracting the expected score under the sampler, preserving the learning signal without introducing bias from clipping or masking.

2026-11-09 18:33 GMT+8

Reinforcement learning (RL) training for large language models (LLMs) typically employs a disaggregated architecture: an inference engine is responsible for generating rollouts, while a training engine computes policy gradients. These components run on different hardware, utilizing distinct precisions, kernels, and batch shapes; consequently, they produce different probabilities for the same token—a phenomenon known as Training-Inference Mismatch (TIM). Asynchronous training further exacerbates this issue, as policy updates may be based on rollouts that are already stale.

The core contribution of Marek and Ryabinin’s paper lies not merely in proposing a correction method, but in providing a deep analysis of why TIM causes instability. They demonstrate that under a constant reward setting, the theoretical policy gradient should be zero (since all actions receive the same reward, there is no learning signal). However, when TIM is introduced, the probabilities of the sampler q and the trainer p no longer cancel out, causing the expected gradient to become non-zero—the trainer begins to mimic the distributional differences of the sampler regardless of whether the reward provides meaningful information. This phenomenon is referred to as gradient drift.

More generally, by decomposing the policy gradient into covariance terms, two parts can be separated: the drift term equals the expected reward multiplied by the expected score of the ratio of sampler to trainer probabilities; the signal term is the covariance between the reward and the score. The drift term does not distinguish between successful and unsuccessful rollouts, relying solely on the mean of the rewards. In online RL, where the sampler is continuously refreshed from the trainer, drift forms a feedback loop: TIM leads to biased updates, which propagate to the sampler, which then produces new biased rollouts—errors accumulate gradually until training collapses.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Score Centering works by subtracting the expected score under the sampler from the token-level scores calculated by the trainer at each prefix position. This subtraction ensures that the centered scores have an expectation of zero under the sampler distribution, thereby completely eliminating the drift term in the covariance decomposition while leaving the signal term unchanged (since subtracting a constant does not affect covariance). Unlike importance sampling, this is an additive correction that avoids high-variance multiplicative weights and requires no clipping or masking to limit extreme values.

In practical implementation, precisely calculating the expected score over the entire vocabulary for the sampler is prohibitively expensive (LLM vocabularies typically exceed 150,000 tokens). The paper employs a top-k approximation: retaining only the 128 tokens with the highest sampler probabilities and approximating the remainder using scaled tail probabilities from the trainer. A key derivation utilizes the property that the expected score is zero under the trainer’s own distribution, allowing the correction term to be computed using only top-k tokens. Experiments show that k=128, and even k=32, achieve effects comparable to full-vocabulary approximation, though the guarantee of drift elimination is only approximate in these implementations.

Under mild TIM caused by quantization, Score Centering alone is sufficient to stabilize training. However, under severe staleness (such as in asynchronous training where the sampler synchronizes only once every 64 steps), Score Centering alone is insufficient—because the learning signal is still measured under the sampler distribution, the deviation from the trainer distribution requires correction via importance sampling. The paper suggests combining Score Centering with truncated or masked importance sampling when significant policy differences may arise between the trainer and the sampler.

Cameron Wolfe is a Deep Learning Researcher at Netflix holding a Ph.D., known for his communications that deconstruct AI research papers. He cites Nathan Lambert, noting that many frontier RL laboratories are already using this method, albeit previously unpublished.

Sources

  • cameronrwolfe.substack.com↗
Continue researching →