ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
Long-term reading2 min readDisconfirmAI Editorial

InstructGPT Paper Summary: How Human Feedback Enables Language Models to Follow Instructions

This paper, published by the OpenAI team in 2022 titled "Training language models to follow instructions with human feedback…"

Verified 2026-11-16 16:14 GMT+8

This paper, published by the OpenAI team in 2022 and titled "Training language models to follow instructions with human feedback" (训练语言模型以遵循人类反馈的指令), raises a core question: why are larger language models not necessarily better at understanding user intent? The authors note that traditional large language models (such as GPT-3), despite their massive parameter counts, frequently generate unrealistic, toxic, or unhelpful content because their objective function is to predict the next word rather than "safely and helpfully follow user instructions." This misalignment between the model's objective and user intent is referred to as "misaligned."

To address this issue, the paper proposes a method called RLHF (Reinforcement Learning from Human Feedback). This method consists of three steps: first, supervised fine-tuning (SFT) of GPT-3 using prompts and demonstration data written by annotators to obtain a baseline model; second, collecting human preference ranking data for model outputs to train a reward model (RM) that predicts which output humans prefer; third, using this reward model as a signal to further fine-tune the supervised model via the PPO algorithm to maximize human preference scores. The resulting model is termed InstructGPT.

Experimental results show that in human evaluations, the InstructGPT model, with only 1.3B parameters, generated outputs more preferred than those of the 175B-parameter GPT-3, despite having 100 times fewer parameters. Additionally, InstructGPT showed significant improvements in truthfulness; for example, in the TruthfulQA benchmark, it generated true and informative answers approximately twice as often as GPT-3. However, the paper also notes limitations: InstructGPT still makes simple errors, such as fabricating facts or failing to detect instructions containing false premises; and there is performance regression on public NLP datasets, known as the "alignment tax," although this can be mitigated through updates that mix pretraining distributions (PPO-ptx).

Sign in / Sign up
ReadingResearchRadarInvestment framework

For readers seeking to deeply understand the foundational technologies of current large model alignment, this classic 2022 paper, "Training language models to follow instructions with human feedback," is essential reading, recommended starting from the introduction and methods sections.

Sources

  • arxiv.org《Training language models to follow instructions with human feedback》(2022)↗
Continue researching →