Direct Preference Optimization: How Language Models Skip Reinforcement Learning to Achieve Human Alignment

Large unsupervised language models possess broad world knowledge but are difficult to precisely control in their behavior. Traditional methods such as Reinforcement Learning from Human Feedback (RLHF) align preferences by fitting a reward model and fine-tuning the policy using reinforcement learning, but this process is complex and unstable. This paper proposes Direct Preference Optimization (DPO), proving that this constrained reward maximization problem can be solved exactly through single-stage policy training, essentially reducing it to solving a classification problem on human preference data.

The core insight of DPO lies in leveraging the mapping relationship between the reward function and the optimal policy. It implicitly optimizes the same objective as existing RLHF algorithms (reward maximization with KL divergence constraints) but without explicitly fitting a reward model, sampling from the language model during the training loop, or requiring extensive hyperparameter tuning. This simplification makes the algorithm stable, high-performing, and computationally lightweight.

In terms of specific mechanisms, DPO relies on theoretical preference models such as Bradley-Terry to measure the fit between the reward function and empirical preference data. Unlike the traditional path of first training a reward model and then optimizing the policy, DPO defines the preference loss as a direct function of the policy via variable substitution. Given a dataset of human preferences for model responses, DPO directly optimizes the policy using a simple binary cross-entropy objective, avoiding explicit learning of the reward function or sampling from the policy during training.

Experiments show that DPO is at least as effective as existing PPO-based RLHF methods in sentiment adjustment, summarization, and dialogue tasks. Notably, fine-tuning with DPO exceeds the capability of RLHF in controlling generated sentiment and improves response quality in summarization and single-turn dialogues, while being significantly simpler to implement and train.