Direct Preference Optimization

Direct Preference Optimization (DPO) is a method for aligning a language model with human preferences without training a separate reward model or running reinforcement learning. Rafailov et al. introduced it in 2023 (arXiv:2305.18290) as a replacement for the reward-model-plus-PPO pipeline used in RLHF. DPO reparameterizes the reward-maximization objective so the optimal policy has a closed form in terms of the reference model and the preference data. This lets the model be fine-tuned directly on pairs of preferred and dispreferred responses with a simple classification-style loss, skipping the separate reward model and the instability of on-policy reinforcement learning.

RLHF was established for LLMs by InstructGPT (Ouyang et al., 2022), building on earlier work by Christiano et al. (2017). It trains a reward model on human preference labels, then optimizes the policy against it with PPO. That pipeline is compute-heavy and prone to instability. DPO collapsed it into a single supervised-style stage, and its success led to a family of related objectives, including IPO, KTO, and SimPO, collectively called direct alignment methods. A 2026 survey (Raheja and Pochhi, arXiv:2601.06108) unifies these under three design axes: the preference model used, how deviation from the reference policy is regularized, and whether learning is online or offline.

Direct alignment methods are cheaper to run than RLHF but carry their own documented failure modes. These include reward overoptimization, length hacking (where longer responses are rewarded regardless of quality), mode collapse, and likelihood displacement (where probability mass shifts toward unintended completions). These failure modes remain active subjects of alignment research.