Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Shows that the RLHF objective can be optimised directly on pairs of preferred and rejected answers with a simple classification-style loss, with no separate reward model and no RL loop.

Why it matters

It matched or beat PPO-based alignment while being far simpler, and became a go-to alignment method for open models.

Read the paper: arXiv:2305.18290

Plain-language summary written by SolxLab. Read the full paper at the official link; all rights remain with the authors.