Compare two responses
A preference pair identifies a preferred response and a rejected response for the same prompt. DPO uses these comparisons to optimize a language model relative to a reference model.
Look at relative log probabilities
The DPO objective compares how the model changes the log-probability gap between preferred and rejected answers relative to the reference. A coefficient controls the scale of that difference in the objective. The gap and β determine one pair’s loss; this does not run a model update.
INTERACTIVE EXPLORATION
Inspect a DPO loss term
Change the model-versus-reference log-probability gap. β = 1.
What this does and does not mean
DPO does not require a separately trained reward model or running PPO during its usual optimization. It is related to a particular RLHF formulation, but preference quality and evaluation remain important. The experiment shows a single loss term.
Check your understanding
Does ordinary DPO training require a separately trained reward model?
No.