Start with a pair
For the same prompt, a preference dataset can contain answers A and B with a label indicating which was preferred. A reward model assigns a score to each response, often trained so preferred responses receive higher scores.
Model a preference
A common illustrative pairwise model uses sigmoid(score A − score B). The probability is 0.5 when the scores are equal. The score itself is not a directly measured unit of truth or usefulness. When the scores are tied, the illustrative preference model assigns equal probability to both answers. Change one score to see the relative comparison move.
INTERACTIVE EXPLORATION
Compare reward scores
Adjust score A while score B stays at 0; see the resulting illustrative pairwise probability.
A proxy can fail
Preferences depend on who labels and on the task. Optimizing against an imperfect reward model can exploit its mistakes. Evaluation on held-out comparisons and other checks remains important; a reward model alone does not guarantee safe or factual answers.
Check your understanding
If A and B receive the same score, what probability does this pairwise model assign to preferring A?
0.5, because sigmoid(0) = 0.5.