What a record contains
A simple preference record includes a prompt, two responses and a preference label. The label reflects the annotation process and may be noisy, inconsistent or dependent on the instructions given to reviewers.
Modeling a comparison
One illustrative model assigns scores sA and sB and uses sigmoid(sA − sB) as the probability that A is preferred. Equal scores give 0.5. Scores here are arbitrary illustrative values, not measurements of quality. Make the two scores equal, then increase score A. The comparison changes because the score difference changes, not because either score proves factual correctness.
INTERACTIVE EXPLORATION
Compare two scores
Adjust score A while score B remains 0. The displayed sigmoid of their difference is an illustrative pairwise preference model.
How preferences are used
A learned reward model can use comparisons, and methods such as DPO use preference pairs in a different objective. Neither a single preference label nor a high model score guarantees that an answer is truthful or safe.
Check your understanding
If the two scores are equal, what does sigmoid(sA − sB) equal?
0.5, because the score difference is zero.