Collect preferences
One approach collects comparisons of model responses and trains a reward model to predict preferences. The feedback depends on annotators, instructions and the sample of prompts.
Optimize a policy
Reward-model-based RLHF can optimize a language model to score well according to a learned reward while controlling deviation from a reference policy. PPO is one possible optimizer, not a definition of RLHF. The two quantities combine by subtraction. This is a deliberately simplified objective, not a complete RLHF algorithm.
INTERACTIVE EXPLORATION
Balance a toy objective
For reward 3 and penalty λ × 2, change λ. This is a toy objective, not full RLHF.
Watch the proxy
A learned reward is an imperfect proxy for human goals. Higher reward-model score alone does not establish truthfulness or safety. The experiment evaluates a toy reward-minus-penalty objective, not a complete RLHF system.
Check your understanding
Does a higher reward-model score prove an answer is factually correct?
No. The reward model is an imperfect proxy.