Compare old and new
PPO uses a probability ratio between a new policy and the policy that generated data. Its clipped objective limits the incentive from some changes beyond a chosen range.
Understand clipping
For a single sample, the clipped surrogate uses min(rA, clip(r, 1−ε, 1+ε)A). The sign of advantage A matters. Clipping this objective is not a strict guarantee that every probability or the total policy change stays within the range. Compare the original ratio, its clipped version, and the one-sample objective for positive advantage.
INTERACTIVE EXPLORATION
Inspect one clipped term
Fix advantage at 1 and ε at 0.2. Change the new-to-old probability ratio.
Not the entire algorithm
Training also involves sampled data and other implementation choices. The experiment evaluates one surrogate term, not a full PPO update.
Check your understanding
With positive advantage 1, ratio 1.5 and ε = 0.2, what is the clipped surrogate term?
1.2.