Compare with a baseline
An advantage function Aπ(s,a) = Qπ(s,a) − Vπ(s) compares action value with the state value under a policy.
Positive and negative
A positive advantage means an action is estimated to be better than the policy average at that state; a negative advantage means worse. These are estimates, not guarantees about one outcome. Subtract the baseline from the action value to obtain the advantage.
INTERACTIVE EXPLORATION
Compare an action to its baseline
Change Q while the state value V is fixed at 3.
Why baselines help
In policy-gradient estimation, suitable baselines can reduce variance without changing the expected gradient under the relevant assumptions. The experiment just subtracts two supplied numbers.
Check your understanding
If Q = 5 and V = 3, what is the advantage?
2.