Two jobs
The actor represents a policy. The critic estimates a value or a related quantity that can help the actor improve.
A learning signal
Some actor-critic methods use an estimated advantage or TD error as part of the actor’s update. Implementations differ; neither component is automatically correct. Add the reward and discounted next-state value, then subtract the current value.
INTERACTIVE EXPLORATION
Calculate a TD error
For V(now) = 2, V(next) = 4 and γ = 0.5, change the reward.
A small example
For a one-step TD error δ = r + γV(s′) − V(s), changing a reward changes the signal. Our experiment uses supplied numbers and does not update a network.
Check your understanding
For r = 1, γ = 0.5, V(next) = 4 and V(now) = 2, what is δ?
1.