Wait for a return
Monte Carlo methods can update a value estimate from a completed sampled return. A single return is noisy, so repeated observations can improve the estimate.
Bootstrap from the next state
A one-step TD target uses reward plus discounted estimated next-state value. It can update before an episode finishes, but relies on another estimate. Compare the current estimate with the value after one step toward the fixed target.
INTERACTIVE EXPLORATION
Move toward a target
For V = 2 and target T = 4, change learning rate α.
A concrete update
With value V, target T and learning rate α, an incremental update is V ← V + α(T − V). The experiment compares two hand-specified targets; it does not simulate an environment.
Check your understanding
If V = 2, T = 4 and α = 0.5, what is the updated value?
3.