Break return into two parts
For a policy π, the state value is expected return when starting in a state and following π. In a one-step decomposition, it equals the expected immediate reward plus discounted value of the next state.
A deterministic example
If a transition always gives reward 2 and leads to a state with value 4, with γ = 0.5, then the one-step value is 2 + 0.5 × 4 = 4. In stochastic environments, take an expectation over actions and transitions. The diagram combines immediate reward with discounted next-state value for this deterministic example.
INTERACTIVE EXPLORATION
Look one step ahead
The immediate reward is 2 and next-state value is 4. Change γ in the deterministic one-step expression.
An equation, not a full learning algorithm
Bellman equations describe consistency relationships among values. Methods such as temporal-difference learning use samples to estimate values when the transition distribution is not known.
Check your understanding
With reward 2, next-state value 4 and γ = 0.5, what is the one-step value?
4.