The interaction loop
At each step, an agent receives an observation, chooses an action according to a policy and receives feedback. An environment may be stochastic; an observation need not reveal the full state.
Future rewards matter
For a finite sequence of rewards r₁, r₂, r₃, one discounted return is r₁ + γr₂ + γ²r₃. The discount factor γ controls how strongly later rewards contribute to that defined return. The example below is not itself a learning algorithm. Set γ near zero and then near one. Later rewards matter less with a smaller discount; the reward sequence itself stays the same.
INTERACTIVE EXPLORATION
Change the discount factor
Rewards are fixed at [1, 2, 4]. Change γ and see each contribution to the discounted return.
Learning from experience
Value-based methods estimate future return; policy-based methods improve a policy directly. These approaches can be combined, and model-based methods use an environment model for planning or learning.
Check your understanding
For rewards [1, 2, 4] and γ = 0, what is the discounted return?
1: only the first term remains.