A value for a state and action
Q(s,a) estimates a return associated with taking action a in state s. The particular Q function and its target depend on the policy or control objective.
One Q-learning target
A tabular Q-learning update uses target r + γ maxₐ′ Q(s′,a′), then moves the current estimate toward that target. Q-learning is an off-policy control method. Add the reward to the discounted best next-state estimate to form the target.
INTERACTIVE EXPLORATION
Calculate a Q target
Reward is 1 and next-state Q values are 2 and 4. Change γ.
What the example assumes
The experiment supplies two next-state Q values directly. Real learning must gather transitions, explore, and address estimation and data issues.
Check your understanding
For reward 1, γ = 0.5 and best next Q = 4, what is the target?
3.