A parameter controls a choice
Consider two actions. Let p = sigmoid(θ) be the probability of A; B has probability 1 − p. If expected rewards are 0.8 for A and 0.2 for B, expected one-step reward is J = 0.8p + 0.2(1 − p).
Follow the gradient
For this toy problem dJ/dθ = 0.6p(1 − p), so a small gradient-ascent step increases θ. This is an exact derivative of the simple known-reward example, not a general policy-gradient estimator. Changing θ changes the action probability and the exact gradient in this one-step example; it does not perform an update.
INTERACTIVE EXPLORATION
Change a policy parameter
For p(A) = sigmoid(θ), rewards 0.8 and 0.2, inspect expected reward and its exact derivative.
Real feedback can be noisy
In RL the expected return is usually unknown and estimated from sampled trajectories. Policy-gradient methods estimate how parameters affect expected return; baselines and value estimates can help manage variance.
Check your understanding
If both actions had equal expected rewards, what would the toy objective’s gradient be?
Zero: changing the action probabilities would not change expected reward in this one-step example.