A question with a measurable answer
Suppose the loss for one parameter is L(w) = (w − 3)². At w = 0, the loss is 9. We want to reduce it, but pretend we do not already know the minimum. The derivative, 2(w − 3), tells us the local slope.
Take one step
Gradient descent moves opposite that slope: w ← w − η · 2(w − 3). Starting at w = 0 with η = 0.1, the slope is −6, so the next w is 0.6. In the experiment, take several steps. Then reset and try η = 1: the value alternates between 0 and 6. Try a small learning rate and take several steps. Reset and try a larger one: compare the actual parameter and loss values, not only the position of the plotted dot.
Try gradient descent
Minimize L(w) = (w − 3)², starting at w = 0. Change the learning rate and take a step.
The dot is clamped to the visible chart when the parameter moves beyond its range. Values show the actual parameter.
Why does the step size matter?
A step that is too large can jump past a minimum and even move farther away. A very small step can take many updates. The precise boundary here is specific to this bowl; neural-network losses can be multidimensional, noisy and non-convex.
Check your understanding
Starting at w = 0, what is the first update with η = 0.1?
The gradient is −6, so w becomes 0 − 0.1 × (−6) = 0.6.