A loss over examples
Suppose two examples have losses ½(w − 1)² and ½(w − 3)². Their gradients are w − 1 and w − 3. Averaging both gives w − 2, the gradient of their average loss.
Batch versus minibatch
A full-batch update uses all examples for its gradient. A minibatch uses a subset, so its direction can differ from the full-batch direction. This introduces variability that depends on sampling; the small example here is deterministic. Find a parameter where the two example gradients point in different directions. Their average shows how a full batch can balance opposing suggestions.
INTERACTIVE EXPLORATION
Compare batch and individual gradients
Choose a parameter and see the gradient from each example and their mean. No parameter update is applied here.
Optimization is not evaluation
A decreasing training loss tells us the optimizer is fitting its objective. It does not prove the model generalizes. Initialization, learning rate, minibatch size and model architecture all influence training.
Check your understanding
At w = 0, what is the average gradient of w − 1 and w − 3?
−2. The two individual gradients are −1 and −3.