L2 regularization
One way to discourage large parameters is to add λw² to the training objective. For the toy objective J(w) = (w − 2)² + λw², the minimizer is w* = 2/(1 + λ), for λ ≥ 0.
A trade-off, not a magic fix
Increasing λ pulls the optimum toward zero in this example, but also increases the original data-fitting loss. The best strength depends on the task and should be selected using suitable validation data. Increase λ and watch the optimum move toward zero. The penalty is one part of the objective; lower total loss in this toy does not establish better test performance.
INTERACTIVE EXPLORATION
Change the L2 penalty
For J(w) = (w − 2)² + λw², change λ. The readout shows the analytic optimum and its two objective components.
Leakage is a different problem
A feature calculated using future information, or a scaler fitted on the full dataset before splitting, can leak test information. A penalty on weights cannot repair that mistake; prevent leakage in the data pipeline and split.
Check your understanding
For λ = 1, what w minimizes (w − 2)² + λw²?
w = 1. The derivative is 2(w − 2) + 2w, which is zero at 1.