From input to output
A simple layer computes a = σ(Wx + b). The matrix W and bias b are parameters, while σ is an activation function. An output layer then produces scores or other quantities appropriate to the task.
Why layers help
Hidden units let a model combine intermediate features. In an image classifier, early computations may respond to simple patterns and later ones to more complex combinations, though individual units need not map neatly to human-named concepts. Follow the input to the two hidden activations, then to their combined output.
INTERACTIVE EXPLORATION
Follow values through a small network
The hidden layer computes ReLU(x) and ReLU(−x). Change x and the second output weight to see which paths contribute.
Architecture versus learning
A network’s structure describes the computation. Training chooses parameters using data and an objective. Backpropagation computes gradients of the objective; an optimizer uses those gradients to update parameters.
Check your understanding
Does drawing a network diagram tell us its learned weights?
No. Architecture specifies the computation; weights must be initialized and learned or otherwise supplied.