A small calculation
Given x₁ = 2 and x₂ = −1, weights w₁ = 0.5 and w₂ = 2, and bias b = 1, the weighted sum is 0.5 × 2 + 2 × (−1) + 1 = 0. Applying ReLU, max(0, z), gives 0.
Why activation matters
Two affine transformations composed together are still affine: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂). Adding layers without nonlinearities therefore does not create nonlinear decision boundaries. Follow the input through its weighted sum and the ReLU activation.
INTERACTIVE EXPLORATION
Turn a neuron on and off
Move the input x₂. Follow the weighted sum through the ReLU activation; other weights and inputs stay fixed.
What it does not mean
A nonlinear activation expands what a suitable network can represent; it does not guarantee that training finds a useful solution. Different activations have different behaviors and trade-offs.
Check your understanding
Can a stack of affine layers alone express a nonlinear boundary?
No. Their composition is another affine transformation.