A branch learns a change
If F(x) = ax, then y = (1 + a)x. When a = 0 the block is an identity mapping. This does not mean that every trained residual block is an identity, only that this configuration is easy to express.
What about gradients?
For the simple scalar example, dy/dx = 1 + a. The direct addition contributes a path with derivative 1, while the residual branch contributes a. The total can still be small or zero if the contributions cancel. The diagram lists the input and branch contribution before their sum; the original input takes the direct path.
INTERACTIVE EXPLORATION
Adjust a residual branch
Change the scalar a in F(x) = ax. Watch the direct and residual paths combine for x = 2.
Why use them?
Residual connections helped optimize substantially deeper image models in the ResNet work. They also occur in other architectures, including transformers, but their detailed placement and normalization differ.
Check your understanding
If F(x) = 0, what does y = x + F(x) equal?
y = x: the block is an identity mapping.