A state that changes
An RNN updates a hidden state using the current input and the previous state. Think of the state as a summary available to the next step, rather than a perfect record of the entire past.
One tiny recurrence
Our experiment uses hₜ = a × hₜ₋₁ + xₜ, with h₀ = 0. Changing a controls how much of the previous state carries forward. This deliberately simple linear recurrence is not a trained RNN. Compare carry factors of zero and one. With later inputs fixed at zero, the three displayed states show how this particular recurrence retains or loses its initial signal.
INTERACTIVE EXPLORATION
Carry information forward
For inputs [2, 0, 0], change the carry factor a in hₜ = a × hₜ₋₁ + xₜ.
Why gates?
Training ordinary RNNs on long dependencies can be difficult because gradients can shrink or grow across many steps. LSTMs use gates and a cell state to help regulate information flow; they are more complex than the toy recurrence.
Check your understanding
If h₀ = 0, a = 0.5 and inputs are 2 then 0, what is h₂?
h₁ = 2 and h₂ = 0.5 × 2 = 1.