An encoder and a decoder
A sequence-to-sequence system has an encoder that processes an input sequence and a decoder that generates an output sequence. The components and their exact architecture vary.
Generate one step at a time
An autoregressive decoder can predict one output token given the input representation and previous output tokens, then repeat until it emits an end marker. The output need not have the same length as the input. The boxes compare input and output lengths. They do not show an actual translation or a trained encoder and decoder.
INTERACTIVE EXPLORATION
Choose an output length
The input has three toy tokens. Change the number of output tokens to see how sequence lengths can differ.
What if one summary is too small?
Early encoder-decoder designs sometimes compressed the input into a fixed-size vector. Attention provided a way for a decoder to draw on different input representations at different output steps. This is historical context, not a requirement that all seq2seq models use RNNs.
Check your understanding
Must a translated output have the same number of tokens as the input?
No. The decoder can produce a sequence of different length.