Start with a sequence
A transformer receives representations of tokens, together with a way to incorporate position. A block combines self-attention with a position-wise feed-forward network, residual connections and normalization. Different transformer variants arrange these pieces differently.
Which tokens may a position use?
In a causal language model, a position must not use later tokens when predicting from the prefix. Select a query position below: earlier positions and the position itself are allowed; later positions are masked. The example assigns equal scores to all allowed tokens so their weights are equal. Choose a query position. Only the current and earlier positions are visible in this causal example; the equal visible weights follow from equal toy scores.
INTERACTIVE EXPLORATION
Apply a causal mask
Choose the query position in a three-token sequence. Allowed scores are all zero, so softmax gives them equal weight.
From a block to a language model
After multiple blocks, an output layer can produce next-token scores. A decoding method selects a next token and repeats. The small mask demonstration shows only one query and omits learned projections and the rest of the transformer.
Check your understanding
In causal self-attention, may the first token use a future token?
No. A causal mask prevents it from attending to later positions.