Why should a word look around?
The word “bank” can mean different things in “river bank” and “bank account”. A model needs a way to combine information from context. Attention provides a calculation for mixing value vectors using weights based on a query and keys.
From scores to a weighted sum
In the experiment, two values are fixed at 2 and 6. Softmax turns two scores into weights that sum to one. Change the first score: the first weight grows, and the output moves toward 2. When the weights are 0.25 and 0.75, the output is 0.25 × 2 + 0.75 × 6 = 5. Increase the first key’s score and follow the thicker connection. More weight on the first value pulls the weighted output toward 2 rather than 6.
INTERACTIVE EXPLORATION
Change what the query attends to
Two keys have scores s and 0. Their scalar values are fixed at 2 and 6. Change s and watch the weighted output.
What the picture leaves out
Actual attention often uses learned query, key and value projections, a scale factor, masks and multiple heads. The illustration uses scalar values and hand-chosen scores. Its weights should not be read as an explanation produced by a trained model.
Check your understanding
If the weights are 0.25 and 0.75 and the values are 2 and 6, what is the output?
0.25 × 2 + 0.75 × 6 = 5.