Tokenization happens before the network
A tokenizer maps text into a sequence of token IDs. A token can represent a word, part of a word or other text unit depending on the tokenizer. Token IDs are arbitrary identifiers, not meaningful numerical magnitudes.
A learned lookup
An embedding table associates each token ID with a vector. If the table has vocabulary size V and embedding width d, it can be represented as a V × d matrix. Different IDs select different rows. Look up the token in the invented table, then project its vector. The coordinates are not a trained embedding.
INTERACTIVE EXPLORATION
Look up a token vector
Choose a token in a tiny made-up embedding table and project its two coordinates onto [1, −1].
From token identity to context
In a simple model the initial lookup for the same token is identical across contexts; later layers can change the representation based on surrounding tokens and positional information. A small illustration cannot capture the rich structure of a trained embedding space.
Check your understanding
Does token ID 8 inherently mean twice as much as token ID 4?
No. IDs are lookup keys; their numerical ordering does not encode semantic distance.