Create a prediction task
For example, a next-token objective can use the beginning of a sequence as input and a later token as the target. A masked-token objective instead hides selected tokens. These are different tasks with different information available to the model.
What is learned?
By optimizing such objectives over many examples, a model can learn representations useful for other tasks. Whether a representation transfers well depends on the data, objective, architecture and downstream task. The green words are visible; the orange word is the target. Move the cut to make a different training example from the same text.
INTERACTIVE EXPLORATION
Create a next-token target
Move the cut through “cats chase mice”. The visible prefix becomes input and the next word becomes the target.
Labels are not absent
The target is derived automatically from the data rather than manually annotated for the pretraining task. The source data can still contain errors, bias or sensitive material.
Check your understanding
For “cats chase mice”, if the input is “cats chase”, what is the next-token target in this simplified example?
“mice”, assuming these words are individual tokens.