The prediction task
An autoregressive language model produces a distribution over the next token given previous tokens. During pretraining, the observed next token provides a training target. Tokens need not be complete words.
Learning from many contexts
Reducing next-token loss can encourage a model to use patterns in its context. This does not guarantee factual accuracy, reasoning ability, or a particular internal representation. Raise one candidate logit and watch the three next-token probabilities redistribute. This is one invented prediction step, not a running language model.
INTERACTIVE EXPLORATION
Choose the next-token distribution
For an invented three-token vocabulary, change one score and observe the resulting probabilities.
How is output generated?
A decoding procedure chooses a token from the predicted distribution and repeats. Greedy decoding always chooses the highest-probability token; sampling is a different choice. The experiment uses invented probabilities, not a trained language model.
Check your understanding
Does a high next-token probability guarantee that a statement is true?
No. It is a model probability conditioned on its context, not a fact check.