Factor a sequence
For three tokens A, B, C, the chain rule gives P(A, B, C) = P(A)P(B|A)P(C|A,B). The identity does not assume the tokens are independent. An autoregressive model approximates these conditional probabilities.
What training can measure
For an observed sequence, next-token cross-entropy averages negative log probabilities assigned to the observed next tokens. A smaller value on representative held-out data means the observed tokens received higher probability on average; it does not by itself establish factual reliability. The three boxes show successive conditional probabilities; their product, shown in the result, is the probability of this sequence.
INTERACTIVE EXPLORATION
Multiply conditional probabilities
Adjust P(C | A, B). P(A) = 0.5 and P(B | A) = 0.4 are fixed.
Generating is a separate decision
After obtaining next-token probabilities, a decoding method chooses a token, appends it to the context and repeats. Greedy choice and sampling can produce different outputs from the same model.
Check your understanding
If the three conditional probabilities are 0.5, 0.4 and 0.25, what is the sequence probability?
0.05: multiply 0.5 × 0.4 × 0.25.