Two kinds of error
Training error measures fit on the examples used to adjust the model. Validation or test error estimates performance on held-out examples, assuming that the split reflects the situation we care about. A large gap can be evidence of overfitting.
Complexity and noise
A flexible model can learn real structure, but it may also fit noise or quirks of a small dataset. A simpler model can underfit. Data quantity, quality, regularization and the training process all affect this trade-off. Compare the same two rules on the training examples and new inputs. Both fit training here; only the rule y = x fits these held-out points.
INTERACTIVE EXPLORATION
Training success versus new inputs
Choose between two fixed rules on a tiny synthetic dataset. Both fit the training examples; compare how they behave on unseen inputs.
What changes in the real world?
Even a sound test result can fail to predict future performance when the deployment distribution changes. Good evaluation includes checking whether data, labels and collection conditions represent the intended use.
Check your understanding
If a model has 99% training accuracy, does that prove it generalizes?
No. Evaluate it on representative held-out examples and consider distribution shift.