A label for every location
Semantic segmentation assigns a class to each pixel. Instance segmentation can distinguish separate objects of the same class. Exact conventions depend on the task and dataset.
Measure overlap
Compare the top row of ground-truth pixels with the predicted bottom row. Change the second prediction and watch the class-1 intersection and union. IoU is intersection divided by union; it summarizes overlap but does not show every boundary error. Toggle one predicted pixel. Compare the prediction with the fixed ground-truth row, then check how the foreground intersection and IoU change.
INTERACTIVE EXPLORATION
Change one pixel prediction
Ground truth is [1, 1, 0, 0]. Change the second predicted pixel; the others are fixed at [1, ?, 0, 0].
A tiny mask
Our experiment has four pixels with ground truth [1, 1, 0, 0]. Changing one prediction changes the intersection and union. This demonstrates the metric, not a trained segmentation network.
Check your understanding
For a perfect predicted mask, what is class IoU?
1, provided the class union is nonempty.