Two kinds of input
A vision encoder and text encoder can map images and text into vectors. A training objective can encourage paired examples to have similar representations.
Compare candidates
A similarity score can rank candidate captions for an image. Softmax over scores yields relative weights for that candidate set, not a universal truth probability. Change one invented image–caption score and compare the two relative weights. The example illustrates competition between captions, not whether a caption is true.
INTERACTIVE EXPLORATION
Compare caption scores
Change the first of two invented image-caption similarity scores.
What is missing?
Vision-language systems vary widely: some align embeddings, others generate text conditioned on images. Our example adjusts invented similarity scores; it is not an image understanding system.
Check your understanding
If two candidate scores are equal, what are their two softmax weights?
0.5 and 0.5.