Chapter 14 · Images, sound and video2021
CLIP: matching images and text
CLIP learns a shared space where images and captions that match sit close together.
Chapter 14 · Images, sound and video2021
CLIP learns a shared space where images and captions that match sit close together.
Pictures (squares) and captions (circles) start scattered. Colour pairs them up.
Nobody tells the model what a sun is. It only learns which picture goes with which caption, and the rest follows.
One network turns a picture into a list of numbers. Another turns a caption into a list of the same length. Both lists are points, or arrows, in the same space.
Train the scattered toy space and watch pairs meet. Then type your own caption and see which drawn picture it points towards.
The diagonal (outlined) shows each picture against its own caption. Training raises the diagonal and lowers every other cell. Pairs that already match are pushed apart from the rest, which is what keeps the space useful.
A teaching toy: two-dimensional arrows and hand-made word meanings. A real model has hundreds of dimensions and learns the meanings itself.
What does a high similarity score mean?
During training, what happens to a wrong picture and caption pairing?
How can the model label a new picture of a cat without cat training?