Chapter 14 · Images, sound and video2020
Vision transformers
Split an image into patches, treat each patch like a token, and run a transformer.
Chapter 14 · Images, sound and video2020
Split an image into patches, treat each patch like a token, and run a transformer.
A picture, like any other.
A vision transformer does this before any clever maths. The patches then flow through the same kind of attention you met for words.
A transformer reads a row of tokens. A sentence already is one. A picture is a grid of pixels, and treating every pixel as a token would be far too many.
Pick a starter picture or paint your own. Change the patch width. In Look mode, tap a patch to see its token.
With 4 by 4 patches your picture becomes 16 tokens. Halve the patch size and tokens grow four times, comparisons sixteen times.
A teaching version on a 16 by 16 picture. Real models cut photos, such as 224 by 224 pixels into 16 by 16 patches, then turn each patch into a vector with learned weights.
What is a token in a vision transformer?
You halve the patch size. What happens to the token count?
Why not make every pixel a token?