Chapter 14 Β· Images, sound and video
Multimodal LLMs
Modern models take text, images and audio in the same context by turning each into tokens.
Chapter 14 Β· Images, sound and video
Modern models take text, images and audio in the same context by turning each into tokens.
Three different inputs: a picture, a sound and some words.
A multimodal model has one trick: whatever comes in, cut it up and turn each piece into a vector. After that, it is all the same stream.
Words become word-piece tokens. A picture becomes patch tokens, as in the vision transformer lesson. Sound becomes slice tokens, as in the speech lesson.
Write something, add pictures and sound, and watch the bill. The numbers are illustrative: real systems differ.
A 1600 Γ 1200 picture is scaled to 1024 Γ 768 before it is cut into patches.
The pictures are doing the spending: 3,080 tokens against 9 for your words. Halving a picture's width and height cuts its tokens to about a quarter.
Illustrative numbers: 16 pixel patches, pictures scaled to fit 1024 pixels, 25 tokens a second of sound, and a rough word-piece count. Real systems each have their own.
How does a model read a picture and a sentence together?
You halve a picture's width and height. What happens to its tokens?
Why do a few large photos crowd a conversation?