Chapter 14 · Images, sound and video2022
Latent diffusion and text-to-image
Denoising in a compressed 'latent' space, guided by text, made fast text-to-image possible.
Chapter 14 · Images, sound and video2022
Denoising in a compressed 'latent' space, guided by text, made fast text-to-image possible.
Start with a picture: 144 pixels.
The picture is never touched during the hard part. The model works on a small code, and the decoder paints the pixels at the end.
Denoising every pixel of a large image, many times over, is slow. Most pixels carry little information: neighbouring pixels are usually similar.
Pick what to ask for, then turn up the guidance. Watch the single result, and the eight results made from eight different starting noises.
Eight runs from eight different starting noises, at guidance 2.0:
Strong guidance: every run obeys the prompt, but the eight results look much alike. Obedience bought with variety.
A teaching toy: nine-number latents and six known pictures, with the real guidance formula. A real model has a much larger latent and learns the pictures itself.
Where does the denoising happen in latent diffusion?
What does turning the guidance scale up do?
When are pixels actually drawn?