Chapter 14 · Images, sound and video2024
Video generation
Video models generate frames that stay consistent over time, often as spacetime patches.
Chapter 14 · Images, sound and video2024
Video models generate frames that stay consistent over time, often as spacetime patches.
A video is a stack of pictures, one after another.
Treating each frame on its own is the easy way and the wrong way: nothing then ties one frame to the next.
At 24 frames a second, five seconds is 120 pictures. Making each one beautifully is not enough: they must agree with each other.
With no memory each frame is invented afresh. With full memory the ball keeps its colour and moves steadily. Watch the flicker meter.
Part memory: the shape mostly survives, but the wobble gives it away. Real models reduce this by looking at many frames at once. At 512 × 288 that clip is about 3,456 tokens, grouping two frames per patch.
A teaching toy: a drawn ball, not a real video model. The token numbers are illustrative, to show how fast they grow with time.
Why is video harder than a single picture?
What is a spacetime patch?
Why are generated clips usually short?