Chapter 6 · The Transformer
The residual stream
Each layer reads from and adds to a shared running vector per token, rather than replacing it.
Chapter 6 · The Transformer
Each layer reads from and adds to a shared running vector per token, rather than replacing it.
The token starts as a vector: the stream.
Each layer drops a small change into the stream. The stream always carries the token on, plus everything added so far.
Each token in a transformer has a list of numbers that travels from the first layer to the last. This running vector is the residual stream.
Tap a layer to turn it off or on. The coloured bars are the six numbers of the stream, shown after each step.
Remove just one layer, then compare how much the output changes:
Try it: with “Replace each time”, switching off one layer rewrites everything after it. With the stream, the token and the other layers’ notes stay in place. A simplified teaching version: six fixed layers, six numbers per token, nothing learned.
What does a layer do to the residual stream?
You switch off one layer. Which setup changes the final output less?
Why can early information reach the last layer?