Chapter 6 · The Transformer2023
Beyond Transformers: state-space models
State-space models read text as a running summary, so cost grows gently with length instead of sharply.
Chapter 6 · The Transformer2023
State-space models read text as a running summary, so cost grows gently with length instead of sharply.
1 comparisons
1 updates
Attention lets every word look back at every earlier word. A state-space model reads left to right and keeps one rolling note.
Self-attention compares each word with all earlier words. Double the text and the number of comparisons roughly quadruples. Very long texts become slow and need a lot of memory.
First slide the text length. Then change how quickly the one-number note forgets.
After 11 more words, under 10% of that first word is left (33% after 5). Real models learn what to keep, so this is a simplified teaching version.
The text becomes twice as long. Roughly what happens to attention's comparisons?
What does a running state do for each new word?
What is the main trade-off of a fixed-size state?