Chapter 6 · The Transformer2017
The Transformer at a glance
'Attention Is All You Need' replaced recurrence with attention, so whole sequences are processed in parallel.
Chapter 6 · The Transformer2017
'Attention Is All You Need' replaced recurrence with attention, so whole sequences are processed in parallel.
Tokens go in at the bottom. Attention and feed-forward repeat. Scores come out at the top.
Language models read a sentence one word at a time, in order. That was slow, and they often forgot the start of a long sentence.
Tap any block of the diagram. Then change the number of layers.
Tokens look at each other and share information.
Every token decides how much to listen to every other token. This is how 'it' finds out what it refers to.
Real transformers also add small helpers (normalisation and shortcut links) around each block. They are left out here to keep the picture clear.
What did the transformer change about reading a sentence?
Which block lets tokens share information with each other?
What does stacking more layers do?