Chapter 6 · The Transformer
Multi-head attention
Several attention heads run side by side, each free to track a different kind of relationship.
Chapter 6 · The Transformer
Several attention heads run side by side, each free to track a different kind of relationship.
Grammar head: Links words by their job: adjectives to nouns, verbs to subjects and objects.
Each head draws its own links. A transformer runs many heads in parallel and combines what they find.
A word can relate to others in many ways at once: by grammar, by distance, by meaning. A single attention pattern would have to squeeze them all into one.
Type up to 8 words, tap a word, and switch heads on or off.
Simplified: these three heads are hand-written rules. In a real model the heads are learned during training and nobody tells them what to look for.
Why use several attention heads instead of one?
How are the heads' results used?
Do real heads get told what to look for?