Chapter 6 · The Transformer
Self-attention
Every token looks at every other token and decides how much each matters to its meaning.
Chapter 6 · The Transformer
Every token looks at every other token and decides how much each matters to its meaning.
“it” looks at: cat 91%, because 1%
Thicker beams mean more attention. Each word decides for itself who matters.
In “The cat sat because it was tired”, the word “it” means nothing alone. To know what it stands for, it must look at the other words.
Type a sentence or pick an example. Tap a word to see where it looks.
“it” pays most attention to “animal”. All the bars add up to 100%.
Simplified: the numbers are hand-made for common words, so “didn't”, “cross” get neutral values. A real model learns its own.
In “The cat sat because it was tired”, why does “it” need attention?
One word's attention weights over all words add up to…
What does a thick beam between two words mean?