Chapter 5 · Turning language into numbers
Position: how models know word order
Attention alone ignores order, so models add position information to each token.
Chapter 5 · Turning language into numbers
Attention alone ignores order, so models add position information to each token.
Two sentences, very different meanings…
Attention looks at all words at once, like a pile of word cards. Position badges put the cards back in order.
Dog bites man and man bites dog use the same three words but say very different things. Order is part of the message.
Edit the sentence and press Shuffle. Switch between a model with no sense of position and one with position codes, and see whether it can tell the two orders apart.
These look identical without position. The model cannot tell them apart.
Each row is a position (0 to 19). Each of the 16 columns is a wave: fast on the left, slow on the right. Violet is positive, coral is negative.
Nearby positions have similar codes. Far apart positions drift apart.
This is the original sine-wave method with only 16 numbers per position. Real models use hundreds of numbers, and many newer models use other position methods.
Why do Transformers need positional encoding?
A model sees only the pile of words {bites, dog, man}. Which sentence is it?
What do you get by adding a position code to a word's embedding?