Chapter 6 · The Transformer
Causal masking
Language models may only look at earlier tokens, never future ones, so they can generate left to right.
Chapter 6 · The Transformer
Language models may only look at earlier tokens, never future ones, so they can generate left to right.
Row “the”: it can look at 1 token. The grey × cells are the future, which stays hidden.
Each row is a token that is looking. It may look at itself and everything to its left, nothing to its right.
A language model reads a text and, at each spot, guesses the next token. To learn, it is shown real text, so the right answer is sitting right there in the data.
Pick a sentence and a token. See what that token can see, what the model guesses next, and how it does when it writes on its own.
“cat” can see 2 tokens: the, cat. The 4 later tokens are hidden.
The mask is on: the score is lower but honest, and the model can write left to right.
A simplified teaching version: attention scores are hand-made and the predictor counts word pairs from six sentences. Words outside that small list are unknown to it.
Token 3 in a sentence can see…
Why is peeking at the next token a problem in training?
What does the mask do to the hidden scores before softmax?