Chapter 6 · The Transformer
Logits
The model's final layer gives a raw score (logit) for every token in the vocabulary; softmax turns them into probabilities.
Chapter 6 · The Transformer
The model's final layer gives a raw score (logit) for every token in the vocabulary; softmax turns them into probabilities.
The cat sat on the ___
1. The model has a score slot for every word.
A logit is just a raw score. It can be negative or any size. It is not a percentage yet.
After all its layers, the model ends with one number for every token it knows. A real model has tens of thousands of tokens, so the row is very long.
Drag the logit of each word for “The cat sat on the ___”. These scores are made up for teaching.
Greedy pick: mat. mat is 3.0× as likely as sofa, because its logit is 1.1 higher.
Try “Add 3 to all”: every logit moves, but the percentages stay put. Only the gaps count.
What is a logit?
You add 10 to every logit. What happens to the percentages?
Word A has a logit 1 higher than word B. Roughly how much likelier is A?