Chapter 6 · The Transformer
Softmax
Softmax turns raw scores into probabilities that add up to 1, exaggerating the biggest.
Chapter 6 · The Transformer
Softmax turns raw scores into probabilities that add up to 1, exaggerating the biggest.
1. Raw scores: any size, even negative.
Softmax keeps the order of the scores, makes everything positive, and rescales so the total is exactly 100%.
A model scores each possible next word. The scores can be any number, positive or negative. They are not probabilities yet.
Drag a score. The percentages always add up to 100. Then try the temperature.
A favourite, but others still have a real chance. Total = 100%. Try “Add 2 to all”: nothing changes, because only the gaps between scores matter.
| Word | Score | e^score | ÷ 13.46 |
|---|---|---|---|
| cat | 2.00 | 7.39 | 55% |
| dog | 1.50 | 4.48 | 33% |
| fish | 0.20 | 1.22 | 9% |
| car | -1.00 | 0.37 | 3% |
Temperature divides every score before softmax. The list here has 4 words; a real model has tens of thousands.
What do the softmax outputs always add up to?
You add 5 to every score. What happens to the probabilities?
You lower the temperature. The top choice gets…