Chapter 7 · Large language models
Top-k and top-p sampling
Sampling only from the top few tokens (top-k) or the smallest set covering p% (top-p) avoids nonsense.
Chapter 7 · Large language models
Sampling only from the top few tokens (top-k) or the smallest set covering p% (top-p) avoids nonsense.
The weather today is ___
1. Every word has odds. The tail holds odd words like “purple”.
Sampling draws a word by its odds. Top-k and top-p first remove the unlikely words, so the draw can only land on sensible ones.
A model gives every word some odds, even silly ones. Each odd word is rare, but there are many of them. Draw often and one will turn up.
Move k and p, then draw 40 words. Coral words are the silly ones. These odds are made up for teaching.
The weather today is ___
With top-k = 1, what does the model do?
Why is top-p called adaptive?
After the tail is cut, what happens to the odds of the survivors?