Chapter 7 · Large language models
Perplexity
Perplexity measures how surprised a model is by text; lower means it predicts the text better.
Chapter 7 · Large language models
Perplexity measures how surprised a model is by text; lower means it predicts the text better.
The model reads a sentence one word at a time.
Perplexity is how surprised a model is, on average, by the text it reads.
You know that a model gives each possible next word a probability. When the real word had a high chance, the model was not surprised. When it had a tiny chance, it was.
This tiny model has only read ten made-up sentences about cats and dogs. Type your own sentence, or press Scramble to reverse it. Each bar is how surprised the model was by that word.
Very predictable. The model barely blinks. Numbers under the bars are surprise in bits. The “end” bar is the model expecting the sentence to stop.
A model reads text it predicts very well. What is its perplexity like?
Why does scrambled text get a high score?
A perplexity of about 4 is easiest to picture as: