Chapter 7 · Large language models
Greedy decoding vs sampling
Greedy always takes the top token and can loop; sampling adds variety.
Chapter 7 · Large language models
Greedy always takes the top token and can loop; sampling adds variety.
Greedy: always the top word
Sampling: draw by the odds
Both start from the same word.
Same model, same start. Greedy walks one straight road and circles. Sampling takes a different road each time.
You know this from before: at each step the model gives odds for every next word. Now it has to turn those odds into one word.
Pick a start word and a length. Greedy never changes. Press New sampling run to roll again. This is a tiny teaching model trained on a few made-up sentences.
Greedy loop
Stuck repeating “the cat ate”. Variety: 18%
Sampling (run 1)
No loop. Variety: 59%
You run greedy decoding twice on the same prompt. What do you get?
Why can greedy decoding loop?
A word has 40% odds. How often does sampling pick it?