Chapter 7 · Large language models
Stop sequences and max tokens
Generation stops at an end token, a stop sequence or a maximum length.
Preparing stop sequences and max tokens
Chapter 7 · Large language models
Generation stops at an end token, a stop sequence or a maximum length.
Max length reached: cut off, even mid-sentence.
Whichever stop sign the model reaches first ends the reply.
You know the loop: predict a token, add it, predict again. Left alone, it would never end. Something has to say stop.
Pick a reply, then set a max token count and a stop sequence. Struck-through tokens were never shown. The model writes the same reply every time, so you can compare.
22 of 22 tokens shown.
A reply stops in the middle of a sentence. What is the likely cause?
What is a stop sequence for?
If you set a high max and no stop sequence, how does the model usually finish?