Chapter 5 · Turning language into numbers2016
How tokenizers are built (BPE)
Byte-pair encoding starts with characters and keeps merging the most frequent pairs into new tokens.
Chapter 5 · Turning language into numbers2016
Byte-pair encoding starts with characters and keeps merging the most frequent pairs into new tokens.
Start: every character is its own token.
Every round, the most common neighbouring pair becomes one new token. Frequent chunks like “lo” and “low” become tokens of their own. A dot (·) marks the start of a word.
Begin with single characters. Any text can be written with them, so nothing is ever unknown. Real tokenizers usually start from raw bytes for the same reason.
Press Merge once and watch the pair table. The top row is the pair that will merge next. Try text with repeated words, then change it to something with no repeats.
None yet.
This is real BPE, simplified: it works on characters and learns from only your short text. Real tokenizers train on huge amounts of text and usually work on bytes.
What does BPE do in each round?
As you do more merges, what happens to the token count of the same text?
Why start from single characters (or bytes)?