Chapter 5 · Turning language into numbers
Tokens
Models do not read letters or words; they read tokens, small chunks of text with ID numbers.
Chapter 5 · Turning language into numbers
Models do not read letters or words; they read tokens, small chunks of text with ID numbers.
Plain text…
A model never sees letters. It sees a list of numbers, one per token. A dot (·) marks a space that belongs to the token.
Before a model can read anything, a tokenizer cuts the text into chunks called tokens. A token can be a whole word, part of a word, a space plus a word, or a single symbol.
Type anything. Try long words, numbers, emoji or another language and watch the token count.
This is a teaching tokenizer. Real ones learn their splits from data, so exact pieces and IDs differ between models.
What does a language model actually receive?
Roughly how many English characters make one token?
Why might the same message cost more in Hindi than in English?