Chapter 5 · Turning language into numbers
Tokens across languages
Some languages need more tokens per word, so the same message costs more and fits less.
Chapter 5 · Turning language into numbers
Some languages need more tokens per word, so the same message costs more and fits less.
“Good morning, how are you today?” in each language, counted by the teaching tokenizer.
All six lines say the same thing. A tokenizer built mostly from English text cuts other scripts into many small pieces, so their bars grow longer.
A tokenizer learns its vocabulary from text. If most of that text is English, English words become single tokens, while words in other languages are split into many small pieces.
Type a message in any language and its English meaning. The tool counts tokens for both and shows the ratio. Then slide the window to see how many messages fit.
Every count here comes from our simple teaching tokenizer, which is built around English. Real tokenizers differ by model, and many split other languages into far fewer pieces than this toy does.
Why might the same message use more tokens in Tamil than in English?
What does a higher token count cost you?
Our counts here come from…