Chapter 5 · Turning language into numbers
Vocabulary and token IDs
A tokenizer's vocabulary is a fixed list (often 50k–200k entries); each token has an ID number.
Chapter 5 · Turning language into numbers
A tokenizer's vocabulary is a fixed list (often 50k–200k entries); each token has an ID number.
Found The → ID 44636
The vocabulary is just a very long numbered list. The tokenizer finds each chunk in the list and hands the model its number.
Before a model is trained, its makers build a vocabulary: a list of tokens, each with its own number. It never changes afterwards. Real vocabularies usually hold tens of thousands to a couple of hundred thousand entries.
Type words and see whether each is a single vocabulary entry or split into pieces. Try names, long words and made-up words.
A leading space belongs to the token, so “ cat” and “cat” have different IDs (the dot · marks a space). IDs and splits come from a simple teaching tokenizer, not a real model.
What is a token ID?
What happens to a word that is not in the vocabulary as a whole?
Can the vocabulary change while you chat with a model?