Chapter 7 · Large language models
Pretraining
Pretraining reads trillions of tokens of text to learn grammar, facts and patterns.
Chapter 7 · Large language models
Pretraining reads trillions of tokens of text to learn grammar, facts and patterns.
The text is not stored. The model reads it, guesses the next token, and nudges its weights. What stays is the pattern.
Books, articles, web pages and code, cleaned up and cut into tokens. Pretraining uses trillions of them.
Pick a dataset size and your reading habits. Numbers here are illustrative.
The bar is one 80-year lifetime. Dataset sizes are not tied to any real model. A model reads it all at once on many chips, not one word at a time. About 1.3 tokens per word is a rough rule for English.
What game does a model play during pretraining?
Why does pretraining need no hand-made labels?
Does the finished model hold a copy of all its training text?