Chapter 6 · The Transformer
Parameters and model size
A model's size is its number of weights: millions, billions, even trillions.
Preparing parameters: counting a model's size
Chapter 6 · The Transformer
A model's size is its number of weights: millions, billions, even trillions.
Every layer adds weights, so the total climbs fast. The numbers here are for teaching.
You know weights from earlier: numbers the model learned. Parameters is the word for all of them together.
Pick a preset or move the sliders. The formula is simplified, and the presets are invented.
Formula: vocabulary × width, plus layers × 12 × width × width (4 for attention, 8 for feed-forward). Real models differ in the details.
What does the parameter count of a model measure?
You double the width. About how many more weights does each layer have?
Why do bigger models need more memory?