Chapter 7 · Large language models
Compute and FLOPs
Training cost is measured in floating-point operations; big models need enormous numbers of them.
Chapter 7 · Large language models
Training cost is measured in floating-point operations; big models need enormous numbers of them.
thousand operations
A big training run needs numbers like the last one.
A FLOP is one tiny sum or product on decimal numbers. Training needs so many that we use powers of ten.
FLOP means floating-point operation: one addition or multiplication on decimal numbers. Neural networks are made of nothing but these.
Choose a model and a dataset, then a cluster. Chip speed is an illustrative slider, not a real product.
Try halving the chips: the time doubles. Try 10× more tokens: the work becomes 10×. All settings here are for teaching and match no product.
About how many FLOPs does training take?
You double the training tokens. The compute…
What is the difference between FLOPs and FLOP/s?