Chapter 7 · Large language models2020
Scaling laws
Loss falls predictably as you scale model size, data and compute together.
Preparing scaling laws: why bigger models learn more
Chapter 7 · Large language models2020
Loss falls predictably as you scale model size, data and compute together.
More compute: loss goes down.
On a log chart, loss slides down as the model grows. Bigger datasets give lower curves. The numbers are illustrative.
You know loss: how wrong the model's next-token guesses are. Lower is better.
Move both sliders. The formula and its constants are invented for teaching; real ones are measured.
Loss = 1.7 + 60 ÷ size^0.2 + 60 ÷ data^0.2. All of these constants are made up. The marker uses your exact data setting, so it can sit between the three curves.
What does a scaling law predict?
Each tenfold increase in size helps…
You make the model huge but give it very little data. Loss will be…