Chapter 13 · Reasoning models2024
Test-time compute
Letting a model spend more computation while answering, not just while training, can raise accuracy.
Chapter 13 · Reasoning models2024
Letting a model spend more computation while answering, not just while training, can raise accuracy.
Accuracy: 0%
The dot shows accuracy for each thinking budget. Easy questions are solved with a little thinking. Hard ones need more.
You know training: a huge one-off cost that builds the model. Test-time compute is the second place. It is the work done while the model answers your question.
Slide the budget. Each puzzle shows the working that fits in the budget. If the working is cut short, the model must blurt a quick guess.
1. What is 47 + 38?
2. A pen costs ₹12. You buy 7 and pay ₹100. What change do you get?
3. 28 pupils need 3 pencils each. Pencils come in boxes of 12. How many boxes?
4. 3 books at ₹45 and 2 notebooks at ₹18. You pay ₹250. Change?
5. A 120 L tank fills at 8 L/min for 6 min, then 5 L/min for 10 min, then 12 L leak out. How much is left?
6. Which is bigger, 2¹⁰ or 10³, and by how much?
A teaching model: a puzzle is solved only when the budget covers all its steps, and otherwise the answer is a guess. Real models can sometimes guess right, and extra thinking can also go wrong.
What is test-time compute?
A puzzle needs 5 steps. You allow 3. What happens in the lesson?
Why not always give the maximum budget?