Chapter 2 · Machine learning basics
Train, validation and test sets
Keep some data hidden from training so you can honestly measure how the model does on new examples.
Chapter 2 · Machine learning basics
Keep some data hidden from training so you can honestly measure how the model does on new examples.
One deck of examples…
Shuffle first, then split. The model learns from the train pile, you tune choices on the validation pile, and the test pile stays sealed until the very end.
A model can simply memorise its training examples and score perfectly. That tells you nothing about new data, like marking a student on the exact questions they practised.
Change the piles or reshuffle. Three tiny models learn from the train pile only. Look at how the memoriser scores on each pile.
| Model | Train | Validation | Test |
|---|---|---|---|
| Memoriser (copies the nearest example) | 0 | 1.33 | 1.08 |
| Smoother (averages 5 neighbours) ✓ chosen | 0.78 | 1.15 | 1.28 |
| Straight line | 1.09 | 1.43 | 0.94 |
The memoriser scores 0 on the pile it studied, but 1.08 on the test pile. Train error alone would have fooled you. Chosen by validation: Smoother, whose honest test score is 1.28.
Reshuffle a few times: with only 40 points, scores wobble. Real projects use far more data. Teaching version: the three models are tiny and simple.
Why test on data the model never trained on?
What is the validation set for?
You tune your model many times against the test set. What happens?