Chapter 2 · Machine learning basics
Regularisation
Regularisation adds a small penalty for complexity so the model prefers simpler explanations.
Chapter 2 · Machine learning basics
Regularisation adds a small penalty for complexity so the model prefers simpler explanations.
Same ten dots and same flexible curve. Only the penalty changes, step by step. Watch the curve smooth out and the weights shrink.
To touch every noisy dot, a wiggly curve needs huge weights that pull in opposite directions. Big weights are a sign of overfitting.
A flexible curve (power 9) is fitted to ten violet dots. Coral rings are test dots it never saw. Slide the penalty from none to strong.
Too weakPenalty too small: weights are huge, the curve still chases the noise, and test error is high. Total weight size: 161.
A simplified teaching version: a curve of power 9, ten training dots, and a penalty on the squared weights (called ridge or L2). Training error always rises with the penalty; test error falls first, then rises.
What does a regularisation penalty charge for?
You raise the penalty and training error rises. Is that always bad?
The penalty is far too strong. What do you get?