Chapter 3 · Neural networks2015
Batch and layer normalisation
Normalisation keeps activations in a steady range so deep networks train faster and more stably.
Chapter 3 · Neural networks2015
Normalisation keeps activations in a steady range so deep networks train faster and more stably.
Each layer can stretch or squash the numbers a bit. After many layers they are all over the place. A normalisation step after each layer puts them back in a tidy range.
Every layer multiplies and adds. After many layers the numbers can be huge, tiny or lopsided. That makes learning jumpy and slow, as you saw with gradients.
Part 1: raise the layers or weight size and watch the red line run away while the green line stays at 1. Part 2: widen the gap between the two features and watch plain training stall or blow up.
Two inputs must be combined to predict a target. When one input is much bigger than the other, plain training struggles. The red line is plain, the green line is with normalised inputs. Both use the same step size.
A simplified teaching version. Part 1 uses random layers; part 2 normalises the inputs, the same idea that layer and batch norm apply inside the network.
What does normalisation do to a group of numbers?
How does layer norm differ from batch norm?
Why does normalisation speed up training?