Chapter 3 · Neural networks
Vanishing and exploding gradients
In deep stacks, signals shrink to nothing or blow up; this made deep networks hard until fixes arrived.
Chapter 3 · Neural networks
In deep stacks, signals shrink to nothing or blow up; this made deep networks hard until fixes arrived.
Learning sends a correction signal backwards from the output. At every layer it gets multiplied. Many small multiplications make it vanish; many big ones make it explode.
A network learns by sending a correction signal, called a gradient, from the output back to every layer. Each layer uses it to nudge its weights.
Start with Sigmoid and 12 layers: the first layers hear nothing. Then switch to ReLU, or raise the weight size above 1 to make it explode, then turn on clipping.
Vanished: the first layers get almost no signal, so they barely learn.
Its slope is never above 0.25, so each layer shrinks the signal to a quarter or less.
A simplified teaching version: each layer gets one typical factor (activation slope × weight size). Real networks differ from layer to layer, but the multiplying effect is the same.
A signal is multiplied by 0.25 at each of 10 layers. What happens?
Which layers suffer most when gradients vanish?
Why does ReLU help deep networks?