Chapter 3 · Neural networks2014
Dropout
Dropout randomly switches off neurons in training so the network cannot rely on any single one.
Chapter 3 · Neural networks2014
Dropout randomly switches off neurons in training so the network cannot rely on any single one.
At every training step, a different random set of neurons is switched off. The network cannot rely on any one of them, so it has to spread what it knows around.
A network with many neurons and few examples can simply memorise them. It scores full marks in practice, then fails on new examples. This is overfitting.
Two identical tiny networks learn to separate the orange and cyan dots. One uses dropout. Compare how each does on 300 new points it never saw. Try different examples; the effect is bigger on some.
Loss on new points: 1.07 without, 0.37 with. Dropout helped here: the gap between practice and new examples is smaller.
This is a real network, but a tiny one: 24 hidden neurons and 20 practice points, trained in your browser. Results vary with the examples, as they do in real projects.
When is dropout applied?
What problem does dropout fight?
With a dropout rate of 0.5, about how many neurons are off on each step?