Chapter 4 · The deep learning era (2006–2016)2015
Residual connections (ResNet)
Skip connections let information bypass layers, making very deep networks trainable.
Chapter 4 · The deep learning era (2006–2016)2015
Skip connections let information bypass layers, making very deep networks trainable.
A signal starts at the output
Learning signals travel backwards, from the output towards the first layer. In a plain stack each layer fades the signal. A shortcut gives it a lane that skips the fading.
More layers let a network build more complex ideas. But before 2015, stacking dozens of layers often made results worse, not better. The early layers received almost no learning signal.
Make the stack deeper and each layer fadier, then switch shortcuts on. Each bar is how much learning signal reaches that layer; the last bar is the first layer, the one furthest from the output.
A teaching version: it shows only how the learning signal flows, on a log scale, using one fade value for every layer. The shortcut lane carries 1 untouched and the layers add on top. A real network is trained on data and varies layer by layer.
What does a residual block output?
How do shortcuts help the first layer of a very deep network?
What problem from an earlier lesson does this help fix?