Chapter 3 · Neural networks
The chain rule, intuitively
Backprop is the chain rule: small changes multiply along the path from weight to loss.
Chapter 3 · Neural networks
Backprop is the chain rule: small changes multiply along the path from weight to loss.
Turn the big gear once and the medium one turns twice. It turns the small one twice more, so the small gear spins four times. Ratios along a chain multiply.
In a network, a weight changes a neuron's sum, which changes its output, which changes the next layer, and finally the loss. That is a chain of links.
First set each link's ratio and nudge the start. Then see the same rule on a real neuron's loss.
Total ratio = 2 × 0.5 × 3 = 3. Set any one link to 0 and the whole chain goes quiet.
Chain: weight w → sum z = w × x → output a = sigmoid(z) → loss = ½ (a − target)².
Gear 1 turns gear 2 three times, and gear 2 turns gear 3 twice. How many times does gear 3 turn per turn of gear 1?
What happens to the whole chain if one link has ratio 0?
How does backprop use the chain rule?