Chapter 3 · Neural networks
Optimisers: SGD, momentum, Adam
Optimisers decide how to step: momentum keeps rolling, Adam adapts each step size.
Chapter 3 · Neural networks
Optimisers decide how to step: momentum keeps rolling, Adam adapts each step size.
All three start together and take steps of the same size setting. Violet and green use tricks to cross the valley floor quickly; coral just follows the slope.
Plain gradient descent (SGD in the lesson on batches) steps straight down the slope. In a narrow valley the slope points mostly at the wall, so it zig-zags and crawls along the floor.
Pick a landscape, set the step size, then press Race or drag the step slider. The dashed circle is the goal.
Steep walls, a gentle floor. Plain SGD zig-zags.
Try a big step size: SGD may bounce wildly or fly off, while Adam stays steadier. Try the plateau with a small step size and see who still moves.
What does momentum add to gradient descent?
Which optimiser gives each parameter its own step size?
On a perfectly round, easy bowl, which wins by a lot?