Chapter 2 · Machine learning basics
Learning rate
The learning rate is the step size: too small is slow, too big overshoots and never settles.
Chapter 2 · Machine learning basics
The learning rate is the step size: too small is slow, too big overshoots and never settles.
too smalljust righttoo big
Same start, same curve, only the step size differs. A tiny step is safe but slow. A huge step overshoots the bottom and can fly out.
The learning rate scales every step. It is set by people before training, not learned by the model.
Set the rate for your ball (coral). The cyan and green balls use a small and a good rate for comparison. Press Run and watch.
Try 0.02 (slow), 0.3 (good), 0.7 (bouncy at first) and 0.9 (trouble). Real training watches the loss and lowers the rate when it stalls.
What happens with a learning rate that is far too big?
What is the downside of a tiny learning rate?
Who sets the learning rate?