Chapter 3 · Neural networks
Activation functions
Activations (ReLU, sigmoid, tanh) add the bends that let networks learn curves, not just lines.
Chapter 3 · Neural networks
Activations (ReLU, sigmoid, tanh) add the bends that let networks learn curves, not just lines.
None (linear): Passes the number through unchanged. Stacks of these are still just a straight line.
The dashed grey line goes in. Each activation function reshapes it: without one, nothing bends.
A neuron multiplies and adds. Stack ten layers of multiplying and adding and you still have just one big multiply and add: a straight line. That cannot learn a curve.
Three neurons each make a bend, then add together. Pick an activation and move the sliders to make the pink curve hug the dotted bump. Then switch to “None” and see why it is stuck.
Blocks negatives, passes positives. Simple, and the common default. A teaching version: one layer of three neurons over a single input.
What happens if every layer has no activation function?
What does ReLU do to a negative number?
Which activation squashes outputs into 0 to 1?