Chapter 4 · The deep learning era (2006–2016)2009
GPUs and the deep learning boom
GPUs do thousands of multiplications at once, which made training big networks practical.
Chapter 4 · The deep learning era (2006–2016)2009
GPUs do thousands of multiplications at once, which made training big networks practical.
Every cell of the grid is its own small sum. None needs the answer of another, so sixteen workers can take one cell each and finish together.
Each neuron multiplies inputs by weights and adds them. Stack thousands of neurons and you get millions of multiply-and-add steps for a single answer. Training repeats that millions of times.
Pick a grid size and a crowd of workers, then press play. Each round, every busy worker finishes one cell. Try 1 worker, then 1000.
Press Run to start the clock.
A grid this size needs 216 small multiplications in all. A simplified version: real chips also wait for data to arrive.
Why do GPUs suit neural networks?
You double the workers but the grid has fewer cells than workers. What happens?
What was a GPU originally built for?