Chapter 15 · Running AI in the real world
GPUs, TPUs and accelerators
AI runs on chips built for huge numbers of parallel multiplications, with fast memory close by.
Chapter 15 · Running AI in the real world
AI runs on chips built for huge numbers of parallel multiplications, with fast memory close by.
Numbers leave memory…
An AI chip is a grid of many small calculators fed from fast memory. A layer of the model is a huge pile of multiplications, so the whole grid works at once.
Running a model means multiplying huge tables of numbers again and again. Each multiplication is simple, and most do not depend on each other.
Choose a model size and precision. The devices below have round, illustrative memory sizes, not real products.
Why do AI models run well on accelerators?
A model has 10 billion parameters stored at 8 bits (1 byte). About how much memory do the weights need?
Which change makes a model need less memory?