Chapter 8 · Shaping models: tuning and alignment
Quantisation
Storing weights with fewer bits (16, 8, 4) shrinks models and speeds them up with small quality loss.
Chapter 8 · Shaping models: tuning and alignment
Storing weights with fewer bits (16, 8, 4) shrinks models and speeds them up with small quality loss.
Plenty of steps: looks smooth
Fewer bits mean fewer allowed values. The shape survives, but it gets a little blocky.
A trained model is mostly a huge list of weights. Each is stored with a fixed number of bits, often 16 or 32.
Slide the bits down. Dots are the original weights and bars are what gets stored. The faint lines are the allowed values. Then switch on the outlier.
Simplified: one scale for all 24 weights and plain rounding. Try the outlier switch to see why that can hurt.
How many distinct codes can a 4-bit number hold?
Going from 16-bit to 8-bit weights roughly does what to model size?
Why can one huge outlier weight hurt quality?