Chapter 15 · Running AI in the real world
On-device AI
Small, quantised models can run on phones and laptops, keeping data private and working offline.
Chapter 15 · Running AI in the real world
Small, quantised models can run on phones and laptops, keeping data private and working offline.
A big model at 16 bits: about 140 GB. Far too big for a phone.
Two tricks make a model small enough for a phone: store each number with fewer bits, and use a model with fewer parameters.
Your text stays on your device, which helps privacy. It works with no internet. There is no waiting for a network, and no per-use server bill.
Choose a device and free up memory by closing apps. Then pick a model and precision. Devices and numbers are round, illustrative values, not real products.
At 4-bit, the largest listed model that fits is 3B.
Request: “Summarise my private note.” Selected model: 3B at 4-bit.
Which is a clear benefit of on-device AI?
What decides whether a model can run on a phone?
Which two tricks shrink a model for a device?