Chapter 18 · Hands-on: set up your AI workbench
Choose a model that fits your computer
Model size and quantisation decide how much memory a model needs; pick the biggest that fits comfortably.
Chapter 18 · Hands-on: set up your AI workbench
Model size and quantisation decide how much memory a model needs; pick the biggest that fits comfortably.
A small model fits easily.
A model has to sit in memory next to your other apps. Fewer bits per weight (quantisation, from the quantisation lesson) shrinks it so a bigger model can fit.
You need the runner from the last lesson. Keep your memory figure from the first lesson in mind.
Names often hold clues: 1b, 8b or 70b means billions of parameters. A tag such as q4 usually means 4-bit quantisation. Tag spellings differ between libraries, so read the model’s page in the runner’s library.
Check: you can say how many billions of parameters a model has just from its name.
The models you downloaded are listed with their size. The file size is close to the memory the weights need.
ollama listCheck: the SIZE column shows each model in gigabytes or megabytes.
Start a chat, ask one question, then leave it with /bye. While it is still loaded, ollama ps shows what is running. The runner unloads idle models after a few minutes.
ollama run llama3.2:1b
# ask a question, then type /bye
ollama psCheck: ollama ps shows the model, its size in memory, and whether it runs on the CPU, the GPU or both.
Open your system’s monitor while the model answers. If memory is nearly full or the computer becomes very slow, the model is too big for comfort.
# Press Ctrl+Shift+Esc to open Task Manager, then the Performance tabCheck: you can see used and free memory while the model answers.
Use the table below with your memory. Choose the biggest model marked comfortable, or one size smaller if you also keep many apps open. Bigger is usually better at hard tasks, but slower.
Check: you have a model size and a bit level written down for the next lessons.
Slide to your memory. Green means comfortable, amber means tight, red means it will not fit. This is a simplified rule of thumb.
| Size | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 1B | 2.4 GBfits | 1.2 GBfits | 0.6 GBfits |
| 3B | 7.2 GBfits | 3.6 GBfits | 1.8 GBfits |
| 8B | 19.2 GBtoo big | 9.6 GBfits | 4.8 GBfits |
| 14B | 33.6 GBtoo big | 16.8 GBtoo big | 8.4 GBfits |
| 32B | 76.8 GBtoo big | 38.4 GBtoo big | 19.2 GBtoo big |
| 70B | 168 GBtoo big | 84 GBtoo big | 42 GBtoo big |
At 4-bit, the biggest comfortable size in the table is 14B.
What does quantising a model do?
You have 8 GB of memory. Which choice is wisest?
What does the 8b in a model name usually mean?