Chapter 18 · Hands-on: set up your AI workbench
Run an open model on your computer
A local model runner downloads an open-weight model once and then chats with you fully offline.
Chapter 18 · Hands-on: set up your AI workbench
A local model runner downloads an open-weight model once and then chats with you fully offline.
Downloading the model file (once)…
A model is a big file of weights, as you saw in the weights lesson. A local runner downloads that file to your disk and loads it into memory. After that, no internet is needed.
One popular open-source option is Ollama. Another is llama.cpp. We use Ollama here because it installs like a normal app. The steps are similar for other runners. Make sure you have the free space and memory you found in the first lesson.
Windows and macOS: download the installer from the official site, ollama.com, and run it like any app. Linux: the site shows an install command for your system. It is a command you copy from the official page; read it first if you are unsure what it does.
When it finishes, close your terminal and open a new one so it can find the new program.
curl -fsSL https://ollama.com/install.sh | shLinux only. Copy the current command from the official site rather than from memory.
Check: in a new terminal, the next step prints a version.
--version asks the program to report its version. It proves the terminal can find it.
ollama --versionCheck: a version number prints. If not, see the troubleshooter below.
pull downloads a model. Start small: a model of about one billion parameters is a good first try, as in the parameters lesson. llama3.2:1b is one example; any small model your runner lists works too. Build your own command in the helper below.
ollama pull llama3.2:1bCheck: a progress bar reaches 100% and the last line says success.
run loads the model and opens a chat in your terminal. Type a question and press Enter. The first answer is slower while the model loads into memory. Type /bye to leave the chat.
ollama run llama3.2:1bCheck: you see a >>> prompt and the model answers your question in plain text.
Switch off Wi-Fi or unplug the network cable, then run the same chat again. Small models are less accurate than large ones, but every word is produced on your own computer.
ollama listCheck: the chat still answers with the internet off, and ollama list shows your model and its size on disk.
Type the name of the model you want. The helper fills in the three commands you need: download, chat and remove.
ollama pull llama3.2:1bollama run llama3.2:1bollama rm llama3.2:1bFrees the disk space when you no longer want this model.
Stuck? Pick what you see:
What do you see?
What does ollama pull do?
Why does the chat still work with Wi-Fi off?
Your computer freezes when the model starts. What is the best fix?