Chapter 19 · Hands-on: build real projects
Train a tiny model from scratch
Training means showing a small network examples, measuring its error and nudging its weights, over and over.
Chapter 19 · Hands-on: build real projects
Training means showing a small network examples, measuring its error and nudging its weights, over and over.
Sample at step 0Thq#v9 ;zk]e%L p2(Xo!
Start: it only guesses…
You will train a model that has never seen any text. At first it guesses random characters. Each step it sees some of your text, measures how wrong its guesses were (the loss), and nudges its numbers to be less wrong. The loss falls and the samples start to look like your text.
Everything in the loss and gradient lessons happens here for real: show examples, measure the error, nudge the weights, repeat. We use a character-level model: it reads one character at a time and predicts the next. It is tiny (about a hundred thousand numbers), so it trains on an ordinary laptop CPU in a few minutes. No GPU is needed.
It will not be smart. It learns the style of your text, such as spelling patterns and word shapes, not facts. That is exactly why it is a good way to see learning happen.
Make a new folder, tiny-model, with its own virtual environment (see Set up Python and a virtual environment). Then install PyTorch, a popular open-source library for training models. pypdf is only needed if you build your text from PDFs in step 2.
mkdir tiny-model
cd tiny-model
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install torch pypdf Check: The line starts with (.venv) and python -c "import torch; print('ok')" prints ok.
The model needs plain text in a file called data.txt. More text gives better results: aim for at least 20,000 characters (a few pages), and the script needs 2,000 as a minimum. Good choices: your own essays or notes pasted into one file; a public-domain book saved as plain text (Project Gutenberg is a popular source); or the text of your own PDFs.
To use PDFs, put them in a folder called pdfs inside tiny-model and run this small script, saved as make_data.py.
from pathlib import Path
from pypdf import PdfReader
pieces = []
for pdf in sorted(Path("pdfs").glob("*.pdf")):
for page in PdfReader(pdf).pages:
pieces.append(page.extract_text() or "")
text = "\n".join(pieces)
Path("data.txt").write_text(text, encoding="utf-8")
print(f"Wrote data.txt with {len(text)} characters")python make_data.py Check: A file data.txt exists in tiny-model with at least a few thousand characters.
Open the run planner below. Type the number of characters in your file and use the sliders to see how the settings change the size of the model and how many times it will see each character. If it will see each character more than about 100 times, it will memorise your text rather than learn its style.
Check: You know your file’s character count (the training script prints it too) and the planner shows no red warning.
Save this as train.py. Read the numbered comments:
CrossEntropyLoss, computes how to improve with loss.backward() (the backpropagation lesson) and nudges the weights with the Adam optimiser.WINDOW, BATCH and STEPS are the numbers from the planner.
import math
import torch
import torch.nn as nn
torch.manual_seed(0)
# 1. Read the text and turn each character into a number
text = open("data.txt", encoding="utf-8").read()
if len(text) < 2000:
raise SystemExit("data.txt is too short. Use at least a few thousand characters.")
chars = sorted(set(text))
to_id = {c: i for i, c in enumerate(chars)}
data = torch.tensor([to_id[c] for c in text])
print(f"{len(text)} characters, {len(chars)} different ones")
# 2. A tiny model: embed each character, read the sequence with a GRU, predict the next one
class TinyModel(nn.Module):
def __init__(self, vocab, size=128):
super().__init__()
self.embed = nn.Embedding(vocab, size)
self.gru = nn.GRU(size, size, batch_first=True)
self.out = nn.Linear(size, vocab)
def forward(self, x):
h, _ = self.gru(self.embed(x))
return self.out(h)
model = TinyModel(len(chars))
optimiser = torch.optim.Adam(model.parameters(), lr=0.003)
loss_fn = nn.CrossEntropyLoss()
print("Parameters:", sum(p.numel() for p in model.parameters()))
print(f"Loss from pure guessing would be about {math.log(len(chars)):.2f}")
# 3. Training: show a batch, measure the error (loss), nudge the weights
WINDOW, BATCH, STEPS = 64, 32, 1500
def get_batch():
starts = torch.randint(0, len(data) - WINDOW - 1, (BATCH,))
x = torch.stack([data[s : s + WINDOW] for s in starts])
y = torch.stack([data[s + 1 : s + WINDOW + 1] for s in starts]) # the same text shifted by one
return x, y
def sample(prompt="The ", length=200, temperature=0.8):
model.eval()
ids = [to_id.get(c, 0) for c in prompt]
with torch.no_grad():
for _ in range(length):
logits = model(torch.tensor([ids[-WINDOW:]]))[0, -1] / temperature
ids.append(torch.multinomial(torch.softmax(logits, dim=-1), 1).item())
model.train()
return "".join(chars[i] for i in ids)
print("\nBefore training:\n" + sample() + "\n")
for step in range(1, STEPS + 1):
x, y = get_batch()
loss = loss_fn(model(x).reshape(-1, len(chars)), y.reshape(-1))
optimiser.zero_grad()
loss.backward()
optimiser.step()
if step == 1 or step % 100 == 0:
print(f"step {step:4d} loss {loss.item():.3f}")
print("\nAfter training:\n" + sample())
torch.save({"model": model.state_dict(), "chars": chars}, "tiny-model.pt") Check: The file is saved next to data.txt.
Run the script. It prints a sample from the untrained model, then the loss every 100 steps, then a sample from the trained one. On a typical laptop this takes from under a minute to a few minutes. If it is much slower, press Ctrl+C and lower STEPS.
python train.pyCheck: The first loss is near the “pure guessing” number the script printed, and the later losses are much lower. The “Before” sample is random symbols; the “After” sample has word-like text with your text’s flavour. Numbers vary between runs and computers.
Edit one value at a time and run again. Ideas: STEPS = 300 (shorter), size=256 in TinyModel (bigger), temperature=0.5 or 1.2 in sample (safer or wilder writing), or a different data.txt. Compare the final loss and the samples each time.
The script also saved the trained weights in tiny-model.pt, so you could load the model later.
Check: Lower temperature gives repetitive but tidy text; higher gives more random text. A model that sees too little data repeats lines from it.
Enter your file size, then move the sliders. See the model’s size and how many times each character is seen. This does no training; it only does the sums.
A good amount of text for a first run.
A sensible number of passes over the text.
Each step shows BATCH × WINDOW characters. Your real training time depends on your computer, so time one short run first. These numbers match the script in step 4.
What do you see?
What does the loss tell you?
Your model repeats lines straight from your text. What is the likely cause?
Why does this tutorial use a tiny character-level model?