Chapter 8 · Shaping models: tuning and alignment
Distillation
A small student model learns to imitate a large teacher, keeping much of its skill at lower cost.
Chapter 8 · Shaping models: tuning and alignment
A small student model learns to imitate a large teacher, keeping much of its skill at lower cost.
The teacher answers with probabilities
A big model gives a probability for every option. The small student copies that spread, not just the winner.
First build a large, capable model. It is expensive to run, which is why we want something smaller.
Two students learn from the same few points. The left only sees the teacher’s top pick. The right sees the teacher’s probabilities. Each map shows what the student predicts, and the score shows how often it agrees with the teacher.
| Example 1 | cat | dog | fox |
|---|---|---|---|
| Hard label | 0% | 0% | 100% |
| Teacher’s soft answer | 6% | 30% | 64% |
The soft student is closer to the teacher: the extra probabilities carried more information per example. With plenty of examples, both catch up. Simplified: the students are tiny linear models, and the agreement is measured on a grid of 441 points.
What does the student try to match?
Why can soft labels help more than hard labels?
What is the usual benefit of distillation?