Chapter 8 · Shaping models: tuning and alignment
Reward models
A reward model predicts how much people would like an answer, turning taste into a number.
Chapter 8 · Shaping models: tuning and alignment
A reward model predicts how much people would like an answer, turning taste into a number.
The judge scores each answer…
A reward model is a judge for answers. It learned from human choices, so it also learned their quirks.
People cannot rate millions of answers. A reward model learns from their choices and then rates any answer with a score, instantly.
Question: Why is the sky blue? Edit the answer, or tap a preset or a trick. Watch the judge's reward and the true quality, a checklist of three facts.
Score and quality are roughly in step.
Why the score moved: length +0.5, flattery +0.0, real facts +1.2. A simplified rule-based judge; real reward models are neural networks, with subtler blind spots.
What does a reward model output?
A model learns that long, flattering answers score high. What is this?
Which helps reduce reward hacking?