Chapter 8 · Shaping models: tuning and alignment2022
RLHF: learning from human feedback
People rank answers, a reward model learns their taste, and the model is trained to score well.
Chapter 8 · Shaping models: tuning and alignment2022
People rank answers, a reward model learns their taste, and the model is trained to score well.
Two answers to one question
People compare answers. A reward model learns what they like. Then the main model is trained to earn that reward.
Take a model that already follows instructions. It can write several different answers to the same question.
Pick the answer you prefer in each pair. After a few picks, the reward model shows what it thinks you like. Then slide how hard the model is pushed toward that reward.
Question: What is a router?
Simplified: five tagged features instead of a neural network. Notice that with no picks, all four answers are equally likely.
What do people usually do in RLHF?
What does the reward model do?
Why keep the model close to how it wrote before?