Chapter 8 · Shaping models: tuning and alignment2023
Direct preference optimisation
DPO learns straight from preferred-vs-rejected pairs without a separate reward model.
Chapter 8 · Shaping models: tuning and alignment2023
DPO learns straight from preferred-vs-rejected pairs without a separate reward model.
Both answers start equally likely
DPO looks at a pair. It makes the better answer more likely and the worse one less likely, comparing against the model it started as.
Freeze a copy of the model as it is now. This is the reference. It tells us how likely each answer was before we changed anything.
Press ✓ on one answer you prefer and ✗ on one you reject, then train. Watch the bars move away from the black marker, which shows where the reference model started.
Question: How do I reset my router?
Pick one preferred and one different rejected answer to enable training. Simplified: one question and four fixed answers, with one score per answer instead of a neural network.
What does DPO train on?
What does DPO do to a rejected answer?
Why keep a reference copy?