Chapter 13 · Reasoning models2022
Self-consistency and voting
Sample several reasoning paths and take the majority answer for more reliable results.
Chapter 13 · Reasoning models2022
Sample several reasoning paths and take the majority answer for more reliable results.
What is 17 × 24?
Each path may reach a different answer.
The model reasons several times, in slightly different ways. Some paths slip. The answer most paths agree on is usually the safest.
You saw that a model samples its words, so it may reason differently each time. On a hard question, one run might take a wrong turn.
Choose how many answers to sample and how often a single sample is right. Then draw again, and compare one answer with the vote.
Majority: 408 ✓ right
Who leads after each sample (green right, coral wrong, grey tied):
Try 20% right: the right answer is the rarest, and voting makes things worse. Wrong answers here are spread evenly, which real mistakes often are not.
Out of 9 samples: 408 ×5, 418 ×2, 398 ×2. What does the vote pick?
When can voting make things worse?
Why do wrong answers help the vote work?