Chapter 13 · Reasoning models
Verifiers and checking
A separate check (tests, a calculator, another model) can catch wrong answers before they are returned.
Chapter 13 · Reasoning models
A separate check (tests, a calculator, another model) can catch wrong answers before they are returned.
Four answers line up at the checkpoint.
A verifier sits between the model and you. A wrong answer is sent back to try again. A right one goes through.
You have seen hallucinations and slips in sums. Asking the same model “are you sure?” often gets a confident yes.
Eight answers wait at the checkpoint. Five are wrong and three are right. Switch checks on and see which wrong answers are caught.
17 × 24 = 418
def is_even(n): return n % 2 == 1
The Eiffel Tower is in Berlin.
Date of birth: 31/02/2012
The Great Wall is easily seen from space with the naked eye.
12 × 12 = 144
def is_even(n): return n % 2 == 0
Water boils at 100 °C at sea level.
Try each check alone. Then try the second model by itself: it is the only one that can look at anything, yet it misses a bug, a bad date and a popular myth, and it doubts a true fact. The specialised checks are narrow but exact.
Which check best catches “17 × 24 = 418”?
Why not ask the same model “are you right?”
A second model flags a right answer. What is that called?