Chapter 8 · Shaping models: tuning and alignment
Benchmarks and evaluations
Benchmarks test models on fixed question sets; they help compare but can saturate or leak into training.
Preparing benchmarks and evals: design your own
Chapter 8 · Shaping models: tuning and alignment
Benchmarks test models on fixed question sets; they help compare but can saturate or leak into training.
Models improve, so the same questions get easier.
Toy models, invented numbers. As a test gets old, high scores stop meaning much.
A benchmark is a set of questions with known answers. Give the same questions to many models and you can compare scores fairly.
Pick what you need the model for, then tick questions. Watch whether your eval can pick the right toy model.
Your eval would pick the wrong winner or cannot separate them.
Every model scores close to 100%. What does that tell you?
What is leakage?
Why build your own eval?