Chapter 12 · Tools, agents and harnesses
Evaluating agents
Agents are tested on whole tasks: did they finish, safely, at what cost and how many steps?
Chapter 12 · Tools, agents and harnesses
Agents are tested on whole tasks: did they finish, safely, at what cost and how many steps?
Each box is one task. Tick: finished. Cross: failed. Amber mark: finished, but did something unsafe on the way. Totals sit under each agent.
A chatbot can be judged by whether its answer is right. An agent does a whole job with many steps, so we judge the job.
Move the sliders to say what matters to you. The ranking updates. Then turn on strict success, so that unsafe runs do not count as wins.
The agents are made up and the 8 tasks are the same for all. Try putting all the weight on one thing: the winner changes. That is why you must decide what “good” means before you test.
An agent finishes every task but deletes files it should not. Best summary?
You only care about speed and cost. Which weights fit?
Why run each task more than once?