Chapter 20 · Hands-on: more projects
Test your app with an AI judge
A test set plus a model-graded rubric lets you score your app's answers and catch regressions.
Chapter 20 · Hands-on: more projects
A test set plus a model-graded rubric lets you score your app's answers and catch regressions.
Answers are marked one at a time…
A test set, a marking rubric and a judge model. Your script runs each test question through your app, asks the judge to mark the answer, and prints a table you can use to compare versions.
As in the lessons on testing your RAG app and on benchmarks, you improve an AI app by measuring it, not by feeling. Reading every answer yourself gets tiring, so a second model can mark them against a rubric. This is called LLM as a judge.
You need Python with a virtual environment, requests installed (as in Call your local model from code) and a local model running. Windows, macOS and Linux desktops work, and the tests run on your computer.
Make a folder, a virtual environment and install requests, as in the earlier setup lessons. Check your model runner is running.
mkdir ai-judge
cd ai-judge
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install requests Check: The prompt starts with (.venv) and your model runner is running with a small model downloaded.
A good test has three parts: a question, the facts the app is allowed to use, and a plain description of what a good answer must contain. Save this as tests.json. It is made-up library information. Replace it later with cases from your own app.
Notice the third test. It is a trap: the facts do not say when the library opens on Sundays, so a good answer says it does not know. Trap tests catch apps that make things up. Use the builder below to write your own.
[
{
"id": "borrow-days",
"question": "How long can I keep a book?",
"facts": "Books can be borrowed for two weeks and renewed once.",
"expected": "Says two weeks, and that it can be renewed once."
},
{
"id": "late-fine",
"question": "What is the fine for a late book?",
"facts": "A fine of two rupees per day applies to late returns.",
"expected": "Says two rupees per day."
},
{
"id": "opening-hours",
"question": "When does the library open on Sundays?",
"facts": "The library is open from 9 am to 5 pm, Monday to Saturday.",
"expected": "Says it does not know or that the facts do not say. It must not invent Sunday hours."
}
] Check: The file is saved as tests.json in ai-judge.
Create evaluate.py and paste this in. It has two versions of your app that differ only in their instructions. For every test it asks the app for an answer, then asks the judge, with strict rules and the expected points, to reply with JSON: pass or fail and a one-line reason. It prints a table and saves everything to results.json.
import json
from pathlib import Path
import requests
URL = "http://localhost:11434/api/chat"
APP_MODEL = "llama3.2" # the model your app uses
JUDGE_MODEL = "llama3.2" # the judge; a different or larger model is better if you have one
# Two versions of your app: only the instructions differ.
VERSIONS = {
"A": "You are a helpful assistant. Answer the question.",
"B": "You are a careful library assistant. Answer in one or two sentences using only the facts "
"you are given. If the facts do not contain the answer, say you do not know.",
}
JUDGE_RULES = """You are a strict marker. Compare the ANSWER with what a good answer must contain.
Rules:
- PASS only if the answer contains what the good answer must contain.
- FAIL if the answer states anything that the FACTS do not support.
- Judge content only. Ignore length and style.
Reply with JSON only, in this form: {"pass": true, "reason": "one short sentence"}"""
def chat(model, system, user, want_json=False):
body = {
"model": model,
"messages": [{"role": "system", "content": system}, {"role": "user", "content": user}],
"stream": False,
"options": {"temperature": 0},
}
if want_json:
body["format"] = "json"
reply = requests.post(URL, json=body, timeout=300)
reply.raise_for_status()
return reply.json()["message"]["content"]
def run_app(system, case):
return chat(APP_MODEL, system, f"Facts: {case['facts']}\n\nQuestion: {case['question']}")
def judge(case, answer):
prompt = (f"FACTS: {case['facts']}\nQUESTION: {case['question']}\n"
f"A GOOD ANSWER MUST CONTAIN: {case['expected']}\nANSWER: {answer}")
raw = chat(JUDGE_MODEL, JUDGE_RULES, prompt, want_json=True)
try:
verdict = json.loads(raw)
return bool(verdict["pass"]), str(verdict.get("reason", ""))
except (json.JSONDecodeError, KeyError, TypeError):
return None, f"judge gave an unreadable reply: {raw[:80]}"
def main():
cases = json.loads(Path("tests.json").read_text(encoding="utf-8"))
results = {name: {} for name in VERSIONS}
for name, system in VERSIONS.items():
for case in cases:
answer = run_app(system, case)
passed, reason = judge(case, answer)
results[name][case["id"]] = {"answer": answer, "pass": passed, "reason": reason}
print(f"{'test':<16}" + "".join(f"{name:^10}" for name in VERSIONS))
for case in cases:
marks = []
for name in VERSIONS:
passed = results[name][case["id"]]["pass"]
marks.append("?" if passed is None else "PASS" if passed else "FAIL")
print(f"{case['id']:<16}" + "".join(f"{m:^10}" for m in marks))
print()
for name in VERSIONS:
scored = [r["pass"] for r in results[name].values() if r["pass"] is not None]
print(f"Version {name}: {sum(scored)} of {len(cases)} passed")
Path("results.json").write_text(json.dumps(results, indent=2), encoding="utf-8")
print("\nSaved results.json. Read the answers and reasons yourself: the judge can be wrong.")
if __name__ == "__main__":
main() Check: The file is saved next to tests.json.
Run the script. Every test is run twice (once per version), plus a judge call each, so a small model on a CPU may take a few minutes.
python evaluate.py Check: You see a table with the test names and PASS or FAIL for versions A and B, then Version A: … of 3 passed and the same for B.
Open results.json. For every test where A and B differ, read the answer and the judge’s reason. Do you agree with the mark? Spot-check some that agree too. If the judge is often wrong, fix the rubric text in JUDGE_RULES or use a stronger judge model, then run again.
Check: You found at least one mark you agree with and thought about any you do not.
Every time your app gets something wrong in real use, add that case to tests.json. Before you change a prompt or model, run the tests; after, run them again. If a test that used to pass now fails, you have found a regression before your users did. Keep the test file in a folder you back up.
Check: You added one new test and ran the script again.
Write test cases and get a tests.json file to copy. Then see how much to trust a gap between two versions. Both run in your browser only and call no model.
[
{
"id": "how-long-can-i",
"question": "How long can I keep a book?",
"facts": "Books can be borrowed for two weeks and renewed once.",
"expected": "Says two weeks, and that it can be renewed once."
},
{
"id": "when-does-the-library",
"question": "When does the library open on Sundays?",
"facts": "The library is open from 9 am to 5 pm, Monday to Saturday.",
"expected": "Says it does not know. It must not invent Sunday hours."
}
]Version B passed 2 more, but with 10 tests that gap could be luck or judge noise. Read the changed cases yourself and add more tests before deciding.
What do you see?
Why include a question the facts cannot answer?
Version B passed 3 more tests than A, out of 10 tests. What is the sensible reading?
Which statement about the judge is right?