Chapter 19 · Hands-on: build real projects
Test and improve your RAG app
Write a few questions with known answers and check whether the right chunks come back before blaming the model.
Chapter 19 · Hands-on: build real projects
Write a few questions with known answers and check whether the right chunks come back before blaming the model.
Running tests…
A scorecard for your app. Each test question has a phrase that the right chunk must contain. If search brings that chunk back, the test passes. A fail tells you search needs fixing, not the model.
Your RAG app from Chat with your PDFs: a simple RAG app sometimes gives a poor answer. There are two possible causes: search brought the wrong chunks, or the model used them badly. Test the first one first. It is quick, it needs no model, and it takes seconds to run.
You work in the same pdf-chat folder with ask.py from the last lesson. The scripts here only import its retrieve function.
Open your PDFs and write four or five questions whose answers you know. For each, copy a short phrase that must appear in the right chunk, such as two weeks. Save them in a file called tests.json in pdf-chat. Use the builder below to avoid typing JSON by hand.
[
{"question": "How long can I borrow a book?", "expect": "two weeks"},
{"question": "What is the fine for returning a book late?", "expect": "two rupees"},
{"question": "When is lunch served?", "expect": "12 to 2"},
{"question": "By what date must exam forms be submitted?", "expect": "10th"}
]Check: The file has no missing commas or brackets. A JSON checker, or the builder below, helps you spot mistakes.
This script takes each question, asks your retrieve function for the top chunks, and checks whether the expected phrase is inside any of them. It prints PASS or FAIL. For each fail it shows what was returned instead, which is the most useful clue. Save as test_rag.py.
import json
import sys
from ask import retrieve
k = int(sys.argv[1]) if len(sys.argv) > 1 else 3
with open("tests.json", encoding="utf-8") as f:
tests = json.load(f)
passed = 0
for t in tests:
hits = retrieve(t["question"], k=k)
found = any(t["expect"].lower() in text.lower() for text, meta in hits)
passed += found
mark = "PASS" if found else "FAIL"
print(f"{mark} {t['question']}")
if not found:
for text, meta in hits:
print(f" got {meta['source']} p{meta['page']}: {text[:70]}...")
print(f"\n{passed} of {len(tests)} passed (top {k} chunks per question)") Check: The file sits next to ask.py, which it imports.
Activate your virtual environment and run the test. Write down the score; it is your starting line. Everything you change next is compared against it.
cd pdf-chat
.venv\Scripts\Activate.ps1
python test_rag.py Check: Each question shows PASS or FAIL and the last line reads like 3 of 4 passed (top 3 chunks per question).
For every FAIL, look at the chunks it returned. Is the right text split across two chunks? Is the PDF text garbled? Is your expected phrase spelled differently in the PDF (for example 10th against 10 th)? Sometimes the test itself is wrong, and fixing it is progress too.
Check: For each fail you can say in a sentence why the right chunk did not come back.
The cheapest experiment needs no re-indexing. Give the script a number to try the top 1, 2, 5 or more chunks. More chunks catch more answers but make the prompt longer and noisier for the model.
python test_rag.py 1
python test_rag.py 5Check: The last line shows a score for each number. Write each down.
Open chunk_pdfs.py and change CHUNK_SIZE (try 300, then 800) or OVERLAP (try 0, then 150), one change per run. After each change, rebuild the chunks and the index, then test again. Change only one number at a time, otherwise you will not know what helped.
python chunk_pdfs.py
python build_index.py
python test_rag.pyCheck: You have a small table: setting changed, score before, score after. Keep the setting with the best score, and put your own numbers in the table, not mine.
Keep tests.json. Whenever you change anything, run it again. A change that raises one score but lowers another is a trade-off, not a win. Passing four small tests does not prove the app is good; add questions as you find real failures. This test checks search only. Reading the final answers still needs your own eyes.
Type questions and the phrase each right chunk must contain, then copy the JSON. Below it, pick what you see in your results for a suggested fix. This does not run a model.
[
{
"question": "How long can I borrow a book?",
"expect": "two weeks"
}
]Write questions from your own PDFs, and copy the phrase exactly as it appears in the PDF text. Nothing you type leaves this page.
What do you see?
A RAG answer is wrong. What do you check first?
Why change only one setting per test run?
You changed CHUNK_SIZE but the score is identical. What most likely happened?