Chapter 19 · Hands-on: build real projects
Chat with your PDFs: a simple RAG app
RAG finds the chunks closest to your question and gives them to the model with the question, so it answers from your documents.
Chapter 19 · Hands-on: build real projects
RAG finds the chunks closest to your question and gives them to the model with the question, so it answers from your documents.
A question arrives…
Your app will turn the question into numbers, pick the three closest chunks from your PDFs, paste them into a prompt with the question, and ask your local model. The answer can name the pages because every chunk carries its file and page.
This joins everything so far. RAG (retrieval-augmented generation, as in the RAG lesson) means: first retrieve useful text, then generate an answer from it. The model does not know your PDFs; we hand it the right pages each time.
You need chunks.json from Turn PDFs into searchable chunks, the embedding setup from Make embeddings on your computer and Set up a local vector database, and a local model running as in Run an open model on your computer and Call your local model from code. We reuse the pdf-chat folder.
Activate the virtual environment in pdf-chat. Make sure your local model runner is running and a small model is downloaded (if you followed the setup lessons it is). Then install requests, a small library for calling web addresses; it may already be there. Your script will talk to the runner at a web address on your own computer.
cd pdf-chat
.venv\Scripts\Activate.ps1
pip install requests Check: You have chunks.json in pdf-chat, the line starts with (.venv), and your model runner is running.
This is the “file the cards” step from the vector database lesson, now for PDF chunks. It embeds every chunk and stores it with its file and page in a collection called pdfs. It deletes any old pdfs collection first, so run it again whenever chunks.json changes. Save as build_index.py and run it.
import json
import chromadb
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
client = chromadb.PersistentClient(path="db")
# start fresh so old chunks never mix with new ones
try:
client.delete_collection("pdfs")
except Exception:
pass
collection = client.create_collection(name="pdfs", metadata={"hnsw:space": "cosine"})
with open("chunks.json", encoding="utf-8") as f:
chunks = json.load(f)
texts = [c["text"] for c in chunks]
vectors = model.encode(texts, normalize_embeddings=True, batch_size=32)
collection.add(
ids=[c["id"] for c in chunks],
documents=texts,
embeddings=vectors.tolist(),
metadatas=[{"source": c["source"], "page": c["page"]} for c in chunks],
)
print("Indexed", collection.count(), "chunks")python build_index.py Check: It prints Indexed … chunks, and the number matches the chunks saved earlier.
One file, four small functions. retrieve turns the question into numbers and asks the database for the 3 nearest chunks (TOP_K). build_prompt pastes those chunks, each numbered with its file and page, above the question, and tells the model to answer only from them and to say so when it cannot. ask_model sends the prompt to the local runner and returns the reply text; stream: False means “send the whole answer at once”. Finally main prints the answer and the sources. Save as ask.py.
import sys
import chromadb
import requests
from sentence_transformers import SentenceTransformer
MODEL_NAME = "llama3.2" # any small model your runner lists
URL = "http://localhost:11434/api/chat"
TOP_K = 3
embedder = SentenceTransformer("all-MiniLM-L6-v2")
collection = chromadb.PersistentClient(path="db").get_collection("pdfs")
def retrieve(question, k=TOP_K):
q = embedder.encode(question, normalize_embeddings=True).tolist()
result = collection.query(query_embeddings=[q], n_results=k)
return list(zip(result["documents"][0], result["metadatas"][0]))
def build_prompt(question, hits):
sources = "\n\n".join(
f"[{n}] ({meta['source']}, page {meta['page']})\n{text}"
for n, (text, meta) in enumerate(hits, start=1)
)
return (
"Answer the question using only the sources below. "
"Cite the source number like [1] after each fact. "
"If the sources do not contain the answer, say \"I could not find that in the documents.\"\n\n"
f"Sources:\n{sources}\n\nQuestion: {question}"
)
def ask_model(prompt):
reply = requests.post(
URL,
json={"model": MODEL_NAME, "messages": [{"role": "user", "content": prompt}], "stream": False},
timeout=300,
)
reply.raise_for_status()
return reply.json()["message"]["content"]
def main():
question = " ".join(sys.argv[1:])
if not question:
sys.exit('Ask a question, for example: python ask.py "How long can I borrow a book?"')
hits = retrieve(question)
answer = ask_model(build_prompt(question, hits))
print(answer)
print("\nSources:")
for n, (text, meta) in enumerate(hits, start=1):
print(f" [{n}] {meta['source']}, page {meta['page']}")
if __name__ == "__main__":
main() Check: The file is saved next to chunks.json and db.
Ask something your PDFs can answer, with the question in quotes after the file name. The first run loads the embedding model and your local model, so it may take a while on a CPU. Small local models can be slow, so be patient.
python ask.py "How long can I borrow a book?" Check: You see an answer that uses facts from your PDF, with a source number like [1], then a Sources list showing file names and page numbers.
Understanding beats magic. Temporarily add print(build_prompt(question, hits)) in main before the model is called, run a question and read exactly what the model receives. The helper below shows the same layout. You will see the model is simply reading the pages you handed it.
print(build_prompt(question, hits))Check: You can find your question, three numbered chunks and the instructions in the printed prompt.
Try a question your documents cannot answer, such as “Who won the match yesterday?”. A well-behaved RAG app says it could not find the answer. Small models sometimes ignore instructions and make something up, which is why the sources list matters: always check the page.
python ask.py "Who won the cricket match yesterday?"Check: The reply says it could not find that in the documents. If it invents an answer, note it; the next lesson shows how to test and improve this.
Change the question, the file names, pages and chunk text, and watch the prompt change. This writes text only; it does not run a model.
Answer the question using only the sources below. Cite the source number like [1] after each fact. If the sources do not contain the answer, say "I could not find that in the documents."
Sources:
[1] (handbook.pdf, page 1)
Books can be borrowed for two weeks and renewed once.
[2] (handbook.pdf, page 4)
A fine of two rupees per day applies to late returns.
Question: How long can I keep a book?About 100 tokens (a rough guess: 4 characters each). More chunks means a longer prompt, which is slower and must fit in the model’s context window.
What do you see?
In a RAG app, how does the model get to know your PDFs?
Why does each chunk keep its file name and page number?
The answer is wrong. What should you check first?