Chapter 19 · Hands-on: build real projects
Turn PDFs into searchable chunks
To search PDFs you first pull out the text, then cut it into overlapping chunks with their page numbers.
Chapter 19 · Hands-on: build real projects
To search PDFs you first pull out the text, then cut it into overlapping chunks with their page numbers.
A PDF page holds text…
A search cannot use a whole PDF; it needs small pieces. You will read each page, cut its text into chunks and keep the file name and page number with every chunk, so an answer can point back to its source.
In Set up a local vector database you searched one-line notes. A PDF is long, so we first cut it up. This step needs no model at all; it is ordinary Python. We keep working in the pdf-chat folder with its virtual environment.
The chunking idea is the same as in the chunking lesson: small enough to be specific, big enough to make sense, with a little overlap so no sentence is lost at a cut.
pypdf is a small, popular, free library for reading PDF files. Move into pdf-chat, activate the virtual environment (see Set up Python and a virtual environment) and install it. It is a small download.
cd pdf-chat
.venv\Scripts\Activate.ps1
pip install pypdf Check: The line starts with (.venv) and the install finishes without a red error.
Make a folder called pdfs inside pdf-chat and copy two or three PDFs into it. Choose PDFs you may use, such as your own notes, a manual or a public handbook. Text PDFs work best; a short one with a few pages keeps testing quick.
mkdir pdfs Check: Your PDFs are inside pdf-chat/pdfs.
Before the full script, check that Python can read your PDF. This tiny script opens the file you name, prints the page count and the first 300 characters of page 1. Save it as read_pdf.py and run it with your own file name.
import sys
from pypdf import PdfReader
reader = PdfReader(sys.argv[1])
print("Pages:", len(reader.pages))
print(reader.pages[0].extract_text()[:300])python read_pdf.py pdfs/yourfile.pdfCheck: You see the number of pages and readable text. If the text is empty or garbled, the PDF may be a scan or picture; see the helper below.
The function split_text cuts text into pieces of about 500 characters. It tries to end each piece at a space, so words stay whole, and starts the next piece 100 characters back, which is the overlap. The loop at the bottom visits every PDF, every page and every piece, and keeps the file name and page number with each chunk. Save it as chunk_pdfs.py.
Use the explorer below to see how CHUNK_SIZE and OVERLAP change the pieces before you run it.
import json
from pathlib import Path
from pypdf import PdfReader
CHUNK_SIZE = 500 # characters per chunk
OVERLAP = 100 # characters shared with the previous chunk
def split_text(text, size=CHUNK_SIZE, overlap=OVERLAP):
text = " ".join(text.split()) # tidy spaces and line breaks
chunks = []
start = 0
while start < len(text):
end = min(start + size, len(text))
if end < len(text):
# try to end at a space so words are not cut in half
space = text.rfind(" ", start + size // 2, end)
if space != -1:
end = space
chunks.append(text[start:end].strip())
if end >= len(text):
break
start = max(end - overlap, start + 1)
# begin the next chunk at a whole word
space = text.find(" ", start, end)
if space != -1:
start = space + 1
return [c for c in chunks if c]
all_chunks = []
for pdf_path in sorted(Path("pdfs").glob("*.pdf")):
reader = PdfReader(pdf_path)
for page_number, page in enumerate(reader.pages, start=1):
text = page.extract_text() or ""
if not text.strip():
print(f"No text on {pdf_path.name} page {page_number} (maybe a scan)")
continue
for piece in split_text(text):
all_chunks.append({
"id": f"{pdf_path.stem}-p{page_number}-c{len(all_chunks) + 1}",
"source": pdf_path.name,
"page": page_number,
"text": piece,
})
Path("chunks.json").write_text(json.dumps(all_chunks, indent=2), encoding="utf-8")
print(f"Saved {len(all_chunks)} chunks from {len(list(Path('pdfs').glob('*.pdf')))} PDF files")
for c in all_chunks[:3]:
print(f"\n[{c['source']}, page {c['page']}] {c['text'][:90]}...") Check: The file is saved in pdf-chat (not inside pdfs).
Run the script. It reads every PDF in pdfs, prints a summary and writes chunks.json. Pages with no readable text are reported and skipped; they are probably scans.
python chunk_pdfs.py Check: You see Saved … chunks from … PDF files and three sample chunks, each showing a file name and page number. A new file chunks.json appears.
Open chunks.json in any text editor. Each entry has an id, the source file, the page and the text. This is exactly what the next lesson will embed and store. Now change CHUNK_SIZE to 300 or 800 and OVERLAP to 0 or 150, save and run again. See how the number of chunks changes.
Check: A smaller size gives more chunks; a bigger one gives fewer. Overlap 0 gives the fewest.
Paste a paragraph from your own PDF and move the sliders. Watch the chunk count and the repeated words at each chunk’s start.
3 chunks. The highlighted words at the start of a chunk are repeated from the end of the one before, so a sentence cut in two is still whole in one of them.
A teaching copy of the Python function in step 4, so your real chunks will look like this. It runs in your browser on the text above only.
What do you see?
Why do we keep the file name and page number with each chunk?
What does overlap do?
pypdf returns no text for a PDF. What is the most likely reason?