15 RAG and Agentic Search
15.5.1 Queries and documents are different (asymmetric retrieval) |
These notes follow the RAG and Agentic Search module of Anthropic Academy’s Building with the Claude API course (anthropic.skilljar.com): Introducing Retrieval Augmented Generation, Text chunking strategies, Text embeddings, The full RAG flow, Implementing the RAG flow, BM25 lexical search, and A Multi-Index RAG pipeline. The code is adapted from the module’s notebooks (001_chunking.ipynb through 005_hybrid.ipynb).
15.1 Learning Objectives
By the end of this lecture you should be able to:
Explain what problem Retrieval Augmented Generation (RAG) solves that “put the whole document in the prompt” cannot, and decide when the extra engineering is worth it.
Implement size-based, sentence-based, and structure-based chunking, predict roughly how many chunks each produces, and choose among them for a given corpus.
Explain what a text embedding is, why individual dimensions are not interpretable, and why queries and documents are embedded differently (asymmetric retrieval).
Trace the full RAG flow—
chunk, embed, store, embed the query, search, prompt— and compute cosine similarity and cosine distance by hand. Implement a minimal in-memory vector index with brute-force nearest-neighbor search, and explain why it must store the chunk text next to each vector.
Explain when brute-force search stops scaling, how approximate nearest-neighbor indexes (IVF, HNSW) trade recall for speed, and why filtering after a top-k search can silently return nothing.
Compute a BM25 score from its parts (term frequency, inverse document frequency, length normalization) and explain when lexical search beats semantic search.
Build a multi-index retriever that fuses semantic and lexical rankings with Reciprocal Rank Fusion (RRF), and work an RRF example by hand.
Recognize retrieval exposed as a tool (agentic search) and contrast it with the fixed pipeline built in this lecture.
15.2 Motivation: An 800-Page Document and a Short Question
15.2.1 The problem
Suppose you are building an assistant that answers questions about an 800-page financial filing: “What risk factors does this company disclose?” The answer is in there, somewhere, but Claude has never seen this document. You have to get the relevant text in front of the model. There are two ways to do it.
Option 1: put everything in the prompt. Extract all the text and paste it in next to the question:
Answer the user's question about the financial document. |
|
<user_question> |
{user_question} |
</user_question> |
|
<financial_document> |
{financial_document} |
</financial_document> |
This is the simplest thing that could work, and for small documents it does work. But it breaks down as documents grow:
Hard limit. The context window is finite. An 800-page filing, or a folder of them, may simply not fit.
Quality. Even within the window, models reason less reliably when the answer is one paragraph buried in hundreds of pages of irrelevant text.
Cost. The API bills per input token, so every question pays for all 800 pages again.
Latency. Long prompts take longer to process, and the user is waiting.
Option 2: retrieve first, then prompt. Ahead of time, break the document into small pieces called chunks. When a question arrives, search the chunks for the few that are relevant, and put only those in the prompt. Ask about risks, and the search finds the “Risk Factors” section; Claude reads a page instead of a book. That is Retrieval Augmented Generation.
You have met this idea before. In the Relational Database Design & MySQL lecture you never scanned a whole table for every query: you built an index once and queried the index. RAG applies the same move to unstructured text, with an LLM consuming the search results. And in the The Elephant-Goldfish Model lecture, the context window was the scarce resource you had to spend carefully. RAG is a way of spending it: it selects what goes in, rather than shipping everything.
15.2.2 Running example: the annual research report
Every example in these notes, and every program in examples/, uses the same document: report.md, a fictional “Annual Interdisciplinary Research Review” (about 18 KB) for a company with divisions in medical research, software engineering, finance, law, product engineering, history, project management, pharmaceuticals, and cybersecurity. Its outline, which we will chunk, embed, and search repeatedly:
# Annual Interdisciplinary Research Review: Cross-Domain Insights |
## Executive Summary |
## Table of Contents |
## Methodology |
## Section 1: Medical Research - Understanding XDR-471 Syndrome |
## Section 2: Software Engineering - Project Phoenix Stability Enhancements |
## Section 3: Financial Analysis - Q3 Performance and Outlook |
## Section 4: Scientific Experimentation - Characterization of Material Composite XT-5 |
## Section 5: Legal Developments - Navigating IP Precedents and Regulatory Shifts |
## Section 6: Product Engineering - Finalizing Model Zircon-5 Specifications |
## Section 7: Historical Research - Re-evaluating the Galveston Accords (1921) |
## Section 8: Project Management - Progress on Project Cerberus Phase 2B |
## Section 9: Pharmaceutical Development - Compound CTX-204b Phase IIa Update |
## Section 10: Cybersecurity Analysis - Incident Response Report: INC-2023-Q4-011 |
## Future Directions |
Two features of this report will keep causing trouble, on purpose:
Words that mean different things in different sections. The course’s simplified version of the medical section calls a new disease “a ‘bug’ we have not seen before,” while the software section talks about “infection vectors in our distributed systems.” Medical words in the software section, software words in the medical section: a trap for any search that matches on surface words.
An exact identifier in two places. The incident ID INC-2023-Q4-011 appears in exactly two sections: Section 2 (Software Engineering) and Section 10 (Cybersecurity). A user who types that ID wants those sections, and only a search that respects exact strings will reliably find both.
We will hit the first trap while chunking (Text Chunking Strategies), watch semantic search fall into the second (Implementing the RAG Flow), and climb out with lexical and hybrid search (BM25 Lexical Search and A Multi-Index RAG Pipeline).
15.3 Introducing Retrieval Augmented Generation
Retrieval Augmented Generation (RAG) answers questions over large or numerous documents in two phases. Preprocessing (offline, once per document set): split the documents into chunks and build a searchable index over them. Query time (once per question): retrieve the chunks most relevant to the question and insert only those chunks, not whole documents, into the LLM’s prompt.
In plain English: instead of handing Claude a haystack and asking it to find the needle, a cheaper, faster search finds the needle first and Claude only ever sees the needle.
Intuition: the librarian and the expert. RAG splits work between two roles. The librarian (the retrieval system) does not deeply understand the question, but it is fast and has seen every page, so it can pull a short stack of candidates. The expert (Claude) understands the question deeply but only reads what it is handed. The expert is never better than the stack it receives, which is why most of this lecture is about the librarian.
PREPROCESSING (offline, once per document set) |
+-----------+ +----------+ +------------------+ +--------------+ |
| documents |--->| chunking |--->| embed each chunk |--->| index/store | |
+-----------+ +----------+ +------------------+ +------+-------+ |
^ |
QUERY TIME (once per question) | search |
+------------+ +-----------------+ | |
| user query |--->| embed the query |---------------------------+ |
+------------+ +-----------------+ | |
v |
+------------------------+ |
| top-k relevant chunks | |
+-----------+------------+ |
v |
+------------------------+ |
| prompt Claude with the | |
| question + those chunks| |
+------------------------+ |
15.3.1 Benefits and challenges
RAG is a trade: you give up simplicity and get scale and efficiency.
Benefit | Challenge |
Claude focuses on only the relevant content | requires a preprocessing (chunking) step |
scales to very large documents | needs a search mechanism that finds relevant chunks |
works across many documents | retrieved chunks may lack context Claude needs |
smaller prompts: lower cost and latency | many chunking strategies, no single best one |
Common misconceptions.
“RAG is a search box bolted onto an LLM.” It is a pipeline of separately tunable stages (chunking, embedding, indexing, retrieval, prompting), and the system is only as good as its weakest stage. A great embedding model cannot rescue bad chunks.
“More retrieved context is always better.” Irrelevant chunks cost tokens, distract the model, and, if they merely look related (the “bug” trap), can cause a confidently wrong answer.
“RAG teaches the model the documents.” Nothing is learned; no weights change. Every request re-supplies the context. That is a feature: edit a document and the next question sees the edit.
Practical implications: when to use RAG. RAG adds real work: a preprocessing pipeline, an index to maintain, and a retrieval step to tune. It pays for itself when documents do not fit in a prompt (or fitting them is too slow or expensive), when you must search across many documents, or when per-question cost and latency matter at scale. If your documents fit comfortably and cost is not a concern, putting the whole document in the prompt is a perfectly good, simpler answer.
Worked example: deciding. Three projects, three decisions:
Project | Decision | Why |
Q&A over our 18 KB report.md | no RAG needed | fits easily; stuffing is simpler |
Q&A over one 800-page filing | RAG | too big, too slow, too costly per question |
search across 5,000 support tickets | RAG | many documents; must search across all of them |
We build RAG over report.md anyway because it is small enough to inspect every chunk by eye. The techniques transfer unchanged to the 800-page case.
15.4 Text Chunking Strategies
A chunk is a contiguous span of a source document, small enough to be embedded and retrieved on its own, and ideally covering one coherent idea. A chunking strategy is the rule that decides where one chunk ends and the next begins.
In plain English: chunking cuts the document into the pieces the search engine will later hand to Claude.
Why it matters so much. Every later stage (embedding, indexing, retrieval, the final answer) works on whatever pieces chunking produced. Information destroyed at a chunk boundary cannot be recovered downstream. Chunking is the highest-leverage decision in the pipeline, and the one most often made carelessly.
The “bug” problem. Recall the running example: the medical section describes a new disease as “a ‘bug’ we have not seen before.” Suppose a careless chunker produces a chunk containing that sentence but cut off from the surrounding medical context. A user asks, “How many bugs did engineers fix this year?” The isolated chunk mentions a bug and looks relevant, so it gets retrieved, and Claude answers a software question with medical research. Good chunking keeps enough context in each chunk that its topic stays unambiguous.
15.4.1 Strategy 1: size-based chunking
Split the text into fixed-length windows of characters (or tokens), with each window optionally overlapping the previous one by a fixed amount.
In plain English: slide a fixed-size window across the text. Each new window starts chunk_overlap characters before the previous one ended, so text cut at a boundary reappears at the start of the next chunk.
def chunk_by_char(text: str, chunk_size: int = 150, chunk_overlap: int = 20) -> list[str]:
"""Split text into fixed-size character windows with overlap."""
if chunk_overlap >= chunk_size:
raise ValueError("chunk_overlap must be smaller than chunk_size")
chunks = []
start_idx = 0
while start_idx < len(text):
end_idx = min(start_idx + chunk_size, len(text))
chunk_text = text[start_idx:end_idx]
chunks.append(chunk_text)
# Step back by the overlap -- unless we just consumed the end of the text.
start_idx = end_idx - chunk_overlap if end_idx < len(text) else len(text)
return chunksImportant parts. The guard chunk_overlap >= chunk_size matters: without it the window would never advance and the loop would never end. The conditional on the last line stops the window from stepping back once it has reached the end of the text, which would otherwise produce a tiny, redundant final chunk.
Worked example: a trace. Take text = "abcdefghijklmnopqrstuvwxyz" (26 characters), chunk_size=10, chunk_overlap=3:
iteration start end chunk next start |
1 0 10 "abcdefghij" 10 - 3 = 7 |
2 7 17 "hijklmnopq" 17 - 3 = 14 |
3 14 24 "opqrstuvwx" 24 - 3 = 21 |
4 21 26 "vwxyz" end == len -> 26, stop |
Each chunk begins with the last three characters of the one before it (hij, opq, vwx). That repetition is the overlap.
On the running example. report.md with chunk_size=150, chunk_overlap=20 gives 141 chunks averaging 149.7 characters (output verified below). Most are cut mid-sentence, and section headings are routinely separated from their content.
Common misconception. “Overlap fixes mid-sentence cuts.” It does not: any single chunk can still start or end mid-word. Overlap only guarantees that the cut-off text also appears in a neighbor, so nothing vanishes completely.
Practical implications. Size-based chunking assumes nothing about structure, so it works on any input: Markdown, plain text, logs, even source code. That robustness is why size-based chunking with overlap is the most common production default. It is never great, but it never fails outright.
15.4.2 Strategy 2: sentence-based chunking
Definition. Split the text into sentences, then group a fixed number of consecutive sentences into each chunk, optionally repeating the last few sentences at the start of the next chunk.
import re
def chunk_by_sentence(text: str, max_sentences_per_chunk: int = 5,
overlap_sentences: int = 1) -> list[str]:
"""Group whole sentences into chunks, with sentence-level overlap."""
sentences = re.split(r"(?<=[.!?])\s+", text.strip())
chunks = []
start_idx = 0
while start_idx < len(sentences):
end_idx = min(start_idx + max_sentences_per_chunk, len(sentences))
current_chunk = sentences[start_idx:end_idx]
chunks.append(" ".join(current_chunk))
start_idx += max_sentences_per_chunk - overlap_sentences
if start_idx < 0:
start_idx = 0
return chunksImportant parts. The regex (?<=[.!?])\s+ is the whole trick. It matches runs of whitespace, but only when the character just before is ., !, or ?. The (?<=...) is a lookbehind: it checks the punctuation without consuming it, so the period stays attached to its sentence. Splitting on \s+ alone would split between every word.
Worked example. With 5 sentences per chunk and 1 sentence of overlap, the window advances by 4 sentences each time:
sentences: s0 s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 |
chunk 1: [s0 s1 s2 s3 s4] |
chunk 2: [s4 s5 s6 s7 s8] <- s4 repeated (overlap) |
chunk 3: [s8 s9 s10] <- s8 repeated |
On report.md this gives 33 chunks averaging about 701 characters: far fewer and larger than size-based chunking, and no sentence is ever broken.
Common misconception. “A regex can find every sentence boundary.” It cannot: “Dr. Smith” and “v2.1 release” contain periods that are not sentence ends. The splitter is a heuristic, which is fine for chunking (a slightly wrong boundary is cheap) but not for grammar checking.
Practical implications. A good middle ground: it needs no document structure (unlike strategy 3) and breaks no sentences (unlike strategy 1). A natural choice for general prose.
15.4.3 Strategy 3: structure-based chunking
Definition. Split the text at the document’s own structural markers. For Markdown, those are the section headers.
import re
def chunk_by_section(document_text: str) -> list[str]:
"""Split a Markdown document on its level-2 ("## ") section headers."""
pattern = r"\n## "
return re.split(pattern, document_text)Worked example. report.md has 14 level-2 headers, so chunk_by_section returns 15 chunks: the title line before the first header, plus one chunk per header. One chunk is exactly “Section 2: Software Engineering,” another is exactly “Section 1: Medical Research.” Chunk boundaries now line up with topic boundaries, which is the strongest defense against the “bug” problem: the medical “bug” never shares a chunk with software text. Notice that re.split consumes the delimiter, so every chunk after the first starts with the header text without its ## prefix (for example, "Section 2: Software Engineering - ..."). That header line stays inside the chunk, which helps retrieval.
Common misconception. “Structure-based is best, so always use it.” It only works when you can guarantee consistent formatting. Scanned PDFs, chat logs, and plain .txt files usually have no ## headers, and on such a file chunk_by_section silently returns the whole document as one giant chunk.
Practical implications. This is the best option when you control the format (internal reports, documentation sites, your own Markdown), and a risky one otherwise. The other weakness: sections vary wildly in size (70 to 2,185 characters in report.md), so some chunks may be too large for a tight prompt budget.
15.4.4 Strategy 4: semantic-based chunking
Definition. Split the text into sentences, use an NLP model (typically embeddings, Text Embeddings) to measure how related each sentence is to the next, and cut a chunk boundary wherever relatedness drops.
This produces the most topically coherent chunks, but it is the most expensive (a model call per sentence) and the most complex to implement and tune. The course presents it conceptually. You will build one in Exercise A1.
15.4.5 Comparing the strategies on the running example
The program examples/001_chunking/chunking.py runs all three code-based strategies on report.md:
cd notes/rag/Claude_rag/examples/001_chunking
python chunking.pyExpected output (verified):
--- 1. Size-based: chunk_by_char --- |
chunk count: 141 |
avg length: 149.7 chars |
min/max: 105 / 150 chars |
|
--- 2. Sentence-based: chunk_by_sentence --- |
chunk count: 33 |
avg length: 701.3 chars |
min/max: 237 / 1023 chars |
|
--- 3. Structure-based: chunk_by_section --- |
chunk count: 15 |
avg length: 1216.6 chars |
min/max: 70 / 2185 chars |
Strategy | Needs structure? | Splits sentences? | Cost | Best for |
size-based | no | yes | very low | universal fallback; code; unstructured text |
sentence-based | no | no | low | general prose without reliable formatting |
structure-based | yes | no | low | reports and docs whose format you control |
semantic-based | no | no | high | when coherence matters more than cost |
There is no universally best strategy. The right choice depends on your documents, what you can guarantee about their format, and your budget. From here on we use chunk_by_section, because report.md is well-structured Markdown and its 15 chunks are small enough to inspect by hand.
15.5 Text Embeddings
With the document chunked, the next problem is search: given a question, which chunks are relevant? The dominant modern answer is semantic search, which ranks chunks by meaning rather than by shared words. Its foundation is the text embedding.
A text embedding is a fixed-length vector of real numbers (typically hundreds to a few thousand dimensions) produced by an embedding model, such that texts with similar meanings map to vectors that are close together.
In plain English: feed text in, get a long list of numbers out. Texts that mean similar things get similar lists. The model has turned “meaning” into “geometry,” so “how close are these two lists?” approximates “how similar are these two texts?”
Intuition: scores for hidden qualities. Think of each number as a score for some quality of the text. It is tempting to imagine that dimension 17 means “how much this text talks about oceans” and dimension 340 means “how happy the text sounds.” Those are only illustrations. The real meaning of each dimension is learned during training and is not human-interpretable: you cannot look at one number and say what it measures. To make the math concrete, The Full RAG Flow will use a fictional 2-dimensional model where we do know what each dimension means.
15.5.1 Queries and documents are different (asymmetric retrieval)
Search is asymmetric. A question (“What risks does this company face?”) looks nothing like the passage that answers it (“Risk Factors: our primary exposure is...”). Many embedding models therefore let you say which role a text plays, query or document, and adjust the representation accordingly. Voyage AI is a company that builds AI models specialized in searching, retrieving, and ranking information. It exposes this as an input_type argument:
def generate_embedding(text, model="voyage-3-large", input_type="query"):
result = client.embed([text], model=model, input_type=input_type)
return result.embeddings[0]15.5.2 Local embeddings with BGE-M3
Anthropic does not offer an embedding model. If you use VoyageAI, it requires an account and an API key and bills per token. We substitute BAAI/bge-m3, an open-weight model you run on your own machine through the FlagEmbedding library. “M3” stands for Multi-Lingual (100+ languages), Multi-Functionality (dense, sparse, and multi-vector retrieval from one model), and Multi-Granularity (inputs up to 8,192 tokens). We use only its dense output: one 1,024-number vector per text, the direct equivalent of a VoyageAI embedding.
from typing import List, Union
QUERY_INSTRUCTION = "Represent this sentence for searching relevant passages: "
_model = None
def _get_model():
"""Lazily load and cache BGE-M3 (loading takes seconds; do it once)."""
global _model
if _model is None:
from FlagEmbedding import BGEM3FlagModel # imported here so tests can mock it
# use_fp16 halves memory with negligible accuracy loss.
_model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)
return _model
def generate_embedding(
text: Union[str, List[str]], input_type: str = "query"
) -> Union[List[float], List[List[float]]]:
"""Embed one string or a list of strings.
Same signature as the VoyageAI version, so nothing downstream
changes when you swap providers.
"""
if input_type not in ("query", "document"):
raise ValueError('input_type must be "query" or "document"')
is_list = isinstance(text, list)
inputs = text if is_list else [text]
# Asymmetric retrieval: only the query side gets the instruction prefix.
if input_type == "query":
inputs = [QUERY_INSTRUCTION + t for t in inputs]
output = _get_model().encode(
inputs, batch_size=12, max_length=8192,
return_dense=True, return_sparse=False, return_colbert_vecs=False,
)
embeddings = output["dense_vecs"].tolist() # numpy -> plain Python lists
return embeddings if is_list else embeddings[0]Important design decisions.
Same signature as the original. The rest of the pipeline calls generate_embedding(text, input_type) and receives plain lists of floats. It never knows or cares which provider is behind it. This is the boundary discipline from the Layered Architecture lecture: depend on an interface, not an implementation.
The query prefix is the local version of input_type. BGE-M3 accepts an optional instruction on the query side. Documents are embedded as they are.
Lazy loading. The heavy import and the roughly 2.3 GB model download happen on first use, not at import time, so the tests can run with a fake embedding function on a laptop with no model installed.
Lists in, lists out. Accepting a list lets callers embed many chunks in one batched call (Implementing the RAG Flow relies on this).
VoyageAI | BGE-M3 (these notes) | |
cost | pay per token | free after one-time download |
data location | sent to VoyageAI servers | stays on your machine |
setup | account + API key | pip install, ~2.3 GB download |
dense vector size | 1024 (voyage-3-large) | 1024 |
Worked example: running it.
cd notes/rag/Claude_rag/examples/002_embeddings
pip install -r requirements.txt # FlagEmbedding, torch, pytest
python embeddings.pyThe program chunks report.md, embeds chunk 0 as a document and a question as a query, and prints the shape of each:
Chunked report.md into 15 sections. |
|
Embedding for chunk 0 (document): |
dimensions: 1024 |
first 8 values: [ ... eight small floats ... ] |
|
Embedding for query 'What did the software engineering department do this year?': |
dimensions: 1024 |
first 8 values: [ ... eight small floats ... ] |
The exact floats depend on your hardware and library versions, and they mean nothing individually, which is the point. What matters is that both vectors have 1,024 dimensions and can be compared.
Common misconceptions.
“Embeddings are exact.” They are a statistical approximation of meaning. Two texts can land close together for reasons a human would not call “similar,” and an arbitrary ID like INC-2023-Q4-011 has little meaning for the model to encode at all (BM25 Lexical Search).
“Bigger embedding models are always better.” They cost more memory and time per chunk. For many corpora a smaller model retrieves just as well.
“Embeddings need an API.” Only if you choose a hosted provider. Open-weight models like BGE-M3 run entirely offline.
Practical implications. Generating an embedding is quick and mechanical. The real engineering is what comes next: storing many embeddings and quickly finding the ones closest to a query.
15.6 The Full RAG Flow
This section walks the whole pipeline with numbers small enough to compute by hand. Implementing the RAG Flow then implements it for real.
15.6.1 A “perfect” two-dimensional embedding model
Imagine an embedding model that returns exactly two numbers, and, unlike any real model, we know what they mean:
dimension 1: how much the text is about the medical field;
dimension 2: how much the text is about software engineering.
Step 1: chunk. Two chunks from the running example:
Medical: “This year saw significant strides in our understanding of XDR-471, a ‘bug’ we have not seen before.”
Software: “This division dedicated significant effort to studying various infection vectors in our distributed systems.”
Step 2: embed each chunk.
medical chunk -> [0.97, 0.34] mostly medical; "bug" nudges the software score up |
software chunk -> [0.30, 0.97] mostly software; "infection vectors" nudges medical up |
Embedding APIs usually normalize each vector to length 1, which makes similarity comparisons independent of text length:
|medical| = sqrt(0.97^2 + 0.34^2) = 1.0279 -> [0.944, 0.331] |
|software| = sqrt(0.30^2 + 0.97^2) = 1.0153 -> [0.295, 0.955] |
Step 3: store both vectors, together with their chunk text, in a vector database: a store built for saving vectors and finding the ones nearest to a given vector.
Here preprocessing ends. Everything so far happens before any user shows up. Now we wait for a question.
Step 4: embed the user’s question with the same model: “I’m curious about the company. In particular, what did the software engineering dept do this year?”
query -> [0.10, 0.89] |query| = 0.8956 -> [0.112, 0.994] |
All three normalized vectors lie on the unit circle:
software axis |
1.0 + * query [0.112, 0.994] |
| * software [0.295, 0.955] |
| |
| |
| * medical [0.944, 0.331] |
| |
0.0 +------------------------------------------> medical axis |
0.0 1.0 |
The query and the software chunk point in nearly the same direction. The medical chunk points somewhere else. “Nearest” needs a precise definition, and that definition is cosine similarity.
15.6.2 Step 5: cosine similarity and cosine distance
For vectors a and b:
cosine_similarity(a, b) = (a . b) / (|a| * |b|) range [-1, 1] |
cosine_distance(a, b) = 1 - cosine_similarity(a, b) range [0, 2] |
where a . b is the dot product (multiply matching entries, then add) and |a| is the vector’s length.
In plain English: cosine similarity measures the angle between two vectors and ignores their lengths. 1 means same direction (same meaning), 0 means perpendicular (unrelated), and -1 means opposite directions. Cosine distance flips this to a “smaller means closer” scale, which is how most vector databases report results.
Worked example. For unit-length vectors the denominator is 1, so similarity is just the dot product:
query . software = 0.112*0.295 + 0.994*0.955 = 0.033 + 0.949 = 0.982 |
query . medical = 0.112*0.944 + 0.994*0.331 = 0.106 + 0.329 = 0.434 |
|
similarity distance (1 - sim) |
software chunk 0.982 0.018 <- nearest: retrieved |
medical chunk 0.434 0.566 |
The vector database returns the software chunk, which is what a human reader would choose.1The course video quotes 0.983 and 0.398. The first matches our computation up to rounding. The second is the raw dot product of the unnormalized vectors, 0.10*0.97 + 0.89*0.34 = 0.3996. The true cosine similarity is 0.434. The ranking is the same either way.
15.6.3 Step 6: build the prompt
Only after retrieval do we write the prompt that goes to Claude:
Answer the user's question about the financial document. |
|
<user_question> |
How many bugs did engineers fix this year? |
</user_question> |
|
<report> |
## Section 2: Software Engineering |
This division dedicated significant effort to studying various |
infection vectors in our distributed systems |
</report> |
Claude never sees the medical section, the methodology, or anything else. It sees only what retrieval found. That is the whole point of RAG, and also its risk: Claude’s answer can be no better than the chunks retrieval chose.
15.6.4 The six steps at a glance
Step | Phase | What happens | In the example |
1 chunk | preprocessing | split documents | medical, software chunks |
2 embed | preprocessing | chunk -> vector (normalized) | [0.944,0.331], [0.295,0.955] |
3 store | preprocessing | save (vector, text) pairs | vector database |
4 embed query | query time | question -> vector, same model | [0.112,0.994] |
5 search | query time | rank stored vectors by similarity | software 0.982 > medical 0.434 |
6 prompt | query time | question + top chunks -> Claude | Section 2 only |
Common misconceptions.
“Similarity 0 means opposite meaning.” Zero means unrelated (perpendicular). Opposite would be near -1, and real text embeddings rarely go there.
“Real models separate topics this cleanly.” Our 0.98 vs. 0.43 gap is an artifact of having only two dimensions. Real 1,024-dimensional embeddings produce a much more crowded continuum. In the next section the top two real results are 0.71 and 0.72 apart in distance.
“The query can use a different embedding model.” It cannot. Query and chunks must come from the same model, or their coordinates mean different things and the distances are noise.
15.7 Implementing the RAG Flow
Now the real thing: real BGE-M3 embeddings, a real (if tiny) vector store, and the real report. The implementation follows the same five steps: chunk, embed each chunk, store, embed the question, search.
15.7.1 The VectorIndex class
VectorIndex is deliberately minimal: a list of vectors, a parallel list of documents, and brute-force search that measures the distance to every stored vector. Production vector databases (pgvector, FAISS, Pinecone, Weaviate) use approximate-nearest-neighbor structures such as HNSW so that search stays fast with millions of vectors (From a list to a vector database). Brute force makes every step visible, and for a few thousand chunks it is plenty fast.
import math
from typing import Any, Dict, List, Optional, Tuple
class VectorIndex:
"""A tiny in-memory vector store with brute-force nearest-neighbor search."""
def __init__(self, distance_metric: str = "cosine", embedding_fn=None):
self.vectors: List[List[float]] = []
self.documents: List[Dict[str, Any]] = [] # parallel to self.vectors
self._vector_dim: Optional[int] = None
if distance_metric not in ("cosine", "euclidean"):
raise ValueError("distance_metric must be 'cosine' or 'euclidean'")
self._distance_metric = distance_metric
self._embedding_fn = embedding_fn
def add_document(self, document: Dict[str, Any]):
"""Embed and store one document (a dict with a 'content' key)."""
vector = self._embedding_fn(document["content"], input_type="document")
self.add_vector(vector=vector, document=document)
def add_documents(self, documents: List[Dict[str, Any]]):
"""Embed and store many documents in ONE batched embedding call."""
if not documents:
return
contents = [doc["content"] for doc in documents]
vectors = self._embedding_fn(contents, input_type="document")
for vector, document in zip(vectors, documents):
self.add_vector(vector=vector, document=document)
def add_vector(self, vector: List[float], document: Dict[str, Any]):
# Every vector must have the same dimension as the first one stored.
if not self.vectors:
self._vector_dim = len(vector)
elif len(vector) != self._vector_dim:
raise ValueError(
f"Inconsistent vector dimension. Expected {self._vector_dim}, got {len(vector)}"
)
self.vectors.append(list(vector))
self.documents.append(document)
def search(self, query: Any, k: int = 1) -> List[Tuple[Dict[str, Any], float]]:
"""Return the k (document, distance) pairs closest to the query.
`query` may be raw text (embedded here, as a *query*) or a vector.
"""
if not self.vectors:
return []
if isinstance(query, str):
query_vector = self._embedding_fn(query, input_type="query")
else:
query_vector = query
if len(query_vector) != self._vector_dim:
raise ValueError("Query vector dimension mismatch.")
if k <= 0:
raise ValueError("k must be a positive integer.")
dist_func = (self._cosine_distance if self._distance_metric == "cosine"
else self._euclidean_distance)
distances = [(dist_func(query_vector, v), self.documents[i])
for i, v in enumerate(self.vectors)]
distances.sort(key=lambda item: item[0]) # smallest distance first
return [(doc, dist) for dist, doc in distances[:k]]
@staticmethod
def _euclidean_distance(vec1: List[float], vec2: List[float]) -> float:
return math.sqrt(sum((p - q) ** 2 for p, q in zip(vec1, vec2)))
@staticmethod
def _cosine_distance(vec1: List[float], vec2: List[float]) -> float:
mag1 = math.sqrt(sum(x * x for x in vec1))
mag2 = math.sqrt(sum(x * x for x in vec2))
if mag1 == 0 and mag2 == 0:
return 0.0
if mag1 == 0 or mag2 == 0:
return 1.0
similarity = sum(p * q for p, q in zip(vec1, vec2)) / (mag1 * mag2)
similarity = max(-1.0, min(1.0, similarity)) # clamp float round-off
return 1.0 - similarity
def __len__(self) -> int:
return len(self.vectors)This is a condensed version of the class in examples/003_vectordb/vectordb.py, which adds input validation (type checks, a missing-content check, a clear error when no embedding function was supplied) and a __repr__.
Important parts.
Why store the text next to the vector? A vector search returns positions and distances, which is to say numbers. Step 6 needs the actual chunk text to build the prompt. Every stored “document” therefore carries its content, and search returns documents, not vectors. Forgetting this is the classic first RAG bug.
Why batch? add_documents embeds all chunks in one call. With VoyageAI, that avoided per-request rate limits. With local BGE-M3 it still matters, for a different reason: the model processes a batch on the CPU or GPU far faster than a loop of single texts.
Query vs. document. add_document embeds with input_type="document" and search with input_type="query". This is the asymmetric retrieval from Text Embeddings, applied automatically.
The clamp. Floating-point round-off can produce a similarity of 1.0000000002. Clamping to [-1, 1] keeps the distance from going slightly negative.
Provider-agnostic. VectorIndex only ever sees lists of floats. Swap BGE-M3 for VoyageAI by passing a different embedding_fn. No other line changes.
15.7.2 The five steps in code
from pathlib import Path
text = Path("report.md").read_text()
# 1. Chunk the text by section
chunks = chunk_by_section(text)
# 2-3. Embed every chunk (one batched call) and store (vector, text) pairs
store = VectorIndex(embedding_fn=generate_embedding)
store.add_documents([{"content": chunk} for chunk in chunks])
# 4-5. Embed the question and find the two closest chunks
question = "What did the software engineering department do last year?"
for doc, distance in store.search(question, k=2):
print(f"cosine distance={distance:.4f} {doc['content'].splitlines()[0][:70]}")cd notes/rag/Claude_rag/examples/003_vectordb
pip install -r requirements.txt
python vectordb.pyExpected output. The course’s run (with VoyageAI) ranks:
Section 2: Software Engineering cosine distance ~ 0.71 <- closest |
Methodology cosine distance ~ 0.72 |
BGE-M3 produces different absolute numbers, but Section 2 should still rank first. Look at how close the two distances are: 0.71 vs. 0.72. That is the crowded continuum of real embeddings, not the clean 0.02 vs. 0.57 of our toy model.
15.7.3 Step 6 in code: prompting Claude with the results
The course’s examples stop at retrieval. Finishing the loop takes one small function to build the prompt and one API call. The capstone ask.py in the mini-project (GitHub-Ready Mini-Project: rag-from-scratch) does exactly this:
PROMPT_TEMPLATE = """\
Answer the user's question about the report, using only the excerpts below.
If the excerpts do not contain the answer, say so instead of guessing.
<user_question>
{question}
</user_question>
<report_excerpts>
{excerpts}
</report_excerpts>"""
def build_prompt(question, results):
"""Wrap each retrieved chunk in its own tag, in retrieval order."""
excerpts = "\n".join(
f"<excerpt>\n{doc['content'].strip()}\n</excerpt>" for doc, _ in results
)
return PROMPT_TEMPLATE.format(question=question, excerpts=excerpts)Two prompt-engineering choices are doing real work here. XML tags separate the question from the excerpts, so the model cannot confuse the two. The refusal instruction (“say so instead of guessing”) gives the model a safe exit when retrieval misses. Without it, an irrelevant set of chunks tends to produce a confident, invented answer. With it, a retrieval miss shows up as a visible “I don’t know.”
15.7.4 From a list to a vector database
VectorIndex compares the query against every stored vector. That costs one similarity computation per vector, O(n·d) for n vectors of dimension d. For our 15 chunks at d = 1,024 that is about 15,000 multiply-adds, which takes microseconds. Now imagine every report the company has ever published, chunked: 20 million chunks. That is about 20 billion multiply-adds per question, which is seconds of CPU for every user’s every search. Below roughly 100,000 vectors, brute force is the right answer. Above that, it stops being one.
Approximate nearest neighbors. Production vector databases give up on exactness. An approximate nearest-neighbor (ANN) index returns almost always the true top-k, in a small fraction of the time. Its quality is measured by recall: the fraction of the true top-k it actually returns. Approximation is acceptable here because the embedding is already an approximation of meaning. If the true seventh-best chunk comes back ninth, no one can tell. Two index designs dominate:
IVF (inverted file index). At build time, run k-means to group the vectors into, say, 1,000 clusters. At query time, compare the query to the 1,000 cluster centers, then brute-force only the vectors in the nearest n_probe clusters (say 10), roughly 1% of the data. The failure mode is a true neighbor sitting just across a cluster boundary, in a cluster that was not probed. Raising n_probe raises recall and latency together; probing every cluster is brute force again.
HNSW (Hierarchical Navigable Small World). Link each vector to its near neighbors in a graph, and stack sparser layers on top, like express and local trains. Search starts on the sparsest layer, hops greedily toward the query, and drops a layer when no hop improves. Query time grows roughly logarithmically with corpus size. The price is memory (the graph lives in RAM) and slower inserts. HNSW is the default in most modern vector databases.
An ANN index makes search faster and strictly less accurate than brute force. If a small corpus returns bad results, the index is not the fix; the chunking or the embeddings are.
Where the vectors live. Start with the simplest option that fits, and keep it behind an interface like VectorIndex so you can move up later:
Option | Examples | Reach for it when |
a list + a loop | VectorIndex | under ~100K vectors; prototypes |
embedded library | FAISS, Chroma, sqlite-vec | one application, local files, no server |
database extension | pgvector (PostgreSQL) | vectors should live next to your existing rows |
dedicated service | Qdrant, Weaviate, Milvus, Pinecone | many clients, billions of vectors |
Filtering: the post-filter trap. Real questions often carry a condition: “What were the cybersecurity risks, in the 2023 report only?” Each stored document then needs metadata (year, section, author) next to its content, and the search must combine similarity with a predicate. There are two orders of operations:
Pre-filter: apply the predicate first, then rank the survivors. Always correct, but a very selective filter can push an ANN index back toward scanning.
Post-filter: take the top k, then drop those that fail the predicate. Fast, and silently wrong: if all 5 nearest chunks come from the 2022 report, the result is empty, even though the 2023 report has good matches further down the ranking.
The symptom is a classic bug report: “search works, but returns nothing when I filter.” Real vector databases implement filter-aware search for exactly this reason. Whatever you use, test the filtered path.
Two operational consequences. First, changing the embedding model means re-embedding every chunk: vectors from two models live in different coordinate systems. Store the model name next to the vectors and plan the switch like a database migration. Second, when a document is edited, its old vectors stay in the index until it is re-chunked and re-embedded, so search keeps finding text that no longer exists.
15.7.5 What still goes wrong
Now ask the vector index about the incident: “What happened with INC-2023-Q4-011?” The ID appears in Section 2 (Software Engineering) and Section 10 (Cybersecurity). In the course’s run, semantic search returns Section 10 first, which is good, but ranks Section 3, Financial Analysis second, a section that never mentions the incident. Section 2, which actually contains the ID, is pushed down.
Why? To an embedding model, INC-2023-Q4-011 is an arbitrary string with little meaning to encode. What it does encode is the topic of the question, roughly “incidents, problems, impact,” which Section 3’s discussion of quarterly risks resembles about as much as Section 2’s engineering work does. Semantic search is strong on meaning and weak on exact terms: identifiers, error codes, product numbers, function names. Users type exact terms all the time, so we need a second kind of search.
15.8 BM25 Lexical Search
Lexical search ranks documents by the words they literally share with the query. BM25 (“Best Match 25”) is the standard lexical ranking function. For a query q and document d:
IDF(t) * f(t,d) * (k1 + 1) |
score(q,d) = SUM ---------------------------------------------- |
t in q f(t,d) + k1 * (1 - b + b * |d| / avgdl) |
|
IDF(t) = ln( (N - n(t) + 0.5) / (n(t) + 0.5) + 1 ) |
where f(t,d) is how many times term t appears in d, |d| is the length of d in tokens, avgdl is the average document length, N is the number of documents, n(t) is how many documents contain t, and k1, b are tuning constants (defaults k1 = 1.5, b = 0.75).
In plain English, as the course’s four steps:
Tokenize the query into terms.
Count how often each term appears across all documents. “the” is everywhere; INC-2023-Q4-011 is in two places.
Weight terms by rarity (IDF). Rare terms get high weight, common terms low. This is the step that lets an exact ID dominate the ranking.
Score and rank documents by how many high-weight terms they contain.
Intuition: the three knobs.
IDF: rarity is information. A term found in every document tells you nothing about which document to pick. A term found in one tells you nearly everything.
k1: diminishing returns. The first occurrence of a term adds a lot of score, and the tenth adds little. As f(t,d) grows, the fraction approaches k1 + 1 and stops growing. Small k1 saturates fast (“contains it or not”), large k1 keeps rewarding repetition.
b: length fairness. A long document contains more of everything just by being long. With b = 0.75, longer-than-average documents have their term frequencies discounted. b = 0 turns this off.
15.8.1 Worked example: BM25 by hand
A three-document corpus in the spirit of the report:
d0: "Engineers fixed the bug behind incident INC-2023-Q4-011." 10 tokens |
d1: "The incident response team contained INC-2023-Q4-011 within hours." 11 tokens |
d2: "Revenue grew this quarter and the team hired engineers." 9 tokens |
N = 3, avgdl = 10.0 |
Step 1: tokenize. The default tokenizer lowercases and splits on non-word characters, so hyphens split the ID into four tokens:
"What happened with INC-2023-Q4-011?" |
-> ["what", "happened", "with", "inc", "2023", "q4", "011"] |
what, happened, and with appear in no document and contribute nothing. The work is done by inc, 2023, q4, 011.
Steps 2 and 3: document frequencies and IDF.
term n(t) IDF = ln((3 - n + 0.5)/(n + 0.5) + 1) |
the 3 ln(0.5/3.5 + 1) = ln(1.143) = 0.134 <- everywhere: nearly worthless |
inc 2 ln(1.5/2.5 + 1) = ln(1.600) = 0.470 |
2023, q4, 011 2 0.470 each |
bug 1 ln(2.5/1.5 + 1) = ln(2.667) = 0.981 <- rare: valuable |
Step 4: score. Each ID token appears once (f = 1) in d0 and d1.
d0 (|d| = 10 = avgdl): denominator = 1 + 1.5*(1 - 0.75 + 0.75*10/10) = 2.500 |
per term = 0.470 * 1 * 2.5 / 2.500 = 0.470 x 4 terms = 1.880 |
|
d1 (|d| = 11 > avgdl): denominator = 1 + 1.5*(1 - 0.75 + 0.75*11/10) = 2.6125 |
per term = 0.470 * 1 * 2.5 / 2.6125 = 0.450 x 4 terms = 1.799 |
|
d2: contains none of the query terms = 0.000 |
Both documents with the ID score high. d2 scores zero. d0 edges out d1 only because it is one token shorter: that is the b parameter at work. These are exactly the numbers BM25Index computes (verified against examples/004_bm25/bm25.py).
15.8.2 The BM25Index class
import math
import re
from collections import Counter
from typing import Any, Callable, Dict, List, Optional, Tuple
class BM25Index:
"""A from-scratch Okapi BM25 index with the same API as VectorIndex."""
def __init__(self, k1: float = 1.5, b: float = 0.75,
tokenizer: Optional[Callable[[str], List[str]]] = None):
self.documents: List[Dict[str, Any]] = []
self._corpus_tokens: List[List[str]] = []
self._doc_len: List[int] = []
self._doc_freqs: Dict[str, int] = {} # n(t): documents containing t
self._avg_doc_len: float = 0.0
self._idf: Dict[str, float] = {}
self._index_built = False # statistics are rebuilt lazily
self.k1, self.b = k1, b
self._tokenizer = tokenizer or self._default_tokenizer
@staticmethod
def _default_tokenizer(text: str) -> List[str]:
return [t for t in re.split(r"\W+", text.lower()) if t]
def add_document(self, document: Dict[str, Any]):
tokens = self._tokenizer(document["content"])
self.documents.append(document)
self._corpus_tokens.append(tokens)
self._doc_len.append(len(tokens))
for token in set(tokens): # count each doc once per term
self._doc_freqs[token] = self._doc_freqs.get(token, 0) + 1
self._index_built = False # IDF and avgdl are now stale
def add_documents(self, documents: List[Dict[str, Any]]):
for document in documents:
self.add_document(document)
def _build_index(self):
n_docs = len(self.documents)
self._avg_doc_len = sum(self._doc_len) / n_docs if n_docs else 0.0
self._idf = {
term: math.log((n_docs - freq + 0.5) / (freq + 0.5) + 1)
for term, freq in self._doc_freqs.items()
}
self._index_built = True
def _compute_bm25_score(self, query_tokens: List[str], doc_index: int) -> float:
counts = Counter(self._corpus_tokens[doc_index])
length_ratio = self._doc_len[doc_index] / self._avg_doc_len
score = 0.0
for token in query_tokens:
if token not in self._idf:
continue # unseen term: contributes nothing
tf = counts.get(token, 0)
numerator = self._idf[token] * tf * (self.k1 + 1)
denominator = tf + self.k1 * (1 - self.b + self.b * length_ratio)
score += numerator / (denominator + 1e-9)
return score
def search(self, query_text: str, k: int = 1,
score_normalization_factor: float = 0.1) -> List[Tuple[Dict[str, Any], float]]:
"""Return the top-k documents as (document, score), LOWER = better."""
if not self.documents:
return []
if not self._index_built:
self._build_index()
query_tokens = self._tokenizer(query_text)
if not query_tokens:
return []
raw = [(self._compute_bm25_score(query_tokens, i), doc)
for i, doc in enumerate(self.documents)]
raw = [(s, doc) for s, doc in raw if s > 1e-9] # drop non-matches
raw.sort(key=lambda item: item[0], reverse=True) # higher raw = better
# Flip to VectorIndex's "smaller is better" convention, keeping the order.
return [(doc, math.exp(-score_normalization_factor * s)) for s, doc in raw[:k]]
def __len__(self) -> int:
return len(self.documents)This condenses examples/004_bm25/bm25.py, which uses only the standard library: no model, no key, no network.
Important parts.
Lazy statistics. IDF and avgdl depend on the whole corpus, so adding a document marks them stale, and the next search rebuilds them. Adding 15 chunks costs one rebuild, not 15.
The +0.5 smoothing in IDF keeps the logarithm finite and non-negative even when a term appears in every document.
Flipping the score convention. Raw BM25 is higher = better. Cosine distance is lower = better. search returns exp(-0.1 * raw), which is order-preserving and flips the direction (raw 1.880 becomes 0.829). Now both indexes speak the same “smaller is closer” language, which A Multi-Index RAG Pipeline depends on. Mixing up the two conventions is the most common bug in student hybrid retrievers.
The tokenizer is pluggable. The default splits INC-2023-Q4-011 into four tokens. It still works, because all four are rare, but a tokenizer that keeps the ID whole would make the match sharper (Exercise I3).
15.8.3 Running it on the report
cd notes/rag/Claude_rag/examples/004_bm25
python bm25.pyExpected output (verified):
1. Chunked report.md into 15 sections. |
2. Indexed 15 chunks. BM25Index(count=15, k1=1.5, b=0.75, index_built=False) |
|
3. Top 3 BM25 results for: 'What happened with INC-2023-Q4-011?' |
|
score=0.2713 section='Section 2: Software Engineering - Project Phoenix Stability Enhancements' |
score=0.3317 section='Section 10: Cybersecurity Analysis - Incident Response Report: INC-2023-Q4-011' |
score=0.9391 section='Methodology' |
Remember: lower is better here. The two sections that literally contain INC-2023-Q4-011 take the top two places, which semantic search failed to do in What still goes wrong. Methodology comes a distant third through common words like what and with.
Common misconceptions.
“BM25 understands the query.” It has no notion of meaning, synonyms, or paraphrase. “How reliable are our distributed systems?” will not match a chunk that only says “stability enhancements.” That is semantic search’s job.
“BM25 is obsolete now that we have embeddings.” It is still the default ranking in search engines such as Elasticsearch, because exact rare-term matching is a real and different requirement. Claude Code’s habit of grepping for a function name is the same idea: exact strings beat meaning when you know the exact string.
Practical implications. Neither search is enough alone:
Query | Semantic search | BM25 |
What happened with INC-2023-Q4-011? | topic only; ranks Section 3 too high | finds both sections with the ID |
How did the developers improve uptime? | finds Section 2 by meaning | misses it: no shared words |
errors in the payment system | good | good if the words appear |
The obvious next step is to run both.
15.9 A Multi-Index RAG Pipeline
Hybrid search sends each query to several indexes (here a semantic VectorIndex and a lexical BM25Index) and merges their rankings into one. A retriever is the component that does the forwarding and the merging.
In plain English: ask two specialists, then combine their answers so that a chunk either one strongly recommends, and especially one that both recommend, ends up near the top.
+--------------+ |
+---->| VectorIndex |--- ranking A (by meaning) ---+ |
query_text ---+ +--------------+ | |
| +--------------+ v |
+---->| BM25Index |--- ranking B (by terms) ---> RRF ---> fused top-k |
+--------------+ |
15.9.1 One interface, many indexes
The design rests on an observation: VectorIndex and BM25Index already have the same shape. Both offer add_document, add_documents, and search(query, k) returning (document, score) pairs with lower meaning better. Their internals have nothing in common (embedding vs. tokenizing, cosine vs. BM25), but callers do not need to know. A Python Protocol names that shared shape:
from typing import Any, Dict, List, Protocol, Tuple
class SearchIndex(Protocol):
def add_document(self, document: Dict[str, Any]) -> None: ...
def add_documents(self, documents: List[Dict[str, Any]]) -> None: ...
def search(self, query: Any, k: int = 1) -> List[Tuple[Dict[str, Any], float]]: ...15.9.2 Merging rankings: Reciprocal Rank Fusion
The problem. BM25 scores and cosine distances live on unrelated scales. A cosine distance of 0.3 and a BM25 score of 0.3 do not mean the same thing, so you cannot average them or concatenate the lists.
The solution. Reciprocal Rank Fusion (RRF) ignores the scores and uses only each index’s rank order:
RRF_score(d) = SUM over indexes i of 1 / (k_rrf + rank_i(d)) |
|
rank_i(d) = position of d in index i's results (1 = best); |
an index that did not return d contributes 0 |
k_rrf = a damping constant (60 is the usual default) |
Worked example (the course’s). Searching for INC-2023-Q4-011, suppose the two indexes return:
VectorIndex: Section 2 (rank 1), Section 7 (rank 2), Section 6 (rank 3) |
BM25Index: Section 6 (rank 1), Section 2 (rank 2), Section 7 (rank 3) |
With k_rrf = 1 so the arithmetic is easy:
Chunk | Vector rank | BM25 rank | RRF score |
Section 2 | 1 | 2 | 1/(1+1) + 1/(1+2) = 0.500 + 0.333 = 0.833 |
Section 6 | 3 | 1 | 1/(1+3) + 1/(1+1) = 0.250 + 0.500 = 0.750 |
Section 7 | 2 | 3 | 1/(1+2) + 1/(1+3) = 0.333 + 0.250 = 0.583 |
Fused ranking: Section 2 (0.833), Section 6 (0.750), Section 7 (0.583). Section 2 wins without being first in BM25, because it ranks well in both lists. A unit test in examples/005_hybrid/tests/test_hybrid.py reproduces these three numbers exactly.
What k_rrf does. The same ranks with the production default k_rrf = 60:
k_rrf = 1 k_rrf = 60 |
Section 2 0.833 0.03252 |
Section 6 0.750 0.03227 |
Section 7 0.583 0.03200 |
Same order, much flatter. A small k_rrf makes a single first-place finish count heavily (1/2 vs. 1/4 for third). A large k_rrf makes every position count almost equally (1/61 vs. 1/63), so agreement between indexes matters more than any one index’s top pick. 60 is a robust default for that reason.
15.9.3 The Retriever class
from typing import Any, Dict, List, Tuple
class Retriever:
"""Fuses any number of SearchIndex rankings with Reciprocal Rank Fusion."""
def __init__(self, *indexes: SearchIndex):
if len(indexes) == 0:
raise ValueError("At least one index must be provided")
self._indexes = list(indexes)
def add_document(self, document: Dict[str, Any]):
for index in self._indexes:
index.add_document(document)
def add_documents(self, documents: List[Dict[str, Any]]):
for index in self._indexes:
index.add_documents(documents)
def search(self, query_text: str, k: int = 1,
k_rrf: int = 60) -> List[Tuple[Dict[str, Any], float]]:
if k <= 0:
raise ValueError("k must be a positive integer.")
# Over-fetch (k * 5) from each index so fusion has candidates to work with.
all_results = [index.search(query_text, k=k * 5) for index in self._indexes]
# doc id -> the document plus its rank in each index (inf = not returned)
doc_ranks: Dict[int, Dict[str, Any]] = {}
for idx, results in enumerate(all_results):
for rank, (doc, _score) in enumerate(results):
entry = doc_ranks.setdefault(
id(doc), {"doc_obj": doc, "ranks": [float("inf")] * len(self._indexes)}
)
entry["ranks"][idx] = rank + 1 # ranks are 1-based
def rrf(ranks: List[float]) -> float:
return sum(1.0 / (k_rrf + r) for r in ranks if r != float("inf"))
fused = [(e["doc_obj"], rrf(e["ranks"])) for e in doc_ranks.values()]
fused.sort(key=lambda item: item[1], reverse=True) # RRF: higher = better
return fused[:k]Important parts.
Identity, not content. Documents are matched across indexes with id(doc): the same Python dict object. This works because add_documents passes the same dicts to every index. Build each index from separately created dicts and the fusion silently treats one chunk as two different documents.
Over-fetching. Each index returns k * 5 candidates. If each returned only k, a chunk ranked k + 1 by both indexes (a strong consensus pick) could never be found. Exercise A2 asks when even 5x is not enough.
Missing means zero. A chunk absent from one index’s list gets rank infinity there and contributes nothing. It is not penalized, just not helped.
Convention flip, again. The fused score is higher = better, unlike the indexes it wraps. Know which convention each layer uses.
15.9.4 The hybrid retriever on the report
chunks = chunk_by_section(Path("report.md").read_text())
vector_index = VectorIndex(embedding_fn=generate_embedding)
bm25_index = BM25Index()
retriever = Retriever(bm25_index, vector_index)
retriever.add_documents([{"content": c} for c in chunks]) # same dicts -> both
for doc, score in retriever.search("What happened with INC-2023-Q4-011?", k=3):
print(f"rrf score={score:.4f} {doc['content'].splitlines()[0][:70]}")cd notes/rag/Claude_rag/examples/005_hybrid
pip install -r requirements.txt
python hybrid.py # prints vector-only results, then hybrid resultsExpected result. Per the course, the hybrid ranking becomes:
1. Section 10: Cybersecurity Analysis - Incident Response Report: INC-2023-Q4-011 |
2. Section 2: Software Engineering - Project Phoenix Stability Enhancements |
Both sections that contain the ID now hold the top two places, and Section 3, the semantic false positive from What still goes wrong, has been pushed out. BGE-M3’s exact scores differ from VoyageAI’s, so compare the rankings that hybrid.py prints, not the numbers.
15.9.5 Extensibility
Because every index satisfies one small SearchIndex protocol, adding a third retrieval method (a metadata filter, a knowledge-graph index, an index over a different chunking of the same documents) means implementing three methods and passing the object to Retriever(...). The fusion code does not change. This is the open/closed principle from the Layered Architecture lecture: open to new indexes, closed to modification.
Common misconceptions.
“RRF averages the scores.” It never reads the scores. Only positions matter. That is the point: it is how you combine scores that cannot be compared.
“Hybrid always beats either index alone.” Usually, not always. If one index is badly tuned, fusing it in adds noise. Measure on your own questions.
“More indexes is always better.” Each index adds ingestion cost and query latency. Add one when it catches a failure the others miss, and measure.
Practical implications. Hybrid search is the production default for RAG because real users mix conceptual questions (“how did engineering improve stability?”) with exact lookups (“INC-2023-Q4-011”) and expect both to work. It costs two index builds and two searches per query, both cheap compared with one LLM call.
15.10 From Pipeline to Agentic Search
Everything above is a fixed pipeline: every question goes through the same steps (retrieve top-k, build the prompt, call Claude once). That is fast and predictable, but it cannot decide not to search (“hello” still triggers a retrieval), rephrase a bad query, or search again when the first results are weak.
Agentic search hands the retriever to the model as a tool and lets the model decide whether to search, what query to send, and whether the results are good enough. The model can turn “that thing about the outage last quarter” into a search for INC-2023-Q4-011, read the hits, notice that the incident is discussed in two divisions, and search again for the engineering side.
fixed pipeline: question -> retrieve -> prompt -> Claude -> answer (1 call) |
|
agentic search: question -> Claude --search("INC-2023-Q4-011")--> Retriever |
^ | |
+-------------- results ---------------+ |
... repeat until satisfied ... -> answer (N calls) |
You have used an agentic search system all semester: Claude Code. Its grep is lexical search, reading a README is coarse retrieval, and its loop of search, read, refine, and search again is agentic retrieval over your repository. The Retriever built here is exactly the kind of component you would expose through tool use or an MCP server.
Rule of thumb. Small corpus: put it in the prompt. Large corpus, predictable single-step questions: fixed pipeline. Messy or multi-step questions over many sources: agentic search, trading latency and tokens for judgment.
15.11 GitHub-Ready Mini-Project: rag-from-scratch
The five examples in notes/rag/Claude_rag/examples/ form one repository, one runnable folder per lesson, plus a capstone 006_ask/ that closes the loop with Claude. Chunking and BM25 need only the standard library. Embedding-based folders need BGE-M3, but their tests use fake embeddings, so every test suite runs offline in under a second.
15.11.1 Structure
rag-from-scratch/ |
|- README.md |
|- report.md # shared running example |
|- 001_chunking/ |
| |- chunking.py # chunk_by_char / _sentence / _section |
| +- tests/test_chunking.py |
|- 002_embeddings/ |
| |- embeddings.py # generate_embedding (local BGE-M3) |
| |- requirements.txt |
| +- tests/test_embeddings.py # mocked model + optional slow test |
|- 003_vectordb/ |
| |- vectordb.py # VectorIndex + five-step flow |
| |- requirements.txt |
| +- tests/test_vectordb.py |
|- 004_bm25/ |
| |- bm25.py # BM25Index, stdlib only |
| +- tests/test_bm25.py |
|- 005_hybrid/ |
| |- hybrid.py # SearchIndex, Retriever, RRF |
| |- requirements.txt |
| +- tests/test_hybrid.py # includes the RRF worked example |
+- 006_ask/ |
|- ask.py # retrieve -> prompt -> Claude |
|- hybrid.py # copy of 005_hybrid/hybrid.py |
|- requirements.txt |
+- tests/test_ask.py |
Each folder is self-contained (its own copy of report.md and of any shared code) so students can run any lesson without the others. That duplication is a deliberate teaching trade-off. In your own project, factor the shared code into a package (Exercise I5).
15.11.2 README.md
# rag-from-scratch
Retrieval Augmented Generation built from first principles, one lesson
per folder: chunking, embeddings, a vector index, BM25, hybrid search
with Reciprocal Rank Fusion, and a final step that prompts Claude.
Adapted from Anthropic Academy's *Building with the Claude API*, module
"RAG and Agentic Search". Embeddings use the open-weight BAAI/bge-m3
model locally instead of VoyageAI: no API key, no per-token cost.
## Quick start
python -m venv .venv && source .venv/bin/activate
cd 001_chunking && python chunking.py && cd .. # stdlib only
cd 004_bm25 && python bm25.py && cd .. # stdlib only
pip install -r 005_hybrid/requirements.txt # ~2.3 GB model on first run
cd 005_hybrid && python hybrid.py && cd ..
## Tests (offline, no model download)
pip install pytest
for d in 00*/; do (cd "$d" && python -m pytest -q tests); done
## Asking Claude (optional)
pip install anthropic
export ANTHROPIC_API_KEY=...
cd 006_ask && python ask.py --claude "What happened with INC-2023-Q4-011?"15.11.3 006_ask/ask.py
"""
Capstone: the complete RAG loop -- retrieve, augment, generate.
Retrieval reuses 005_hybrid's indexes and Retriever unchanged. This file
adds the last two steps of the RAG flow: building the prompt from the
retrieved chunks, and (optionally) sending it to Claude.
python ask.py --bm25-only "What happened with INC-2023-Q4-011?" # offline
python ask.py "What did the software engineering dept do?" # hybrid
python ask.py --claude "What happened with INC-2023-Q4-011?" # + Claude
"""
from __future__ import annotations
import sys
from pathlib import Path
from typing import Any, Dict, List, Tuple
from hybrid import BM25Index, Retriever, VectorIndex, chunk_by_section, generate_embedding
PROMPT_TEMPLATE = """\
Answer the user's question about the report, using only the excerpts below.
If the excerpts do not contain the answer, say so instead of guessing.
<user_question>
{question}
</user_question>
<report_excerpts>
{excerpts}
</report_excerpts>"""
def build_retriever(chunks: List[str], bm25_only: bool = False) -> Retriever:
"""Index every chunk in BM25 (and, unless bm25_only, in a VectorIndex)."""
indexes = [BM25Index()]
if not bm25_only:
indexes.append(VectorIndex(embedding_fn=generate_embedding))
retriever = Retriever(*indexes)
retriever.add_documents([{"content": chunk} for chunk in chunks])
return retriever
def build_prompt(question: str, results: List[Tuple[Dict[str, Any], float]]) -> str:
"""Step 6 of the RAG flow: wrap each retrieved chunk in its own tag."""
excerpts = "\n".join(
f"<excerpt>\n{doc['content'].strip()}\n</excerpt>" for doc, _ in results
)
return PROMPT_TEMPLATE.format(question=question, excerpts=excerpts)
def ask_claude(prompt: str, model: str = "claude-opus-5-5") -> str:
"""Send the augmented prompt to Claude (needs ANTHROPIC_API_KEY)."""
from anthropic import Anthropic # deferred: the offline path never needs it
client = Anthropic()
response = client.messages.create(
model=model,
max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
return "".join(block.text for block in response.content if block.type == "text")
def main(argv: List[str]) -> None:
flags = {a for a in argv if a.startswith("--")}
question = " ".join(a for a in argv if not a.startswith("--"))
if not question:
raise SystemExit('usage: python ask.py [--bm25-only] [--claude] "question"')
text = (Path(__file__).parent / "report.md").read_text()
retriever = build_retriever(chunk_by_section(text), bm25_only="--bm25-only" in flags)
results = retriever.search(question, k=2)
print("Retrieved chunks:")
for doc, score in results:
print(f" rrf={score:.4f} {doc['content'].strip().splitlines()[0][:70]}")
prompt = build_prompt(question, results)
if "--claude" in flags:
print("\nClaude's answer:\n" + ask_claude(prompt))
else:
print(f"\nPrompt that would be sent to Claude ({len(prompt)} chars):\n")
print(prompt[:600] + ("..." if len(prompt) > 600 else ""))
if __name__ == "__main__":
main(sys.argv[1:])Design decisions. –bm25-only gives a fully offline path, useful in class and in CI. The Anthropic import is deferred into ask_claude so the package is needed only with –claude. Without –claude, the program prints the prompt it would send, which is the best way to debug retrieval: read exactly what Claude would read. requirements.txt is 005_hybrid’s plus anthropic.
15.11.4 006_ask/tests/test_ask.py
"""Offline tests for ask.py: BM25-only retrieval plus prompt assembly.
No model download, no API key."""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from ask import build_prompt, build_retriever # noqa: E402
from hybrid import chunk_by_section # noqa: E402
REPORT = (Path(__file__).resolve().parent.parent / "report.md").read_text()
def test_bm25_only_retrieves_both_incident_sections():
retriever = build_retriever(chunk_by_section(REPORT), bm25_only=True)
results = retriever.search("What happened with INC-2023-Q4-011?", k=2)
assert all("INC-2023-Q4-011" in doc["content"] for doc, _ in results)
def test_prompt_contains_question_excerpts_and_refusal_rule():
results = [({"content": "Section 2: Software Engineering\nPhoenix..."}, 0.5)]
prompt = build_prompt("What did engineering do?", results)
assert "<user_question>\nWhat did engineering do?\n</user_question>" in prompt
assert "<excerpt>\nSection 2: Software Engineering" in prompt
assert "say so instead of guessing" in prompt
def test_prompt_keeps_retrieval_order():
results = [({"content": "FIRST"}, 0.9), ({"content": "SECOND"}, 0.8)]
prompt = build_prompt("q", results)
assert prompt.index("FIRST") < prompt.index("SECOND")15.11.5 Run it
cd 006_ask
python -m pytest -q tests # 3 passed
python ask.py --bm25-only "What happened with INC-2023-Q4-011?"Expected output (verified, abbreviated):
Retrieved chunks: |
rrf=0.0164 Section 2: Software Engineering - Project Phoenix Stability Enhancemen |
rrf=0.0161 Section 10: Cybersecurity Analysis - Incident Response Report: INC-202 |
|
Prompt that would be sent to Claude (2887 chars): |
|
Answer the user's question about the report, using only the excerpts below. |
If the excerpts do not contain the answer, say so instead of guessing. |
|
<user_question> |
What happened with INC-2023-Q4-011? |
</user_question> |
|
<report_excerpts> |
<excerpt> |
Section 2: Software Engineering - Project Phoenix Stability Enhancements |
... |
With a single index, RRF scores are simply 1/(60 + rank): 1/61 = 0.0164 and 1/62 = 0.0161. Fusion only becomes interesting with two or more indexes.
15.12 Practice Exercises
15.12.1 Basic
B1. Trace chunk_by_char("The quick brown fox jumps", chunk_size=10, chunk_overlap=3) by hand, in the style of the trace in Text Chunking Strategies. List every chunk.
B2. Explain why chunk_by_sentence splits on (?<=[.!?])\s+ and not on [.!?]\s+. What would go wrong with the second pattern?
B3. For a = [0.6, 0.8] and b = [0.8, 0.6], compute cosine similarity and cosine distance by hand. Then do the same for a and c = [-0.6, -0.8], and explain the result in retrieval terms.
B4. Label each item with its RAG step (1 to 6) and its phase (preprocessing or query time): (a) store.add_documents(...); (b) generate_embedding(question, input_type="query"); (c) build_prompt; (d) re.split(r"\n## ", text).
B5. In the BM25 IDF formula, what happens to a term’s weight as it appears in more documents? What happens to the score of a document as k1 approaches 0?
15.12.2 Intermediate
I1. Modify chunk_by_char to chunk by a fixed number of words instead of characters, keeping overlap. Add two tests, one of which checks that no chunk starts mid-word.
I2. Using 003_vectordb, write a query you predict will retrieve Section 9 (Pharmaceutical Development) first without using the words “pharmaceutical” or “CTX-204b.” Run it. If you were wrong, explain why using the content of the chunks.
I3. Give BM25Index a tokenizer that keeps hyphenated identifiers such as INC-2023-Q4-011 as single tokens. Recompute the toy example from BM25 Lexical Search by hand with the new tokenizer, then check the ranking on report.md.
I4. In hybrid.py, run the incident query with k_rrf = 1, 10, and 60. Report the fused rankings in a table and explain any differences using the RRF formula.
I5. The six project folders each copy shared code. Refactor them into one installable package (rag/ with chunking.py, embeddings.py, indexes.py, retriever.py) so that all existing tests pass unchanged except for their imports.
15.12.3 Advanced
A1. Semantic chunking. Implement the fourth strategy: split into sentences, embed each with generate_embedding, and start a new chunk whenever the cosine similarity between consecutive sentences drops below a threshold. Compare its chunks on report.md with chunk_by_section’s, and report how the threshold affects the chunk count.
A2. When over-fetching fails. Retriever.search fetches k * 5 results from each index. Construct a small corpus and query where the true best fused document is missing from the output because of this limit. Derive the smallest over-fetch multiplier that fixes your example, and argue whether any fixed multiplier is sufficient in general.
A3. Agentic search. Expose Retriever.search as a tool that Claude can call (a tool-use loop or an MCP server). Write five questions about report.md that need more than one search, such as “Which divisions were affected by INC-2023-Q4-011, and what did each do about it?” Compare the fixed pipeline with the agent on answer quality, number of API calls, and total tokens.
15.13 Summary
15.13.1 Key takeaways
RAG trades simplicity for scale. Retrieve only the relevant chunks, then prompt. It is worth the work when documents do not fit, or when per-question cost and latency matter. Otherwise, put the document in the prompt.
Chunking sets the ceiling. No embedding model or search algorithm can recover context destroyed at a chunk boundary. Size-based with overlap is the robust default; structure-based is best when you control the format.
Embeddings turn meaning into geometry. Similar meaning gives nearby vectors. Individual dimensions mean nothing to a human. Query and documents must use the same model, and often different input_types.
The RAG flow is six steps: chunk, embed, store (with the text!), embed the query, search by cosine distance, and prompt with only the hits.
Brute force is fine below ~100K vectors. Past that, ANN indexes (IVF, HNSW) trade a little recall for a large speedup. They make search faster, never more accurate. Filter before ranking, not after, or filtered searches can come back empty.
Semantic search misses exact terms; BM25 misses meaning. BM25 weights rare terms (IDF), saturates repetition (k1), and corrects for length (b).
Hybrid search fuses rankings, not scores. Reciprocal Rank Fusion combines indexes whose scores cannot be compared, and rewards agreement between them.
A shared interface makes the pipeline extensible. Any object with add_documents and search can join the Retriever.
Retrieval can become a tool. Agentic search lets the model decide when and what to search, which is how Claude Code already works.
15.13.2 Terminology
Term | Meaning |
RAG | retrieve relevant chunks, put them in the prompt, generate an answer |
chunk | a span of a document sized for individual retrieval |
overlap | text repeated between adjacent chunks so nothing is lost at a cut |
embedding | fixed-length vector; similar meaning -> nearby vectors |
asymmetric retrieval | embedding queries and documents differently (input_type) |
vector database | store of (vector, text) pairs with nearest-neighbor search |
ANN index | approximate nearest-neighbor search: tunable recall, sublinear time |
recall (of an index) | fraction of the true top-k an ANN index returns |
IVF / HNSW | cluster-and-probe index / layered neighbor graph |
pre/post-filtering | predicate before ranking (correct) vs. after top-k (can return nothing) |
cosine similarity | (a . b) / (|a||b|), in [-1, 1]; higher = more similar |
cosine distance | 1 - cosine similarity; lower = more similar |
semantic search | ranking by embedding similarity |
lexical search | ranking by shared terms |
BM25 | lexical ranking from term frequency, IDF, and length normalization |
IDF | weight that grows as a term gets rarer across the corpus |
hybrid search | running semantic and lexical search and merging the results |
Reciprocal Rank Fusion | merging rankings by summing 1/(k_rrf + rank) |
retriever | component that queries one or more indexes and fuses results |
agentic search | retrieval exposed as a tool the model decides when to call |
15.13.3 Common mistakes
Using structure-based chunking on documents without the assumed structure, producing one giant chunk per document.
Storing vectors without their chunk text, so search returns numbers with nothing to put in the prompt.
Embedding queries and documents with different models, or ignoring input_type.
Filtering the top-k after search, and shipping “filtered search returns nothing.”
Switching embedding models without re-embedding the stored chunks.
Confusing cosine similarity (higher is better) with cosine distance (lower is better), or BM25’s raw score with its normalized one.
Averaging or concatenating BM25 scores and cosine distances instead of fusing ranks.
Building indexes from different dict objects, so the id(doc)-based fusion treats one chunk as two.
Relying on semantic search alone when users type IDs, error codes, or product numbers.
Leaving out the refusal instruction, so a retrieval miss turns into a confident invented answer.
15.13.4 Connections
Backward: the context window as a scarce resource (How LLMs Work: From Internet Text to ChatGPT, The Elephant-Goldfish Model); indexes built once and queried many times (Relational Database Design & MySQL); interfaces that hide implementations, as with embedding_fn and SearchIndex (Layered Architecture).
Forward: retrieval as a tool leads straight into tool use, MCP servers, and agents; evaluating a RAG system (does retrieval find the right chunks? is the answer faithful to them?) gets its own lecture; and retrieved text is untrusted input, which makes RAG an attack surface for prompt injection, a topic in the security notes.