17 Local LLMs with Ollama: Running Models and RAG on Your Own Machine
17.5.5 Other ways in: the OpenAI-compatible API and Modelfiles |
17.1 Learning Objectives
By the end of this lecture you should be able to:
Explain what it means to run a large language model locally, what you gain (privacy, cost, offline use, version control over the model) and what you give up (quality, speed, hardware).
Estimate the memory a model needs from its parameter count and quantization level, and use that estimate to pick a model that fits your machine.
Use Ollama from the command line and through its REST API (/api/generate, /api/chat), including streaming responses, generation options, and the timing fields in the response.
Run a dedicated local embeddings server (Infinity) and a vector database (ChromaDB), and explain why each of the three services in a local RAG stack has exactly one job.
Trace a question through a complete local RAG pipeline—
embed, retrieve, threshold, prompt, generate— and compute the cosine distances that decide whether the model is called at all. Orchestrate a six-service Docker Compose stack with health checks, named volumes for model weights, and service-name networking.
Test code that depends on model servers without running any model, by mocking the HTTP boundary and using an in-process vector store.
Recognize the classic bugs of local LLM systems: silently truncated context, mis-ordered embeddings, mismatched embedding models, and an exposed, unauthenticated model port.
17.2 Why Run a Model Locally?
Every LLM call you have made in this course so far left your machine. The prompt traveled to a provider’s data center, a model you cannot see ran on hardware you do not own, and the answer came back along with a bill. For many applications that is exactly the right trade. For some it is not, and the question “could this model run on my own hardware instead?” is now a practical engineering question rather than a research project.
Here are the reasons teams actually make that choice:
Privacy and data control. Medical records, student grades, legal documents, unreleased source code. Some data may not leave the building, by law (HIPAA, FERPA, GDPR) or by contract. A model running on your own hardware never sends the prompt anywhere.
Cost at volume. A hosted API bills per token, forever. A local model costs hardware and electricity, and the marginal cost of one more request is close to zero. Embedding an entire document corpus, re-embedding it every time you change the chunking strategy, and running thousands of evaluation queries are all far cheaper locally.
Offline and edge use. A field laptop, a hospital intranet, a CI runner with no internet access. Local models keep working when the network does not.
Reproducibility. A hosted model can be updated or retired under you; the same prompt can quietly start producing different answers. A local model is a file. Pin the file and the behavior is pinned too.
Learning. Running a model yourself makes the concepts from the How LLMs Work: From Internet Text to ChatGPT notes concrete: you will watch a model load into memory, count tokens per second, and hit the context window limit with your own hands.
The trade-offs are just as real. The models that fit on a laptop are far smaller than the frontier models behind hosted APIs, so they reason less reliably, follow instructions less precisely, and know fewer facts. They are also slower on ordinary hardware. The engineering skill this lecture teaches is building a system that gets useful, trustworthy answers despite using a small model. The main tool for that is the one you met in the RAG and Agentic Search notes: retrieval-augmented generation. A small model that is handed the right three paragraphs and told to answer only from them can do much better than a small model answering from memory.
Connections. This lecture is where several earlier threads meet. From the How LLMs Work: From Internet Text to ChatGPT notes: tokens, context windows, and what a model’s weights are. From RAG and Agentic Search: chunking, embeddings, cosine similarity, and the six-step RAG flow. From Prompt Engineering and Evaluation: structured prompts with XML tags. From Docker Containers: Compose services, volumes, and health checks. From REST APIs and FastAPI: Designing and Building Web Interfaces: the backend that wraps it all in an HTTP endpoint. In the RAG notes you built each piece with a hosted API or a mock. Here every piece runs on your laptop.
17.3 The Running Example: MedKB
Throughout these notes we build one system, the Medical Knowledge Base (MedKB): a web app that answers health questions from a small, curated corpus of public medical information. It is a good fit for a local model because it is exactly the kind of domain where you might not want questions leaving the building, and where a wrong but confident answer is worse than no answer.
The corpus mixes three kinds of documents, each derived from public NIH sources (MedQuAD and MedlinePlus):
Type | What it is | Count | Example |
A: Q&A pair | one question and its answer | 20 pairs | “What are the symptoms of ABCD syndrome?” |
B: health topic | an HTML page with <h2> sections | 5 pages, 35 sections | “Type 2 diabetes”: Definition, Causes, Symptoms, ... |
C: fact sheet | a structured record of codes and synonyms | 7 records | “Abdominal aortic aneurysm — |
The app already has a search box, built in an earlier lecture, that does plain keyword matching: a SQL ILIKE ’%query%’ across all three tables. Keyword search fails in two instructive ways:
Vocabulary mismatch. Search for high blood sugar and you get nothing back from the diabetes page unless that exact phrase appears in it. The page talks about “a high level of sugar (glucose) in the blood.” Same meaning, different words.
No answer, only documents. Even a successful match returns a list of snippets. The user asked a question and wanted an answer.
By the end of these notes, MedKB will answer questions in natural language from its own corpus, entirely on local hardware, and will refuse honestly when the corpus has nothing relevant. We will follow two questions all the way through the system:
Q1: “What is type 2 diabetes?” —
well covered by the corpus (a whole health-topic page). Q2: “What’s the capital of France?” —
not covered at all. The correct behavior is to say so, not to answer from the model’s general knowledge, and certainly not to invent a medical-sounding answer.
17.4 What Is a Local LLM?
Formal definition. A local LLM is a language model
whose weights are stored on, and whose inference runs on, hardware
under your control—
In plain English: the model is a file on your disk, and a program on your machine reads that file and generates text from it.
Intuition. Recall from the How LLMs Work: From Internet Text to ChatGPT notes that a trained model is just a very large set of numbers (the weights) plus a fixed recipe for turning input tokens into a probability distribution over the next token. The recipe is public for open-weight models. So running a model needs only two things: the weights file, and an inference engine that implements the recipe efficiently on your CPU or GPU.
┌────────────────────────── your machine ───────────────────────────┐ |
│ │ |
│ weights file inference engine your program │ |
│ (qwen2.5:1.5b, ──► (llama.cpp inside ◄──► (curl, Python, │ |
│ ~1 GB on disk) Ollama; runs on a FastAPI │ |
│ CPU or GPU) backend) │ |
│ │ |
└───────────────────────────────────────────────────────────────────┘ |
nothing crosses this boundary at inference time |
Common misconception. “Local means private, automatically.” Local means the model does not send your data anywhere. Your system still can: a model server bound to every network interface with no authentication (we will see exactly this in the Compose file later) is readable by anyone on your Wi-Fi. Logs, crash reports, and the frontend’s analytics script are all still leak paths. Locality is a property of one component, not of the application.
Practical implications. Before choosing a model you need to know whether it fits your hardware. That depends almost entirely on two numbers: how many parameters it has, and how many bits each parameter is stored in.
17.4.1 Parameters, quantization, and memory
Formal definition. Quantization stores each model
weight in fewer bits than it was trained with—
In plain English: rounding every number in the model so it takes less space.
Intuition. A photo saved as a high-quality JPEG looks almost identical to the original at a fraction of the file size. Weights are similar: most of them do not need 16 bits of precision for the model to behave almost the same. Quality does degrade, slowly at first and then quickly; 4-bit is the common sweet spot, and below about 3 bits models tend to fall apart.
The memory needed just to hold the weights is roughly:
memory (bytes) ≈ number of parameters × bits per weight / 8 |
Worked example. MedKB uses qwen2.5:1.5b, a model with about 1.5 billion parameters. How much memory does it need?
Precision | Bits/weight | Arithmetic | Weights in memory |
FP32 (training) | 32 | 1.5e9 × 4 bytes | 6.0 GB |
FP16 | 16 | 1.5e9 × 2 bytes | 3.0 GB |
Q8_0 | about 8.5 | 1.5e9 × 1.06 bytes | about 1.6 GB |
Q4_K_M (Ollama’s usual default) | about 4.8 | 1.5e9 × 0.6 bytes | about 0.9 GB |
Step by step for the last row: 4.8 bits is 0.6 bytes (4.8 / 8); 1.5 billion times 0.6 bytes is 0.9 billion bytes, about 0.9 GB. That matches the roughly 1 GB download you will see when you pull the model. (The “bits per weight” for quantized formats is a little above the nominal 4 or 8 because each block of weights also stores a scale factor.)
Weights are not the whole story. While generating, the model also
keeps a KV cache—
Applying the same arithmetic to other sizes gives the table that decides what you can run:
Model size | Q4 weights | Runs comfortably on |
0.5B–1.5B | 0.4–1 GB | any recent laptop, CPU only |
7B–8B | 4.5–5 GB | 16 GB laptop; fast with an 8 GB GPU or Apple silicon |
13B–14B | 8–9 GB | 32 GB machine or a 12–16 GB GPU |
70B | about 40 GB | workstation GPU(s) or 64 GB+ unified memory |
Common misconception. “A bigger model is always better, so run the biggest one that loads.” A model that only just fits will swap to disk or fall back to the CPU and run at a word every few seconds. For an interactive app, a smaller model that answers in two seconds usually beats a larger one that answers in forty. For RAG in particular, the retrieved context does much of the knowledge work, so a small model is often good enough. MedKB uses a 1.5B model on purpose: it runs on every student’s laptop with no GPU.
17.4.2 The local LLM landscape
Ollama is not the only way to run a model locally. The common tools divide by audience:
Tool | What it is | Typical use |
llama.cpp | C/C++ inference engine, GGUF model format | the engine underneath many other tools |
Ollama | model server + registry + CLI built on llama.cpp | developers; apps that call a local REST API |
LM Studio | desktop GUI | trying models interactively |
vLLM, TGI | high-throughput GPU servers | serving many concurrent users in production |
Hugging Face transformers | Python library | research, fine-tuning, full control |
We use Ollama because it gives a developer the shortest path from “nothing installed” to “a model behind an HTTP API,” and it runs cleanly in Docker.
17.5 Ollama
Formal definition. Ollama is a local model server. It downloads open-weight models from a registry, stores them on disk, loads them into memory on demand, and serves generation (and embedding) requests over a REST API on port 11434. A command-line client talks to the same server.
In plain English: Docker for language models. ollama pull fetches a model like docker pull fetches an image, and ollama run starts a chat with it.
Intuition. Ollama separates the slow, rare work (downloading a gigabyte, loading it into memory) from the fast, frequent work (answering a request). A model is loaded on the first request, kept in memory for a while (five minutes by default), and unloaded when idle.
ollama CLI ─┐ |
curl ───────┼──► HTTP :11434 ──► Ollama server ──► model loaded in RAM/VRAM |
Python ─────┘ │ (kept alive ~5 min, |
▼ then unloaded) |
~/.ollama/models |
(weights on disk) |
Common misconception. “The first request is slow, so Ollama is slow.” The first request after a pull or an idle period includes loading the weights into memory. The response tells you exactly how long that took (load_duration, below). Measure the second request, not the first.
17.5.1 The CLI in five commands
ollama pull qwen2.5:1.5b # download the model (about 1 GB, once)
ollama list # models on disk
ollama run qwen2.5:1.5b # interactive chat in the terminal
ollama ps # models currently loaded in memory
ollama rm qwen2.5:1.5b # delete from diskA model name has the form family:tag. The tag usually encodes the size and sometimes the variant and quantization, as in qwen2.5:1.5b, qwen2.5:7b-instruct-q8_0, or llama3.2:3b. Without an explicit quantization, the tag points at the registry’s default, usually a 4-bit quantization. ollama show qwen2.5:1.5b prints the architecture, parameter count, context length, and quantization of a model you have pulled.
Worked example: a closed-book answer. Before building anything, ask the bare model a question from the MedKB corpus. Our corpus has a Q&A pair about the symptoms of ABCD syndrome, a rare genetic disorder.
$ ollama run qwen2.5:1.5b "What are the symptoms of ABCD syndrome?"
ABCD syndrome is a condition characterized by Abdominal pain, Bloating,
Constipation and Diarrhea ...The output above is illustrative and yours will differ, but the pattern is reliable. ABCD actually stands for albinism, black lock (a patch of dark hair), cell migration disorder of the neurocytes of the gut, and deafness. The small model has never seen enough text about this rare condition, so it does what language models do: it produces fluent, plausible text that matches the shape of an answer. It guessed that the letters stand for symptoms and filled them in. This is the problem the rest of the lecture solves. We will not make the model know more; we will hand it the right document and require it to answer from that document.
17.5.2 The REST API
Everything the CLI does goes through the HTTP API, and that API is how applications use Ollama. The two generation endpoints are:
Endpoint | Input | Use it for |
POST /api/generate | a single prompt string | one-shot completions, like a RAG answer |
POST /api/chat | a list of messages with roles | multi-turn conversations |
POST /api/embed | text(s) to embed | embeddings from an embedding model |
GET /api/tags | nothing | list pulled models (a cheap health check) |
Here is the call MedKB makes, issued by hand with curl:
curl http://localhost:11434/api/generate -d '{
"model": "qwen2.5:1.5b",
"prompt": "In one sentence, what is insulin?",
"stream": false
}'With "stream": false, the server waits until generation finishes and returns one JSON object:
{
"model": "qwen2.5:1.5b",
"created_at": "2026-10-17T14:02:11.482Z",
"response": "Insulin is a hormone made by the pancreas that lets cells take in glucose from the blood.",
"done": true,
"done_reason": "stop",
"total_duration": 1843021000,
"load_duration": 21000000,
"prompt_eval_count": 36,
"prompt_eval_duration": 312000000,
"eval_count": 21,
"eval_duration": 1490000000
}The field you want is response. The rest is free instrumentation, and it is worth learning to read:
All durations are in nanoseconds.
load_duration: time to load the model. Tiny here (21 ms) because the model was already in memory; several seconds on a cold start.
prompt_eval_count and prompt_eval_duration: how many input tokens were processed and how long that took. This is the “reading” phase, sometimes called prefill.
eval_count and eval_duration: how many tokens were generated and how long that took. This is the “writing” phase.
Worked example: tokens per second. From the response above:
generation speed = eval_count / (eval_duration / 1e9) |
= 21 / (1490000000 / 1e9) |
= 21 / 1.49 |
≈ 14.1 tokens/second |
|
prompt speed = prompt_eval_count / (prompt_eval_duration / 1e9) |
= 36 / 0.312 |
≈ 115 tokens/second |
Two lessons are hiding in these numbers. First, reading is much faster than writing: the model processes the whole prompt in parallel but must generate output one token at a time. So for RAG, a long retrieved context costs much less than a long answer. Second, 14 tokens per second is roughly reading speed. That is fine if you stream the output to the user and painful if you make them stare at a spinner until the whole answer is ready.
17.5.3 Streaming
With "stream": true (the default), Ollama returns newline-delimited JSON: one small JSON object per line, each carrying the next piece of text, followed by a final object with "done": true and the timing fields.
{"model":"qwen2.5:1.5b","response":"Ins","done":false} |
{"model":"qwen2.5:1.5b","response":"ulin","done":false} |
{"model":"qwen2.5:1.5b","response":" is","done":false} |
... |
{"model":"qwen2.5:1.5b","response":"","done":true,"eval_count":21, ...} |
The following complete program is a terminal chat client for a local model. It uses /api/chat, keeps the conversation history, and prints tokens as they arrive.
"""chat.py -- a streaming terminal chat client for a local Ollama model.
Run: pip install httpx && python chat.py
"""
import json
import sys
import httpx
OLLAMA_URL = "http://localhost:11434"
MODEL = "qwen2.5:1.5b"
def stream_chat(messages: list[dict], model: str = MODEL) -> str:
"""Send the conversation to Ollama, print the reply as it streams in,
and return the full reply text."""
body = {
"model": model,
"messages": messages,
"stream": True,
"options": {"temperature": 0.2}, # low randomness for factual Q&A
}
parts: list[str] = []
# httpx.stream keeps the connection open and lets us read line by line.
with httpx.stream("POST", f"{OLLAMA_URL}/api/chat", json=body, timeout=120.0) as response:
response.raise_for_status()
for line in response.iter_lines():
if not line:
continue
event = json.loads(line) # one JSON object per line
piece = event.get("message", {}).get("content", "")
print(piece, end="", flush=True)
parts.append(piece)
if event.get("done"):
tokens = event["eval_count"]
seconds = event["eval_duration"] / 1e9 # nanoseconds -> seconds
print(f"\n[{tokens} tokens, {tokens / seconds:.1f} tok/s]", file=sys.stderr)
return "".join(parts)
def main() -> None:
# The chat endpoint is stateless: we resend the whole history every turn.
history = [
{"role": "system", "content": "You are a concise assistant. Say so when you are unsure."}
]
while True:
try:
user = input("\nyou> ").strip()
except EOFError:
break
if user in {"", "quit", "exit"}:
break
history.append({"role": "user", "content": user})
print("model> ", end="")
reply = stream_chat(history)
history.append({"role": "assistant", "content": reply})
if __name__ == "__main__":
main()A sample session (output will vary from run to run and machine to machine):
you> What does the pancreas do? |
model> The pancreas makes digestive enzymes and hormones such as insulin |
and glucagon, which regulate blood sugar. |
[24 tokens, 15.3 tok/s] |
|
you> Which of those lowers it? |
model> Insulin lowers blood sugar; glucagon raises it. |
[11 tokens, 15.8 tok/s] |
The important design points:
The server is stateless. The second question, “Which of those lowers it?”, only makes sense because we resent the first exchange in history. This is the “goldfish” from the The Elephant-Goldfish Model notes: the model remembers nothing between requests, so the client owns the memory. It also means every turn sends a longer prompt than the last.
Timeouts matter. A local model on a CPU can legitimately take a minute for a long answer. httpx’s default 5-second timeout would kill the request. We set 120 seconds.
Roles. The system message sets behavior, and user/assistant alternate. Ollama inserts these into the model’s chat template (the special tokens the model was trained on), which is why /api/chat is the right endpoint for conversations.
17.5.4 Generation options and the context-window trap
Both endpoints accept an options object. The ones you will actually use:
Option | Meaning | MedKB setting |
temperature | randomness of sampling; 0 is (nearly) deterministic | low (0–0.3): we want faithful, not creative |
seed | fixes the random sampler for reproducible output | set it in tests and evaluations |
num_predict | maximum tokens to generate | cap it so a rambling answer cannot take minutes |
num_ctx | context window size in tokens | see below |
num_ctx deserves its own warning. A model may support a
32K-token context, but Ollama allocates a much smaller window by
default—
Worked example: the silent truncation bug. Suppose a later version of MedKB retrieves the top 20 chunks instead of the top 3, with long health-topic sections, and the prompt grows to 5,000 tokens while num_ctx is 4,096. What does the model see?
the prompt you built (5,000 tokens) |
┌──────────────────┬────────────────────────────────────┬──────────┐ |
│ system guardrail │ <context> 20 chunks </context>│ Question │ |
└──────────────────┴────────────────────────────────────┴──────────┘ |
◄── dropped ──────►◄──────── what the model sees (4,096) ─────────► |
The beginning of the prompt is discarded to make room—
17.5.5 Other ways in: the OpenAI-compatible API and Modelfiles
Ollama also serves an OpenAI-compatible API under /v1, so existing code written against the OpenAI SDK can be pointed at a local model by changing one line:
from openai import OpenAI
# The SDK requires an api_key; Ollama ignores its value.
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
model="qwen2.5:1.5b",
messages=[{"role": "user", "content": "Name two functions of the liver."}],
)
print(reply.choices[0].message.content)This is the boundary discipline from the Layered Architecture notes paying off: if the model is reached through a narrow, standard interface, swapping a hosted model for a local one (or back) is a configuration change, not a rewrite.
A Modelfile packages a base model with default settings, the way a Dockerfile packages an image:
# Modelfile |
FROM qwen2.5:1.5b |
PARAMETER temperature 0.2 |
PARAMETER num_ctx 8192 |
SYSTEM """You answer medical questions only from provided context. |
If the context does not contain the answer, say so.""" |
ollama create medkb-qwen -f Modelfile
ollama run medkb-qwenModelfiles are convenient for sharing a configured model, but for an application we prefer to send the system prompt and options with each request from code, where they are version-controlled with the rest of the app and visible in code review.
17.5.6 Ollama in Docker
In MedKB, Ollama runs as a Docker Compose service rather than a desktop app. The service definition is short, and every line matters:
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama # pulled models live here
networks:
- medkb_network
healthcheck:
# `ollama list` succeeds once the server is up, even before a model is pulled.
test: ["CMD", "ollama", "list"]
interval: 10s
timeout: 5s
retries: 15
start_period: 30sThe named volume is not optional. The container’s filesystem is thrown away when it is recreated. Without ollama_data mounted at /root/.ollama, every docker compose up after a rebuild would re-download the model. With it, the model is pulled once and survives restarts.
Pulling happens inside the container. The image ships the server, not the model. On first run: docker compose exec ollama ollama pull qwen2.5:1.5b.
Service names are hostnames. Inside the Compose network the backend reaches Ollama at http://ollama:11434, not localhost. From your laptop’s terminal, through the published port, it is http://localhost:11434. Mixing these up is the most common “connection refused” in this lecture.
The published port is unauthenticated. "11434:11434" publishes the port on every interface of your machine. Ollama has no login, so anyone who can reach your machine on that port can use your model, list your models, and pull new ones. For development on a trusted network that is acceptable. Otherwise publish only on loopback ("127.0.0.1:11434:11434") or, since only the backend needs Ollama, do not publish the port at all.
GPUs need extra configuration. This service runs on the CPU. GPU acceleration in a container requires the NVIDIA container toolkit and a deploy.resources section. Docker Desktop on macOS cannot pass the Apple GPU into a container, so on a Mac the native Ollama app is much faster than the containerized one.
17.6 Local Embeddings with Infinity
The RAG flow needs two kinds of model: a generation model to write the answer, and an embedding model to turn text into vectors for search. The RAG and Agentic Search notes covered what an embedding is. The question here is where to run one.
Formal definition. An embeddings server is a service that hosts one or more embedding models and returns a fixed-length vector for each input text, typically over an HTTP API. Infinity is an open-source embeddings server that can serve almost any sentence-transformers model from Hugging Face behind an OpenAI-compatible /embeddings endpoint.
In plain English: a dedicated service whose only job is turning text into vectors.
Why not use Ollama for this too? Ollama does have an /api/embed endpoint and can be the right choice for a quick prototype. MedKB uses a separate server for three reasons:
Model choice. Ollama serves a curated set of embedding models. Infinity serves essentially any sentence-transformers model, so you can choose and compare models by name.
Control. An embedding model turns per-token vectors into one vector per text by pooling (for example, averaging them). With a named sentence-transformers model, the pooling strategy is a known, documented property you can reason about and compare.
One job per service. Embedding is batch-heavy (the whole corpus at ingest time); generation is latency-sensitive (one user waiting). Separate services can be scaled, restarted, and swapped independently.
MedKB uses sentence-transformers/all-MiniLM-L6-v2: small (about 90 MB), fast on a CPU, freely downloadable, mean pooling, and 384 dimensions per vector. It has one limit you must respect: it reads at most 256 word-piece tokens of input and silently ignores the rest. MedKB’s chunkers keep chunks around 80 tokens, well inside that limit. A 1,000-token chunk would be embedded from its first 256 tokens only, and nothing would tell you.
The request and response mirror OpenAI’s embeddings API:
curl http://localhost:7997/embeddings -H "Content-Type: application/json" -d '{
"input": ["What is type 2 diabetes?", "Seizures -- Causes: ..."],
"model": "sentence-transformers/all-MiniLM-L6-v2"
}'{
"object": "list",
"data": [
{"object": "embedding", "embedding": [0.012, -0.087, ...], "index": 0},
{"object": "embedding", "embedding": [-0.041, 0.033, ...], "index": 1}
],
"model": "sentence-transformers/all-MiniLM-L6-v2",
"usage": {"prompt_tokens": 19, "total_tokens": 19}
}17.6.1 The ordering trap
Notice that every item in data carries an index. That is there because the API does not promise that data comes back in input order. A server may process a batch in parallel and return results as they finish. The index tells you which input each vector belongs to.
Worked example. MedKB embeds its corpus in batches of 32. Suppose a batch starts with these three chunks, and the server returns the vectors out of order:
input position chunk text response data[] |
────────────── ───────────────────────────────── ───────────────────── |
0 "Type 2 diabetes -- Definition: ..." {index: 2, emb: v_C} |
1 "Seizures -- Causes: ..." {index: 0, emb: v_A} |
2 "High cholesterol -- Treatment: ..." {index: 1, emb: v_B} |
If the client trusts array order, it pairs the diabetes chunk with v_C, which is the embedding of the cholesterol text. Every chunk is stored next to the wrong vector. Nothing crashes, the shapes all match, and ChromaDB accepts everything. The symptom appears much later and far away: a question about diabetes retrieves text about seizures, and the answer is wrong for no visible reason.
The fix is one line of logic: sort data by index before extracting the vectors, so position i in your output corresponds to position i in your input. Writing that client function, embed_texts(), is part 1 of the lecture activity, and one of its tests sends the response deliberately out of order.
Common misconception. “The generation model and the embedding model must be the same model.” They are completely independent; MedKB generates with Qwen and embeds with MiniLM. What must match is the embedding model used for documents at ingest time and the one used for questions at query time. Vectors from different embedding models live in unrelated spaces, and comparing them gives meaningless distances. If you change the embedding model, you must re-embed the entire corpus.
17.7 Storing and Searching Vectors with ChromaDB
In the RAG and Agentic Search notes you wrote a brute-force vector index in a few dozen lines. MedKB replaces it with ChromaDB, an open-source vector database, running as its own Compose service.
Formal definition. A vector database stores vectors together with their source text and metadata, maintains an index for fast approximate nearest-neighbor search, and answers “which stored vectors are closest to this query vector?”
In plain English: a table where each row is a chunk of text, its embedding, and some labels, plus a very fast “find similar rows” query.
ChromaDB is the cheap service in the stack: it has no model and downloads nothing. It only stores and compares vectors that Infinity produced.
MedKB keeps all three document types in a single collection, distinguished by a doc_type metadata field. Here is the whole wrapper, from backend/app/rag/vector_store.py:
import chromadb
COLLECTION_NAME = "medkb"
def get_client(host: str, port: int) -> chromadb.ClientAPI:
"""Connect to the ChromaDB server running as its own Compose service."""
return chromadb.HttpClient(host=host, port=port)
def get_or_create_collection(client: chromadb.ClientAPI, name: str = COLLECTION_NAME) -> chromadb.Collection:
# hnsw:space = cosine: rank results by cosine distance.
return client.get_or_create_collection(name, metadata={"hnsw:space": "cosine"})
def add_chunks(collection, chunk_ids, doc_types, texts, embeddings) -> None:
"""Add a batch of chunks (with their pre-computed embeddings) to the collection."""
collection.add(
ids=chunk_ids,
embeddings=embeddings,
documents=texts,
metadatas=[{"doc_type": dt} for dt in doc_types],
)
def query(collection, query_embedding: list[float], top_k: int = 3) -> list[dict]:
"""Return the top-k nearest chunks, each with text, doc_type, and distance."""
if collection.count() == 0:
return []
result = collection.query(query_embeddings=[query_embedding], n_results=min(top_k, collection.count()))
return [
{"text": doc, "doc_type": meta["doc_type"], "distance": dist}
for doc, meta, dist in zip(
result["documents"][0], result["metadatas"][0], result["distances"][0]
)
]Points worth noticing:
We supply the embeddings. ChromaDB can embed text for you with a built-in model, but then you no longer control which model is used. Passing embeddings= explicitly keeps the Infinity model the single source of truth for both ingest and query.
The text is stored next to the vector. A vector alone is useless at answer time; the prompt needs the words. This is the same reason your hand-written index stored chunk text.
hnsw:space is set at creation time. HNSW is the approximate nearest-neighbor index described in the RAG notes. The distance metric is fixed when the collection is created and cannot be changed afterward, so it is passed to get_or_create_collection.
Results are a nested list. collection.query accepts a batch of queries, so each result field is a list per query. We send one query and take element [0].
17.7.1 Cosine distance, by hand
With hnsw:space set to cosine, ChromaDB reports a distance, not a similarity:
cosine distance = 1 − cosine similarity |
= 1 − (a · b) / (‖a‖ ‖b‖) |
|
0.0 identical direction (most similar) |
1.0 perpendicular (unrelated) |
2.0 opposite direction |
Smaller is better. This flips the intuition from the RAG notes, where larger similarity was better, and the flip causes real bugs: a threshold check written as distance > 0.6 and one written as similarity > 0.6 mean opposite things.
Worked example. Real embeddings have 384 dimensions, but the arithmetic is the same in 3. Suppose the collection holds three chunks, and a question about diabetes embeds to q = [1, 0, 0]:
Chunk | Vector | Similarity with q | Distance |
diabetes | [1.0, 0.0, 0.0] | 1 / (1 × 1) = 1.000 | 0.000 |
cholesterol | [0.9, 0.1, 0.0] | 0.9 / (1 × 0.906) = 0.994 | 0.006 |
seizures | [0.0, 1.0, 0.0] | 0 / (1 × 1) = 0.000 | 1.000 |
For the cholesterol row: the dot product is 1×0.9 + 0×0.1 + 0×0 = 0.9; its length is √(0.81 + 0.01) = √0.82 ≈ 0.906; so the similarity is 0.9 / 0.906 ≈ 0.994 and the distance is 0.006. A top_k=2 query returns diabetes, then cholesterol. These exact vectors appear in the project’s test_vector_store.py, which checks that the nearest chunk comes back first.
17.8 The Local RAG Pipeline
We now have three services, each with one job:
Service | Job | Model | Address inside Compose |
Ollama | text generation | qwen2.5:1.5b | http://ollama:11434 |
Infinity | embeddings | all-MiniLM-L6-v2 | http://infinity:7997 |
ChromaDB | vector storage and search | none | chromadb:8000 |
The pipeline has two phases. Ingestion runs once and fills the vector store. Querying runs on every question.
INGESTION (once) |
───────────────── |
qa_pairs.json ─┐ ┌──────────┐ ┌──────────┐ |
health_topic ├─► type-aware ───► │ Infinity │ ───► │ ChromaDB │ |
.html │ chunking │ embed │ │ "medkb" │ |
fact_sheets ─┘ (~80-100 chunks) └──────────┘ └──────────┘ |
.json batches of 32 |
|
QUERYING (every question) |
───────────────────────── |
question ─► Infinity ─► ChromaDB ─► best distance ─── > 0.6 ──► "no match" (stop) |
embed top 3 │ |
≤ 0.6 |
▼ |
build tagged prompt ─► Ollama ─► answer + sources |
17.8.1 Ingestion
Ingestion reuses the type-aware chunking from earlier lectures: each document type is split along its natural boundaries.
Q&A pairs are atomic: one pair, one chunk, never split, formatted as "Q: ...\nA: ...". 20 chunks.
Health topics split on <h2> section boundaries, with the page title and section heading prepended to each chunk so it stands on its own (“Type 2 diabetes – Definition: ...”). A section longer than 80 tokens falls back to a sliding window. At least 35 chunks, one per section.
Fact sheets become one chunk per populated field (“ABCD syndrome – Synonyms: ...”). 26 chunks from 7 records.
def ingest_corpus(collection: chromadb.Collection, infinity_base_url: str, batch_size: int = 32) -> int:
"""Embed and store the whole corpus in `collection`. Returns the chunk count.
Skips re-ingesting if the collection is already populated, so backend
restarts during development don't re-embed the corpus every time.
"""
if collection.count() > 0:
return collection.count()
triples = build_corpus_chunks() # (chunk_id, doc_type, text) for every chunk
for i in range(0, len(triples), batch_size):
batch = triples[i : i + batch_size]
ids = [t[0] for t in batch]
doc_types = [t[1] for t in batch]
texts = [t[2] for t in batch]
embeddings = embed_texts(texts, base_url=infinity_base_url)
vector_store.add_chunks(collection, ids, doc_types, texts, embeddings)
return len(triples)Two design decisions:
Batching. One HTTP request per chunk would mean about a hundred round trips; batches of 32 need four. The embedding model also runs faster on a batch than on the same texts one by one. Batching is also exactly why the ordering trap matters: with one text per request there is nothing to mis-order.
Idempotence. If the collection already has data, ingestion does nothing. ChromaDB’s data lives in a named volume, so after the first run, restarting the backend costs nothing. The price: if you change the chunker or the embedding model, the old vectors stay. You must delete the collection (or the chromadb_data volume) to re-ingest. A production system would store a corpus version or embedding model name in the collection metadata and re-ingest when it changes.
17.8.2 Querying: the whole pipeline in one function
Here is answer_question from backend/app/rag/pipeline.py. Every lecture in this unit contributes one line:
NO_MATCH_ANSWER = "I don't have enough information in the knowledge base to answer that question."
# Cosine distance (1 - cosine_similarity) above this is treated as "no good match".
DEFAULT_DISTANCE_THRESHOLD = 0.6
def answer_question(
question: str,
collection: chromadb.Collection,
infinity_base_url: str,
ollama_base_url: str,
top_k: int = 3,
distance_threshold: float = DEFAULT_DISTANCE_THRESHOLD,
) -> dict:
query_embedding = embed_texts([question], base_url=infinity_base_url)[0] # Infinity
results = vector_store.query(collection, query_embedding, top_k=top_k) # ChromaDB
if not results or results[0]["distance"] > distance_threshold: # the gate
return {"answer": NO_MATCH_ANSWER, "matched": False, "sources": []}
chunks = [(r["doc_type"], r["text"]) for r in results]
prompt = build_prompt(question, chunks) # tagged prompt
answer = generate(prompt, base_url=ollama_base_url) # Ollama
return {"answer": answer, "matched": True, "sources": [r["text"] for r in results]}17.8.3 The threshold gate
The most important line is the if. It decides whether the model is called at all, and it is checked before generation for two reasons:
Correctness. If the best chunk is unrelated to the question, handing it to the model does not help. A small model given irrelevant context and a question will often answer anyway, either from memory (which we have seen is unreliable) or by stretching the irrelevant context to fit. The safest answer to an out-of-scope question is a fixed, honest “I don’t know,” and that decision should be made by deterministic code, not by asking the model to judge its own context.
Cost. On a CPU, generation is the slowest step by far (seconds), while embedding and search take milliseconds. Rejecting early saves the expensive call.
Worked example: tracing Q1 and Q2. The distances below are illustrative but representative of what MiniLM produces for this corpus.
Q1: "What is type 2 diabetes?" |
1. embed → 384-dim vector q1 |
2. top 3 from Chroma: |
0.21 narrative_excerpt "Type 2 diabetes -- Definition: Type 2 diabetes is a lifelong ..." |
0.38 narrative_excerpt "Type 2 diabetes -- Causes: Insulin, produced by beta cells ..." |
0.44 narrative_excerpt "Type 2 diabetes -- Symptoms: ..." |
3. gate: best = 0.21 ≤ 0.6 → continue |
4. build_prompt(question, 3 chunks) |
5. Ollama generate → "Type 2 diabetes is a lifelong (chronic) disease in which |
the level of sugar (glucose) in the blood is high ..." |
6. return matched=True, sources=[3 chunk texts] |
|
Q2: "What's the capital of France?" |
1. embed → q2 |
2. top 3 from Chroma: |
0.83 fact_sheet "Abdominal aortic aneurysm -- Semantic Group: Disorders" |
0.86 qa_pair "Q: What causes Abetalipoproteinemia? ..." |
0.88 ... |
3. gate: best = 0.83 > 0.6 → STOP |
4. return matched=False, answer=NO_MATCH_ANSWER, sources=[] |
(Ollama is never called) |
Notice that Q2 still gets three results. Nearest-neighbor search always returns the nearest neighbors, however far away they are. There is no such thing as “no results” from a vector search over a non-empty collection. That is precisely why the threshold must exist: without it, the model would receive an aneurysm fact sheet as the context for a geography question.
Choosing the threshold. 0.6 is a starting point, not a law. The right value depends on the embedding model and the corpus:
Threshold | Effect | Failure mode |
too low (e.g. 0.2) | only near-verbatim questions pass | refuses answerable questions phrased differently |
about right | paraphrases pass, off-topic questions stop | a few borderline cases go either way |
too high (e.g. 0.95) | almost everything passes | model answers off-topic questions from irrelevant context |
The way to choose it is empirical: collect a set of in-scope and out-of-scope questions, record the best distance for each, and pick the value that best separates the two groups. That is a small evaluation, which is the subject of the next lecture.
17.8.4 The prompt
Once past the gate, the retrieved chunks are wrapped in XML tags named for their type, as in the Prompt Engineering and Evaluation notes:
SYSTEM_GUARDRAIL = "Only use the information provided in the tagged context to answer questions."
DOC_TYPE_TAGS = {"qa_pair": "qa_pair", "health_topic": "narrative_excerpt", "fact_sheet": "fact_sheet"}
def wrap_chunk(doc_type: str, chunk_text: str) -> str:
tag = DOC_TYPE_TAGS[doc_type]
return f"<{tag}>{chunk_text}</{tag}>"
def build_prompt(question: str, chunks: list[tuple[str, str]]) -> str:
wrapped = "\n".join(wrap_chunk(doc_type, text) for doc_type, text in chunks)
return f"{SYSTEM_GUARDRAIL}\n\n<context>\n{wrapped}\n</context>\n\nQuestion: {question}"For Q1 the model receives:
Only use the information provided in the tagged context to answer questions. |
|
<context> |
<narrative_excerpt>Type 2 diabetes -- Definition: Type 2 diabetes is a lifelong |
(chronic) disease in which there is a high level of sugar (glucose) in the blood. ... |
</narrative_excerpt> |
<narrative_excerpt> |
Type 2 diabetes -- Causes: Insulin, produced by beta cells in |
the pancreas, ...</narrative_excerpt> |
<narrative_excerpt>Type 2 diabetes -- Symptoms: ...</narrative_excerpt> |
</context> |
|
Question: What is type 2 diabetes? |
The tags give the model a clear structural signal for “this is retrieved material, not an instruction.” That matters more for a small model than a large one: a 1.5B model follows instructions less reliably, so every bit of structure helps. The tags also prepare for prompt-injection defenses later in the course: if a retrieved document ever contains text like “ignore previous instructions,” it is clearly inside a context tag.
Common misconception. “The guardrail sentence guarantees the
model only uses the context.” It does not; it makes that behavior
more likely. Small models in particular still blend in outside
knowledge. That is why MedKB returns the sources with every
answer—
17.8.5 Calling the generator
The last step posts the prompt to Ollama’s /api/generate with "stream": false and returns the response field, exactly as in the curl example earlier. Writing that client function, generate(), is part 2 of the activity.
MedKB uses the non-streaming form because the endpoint returns the answer and its sources as one JSON object. That keeps the backend and frontend simple, at the price of the user waiting for the whole answer. Streaming the answer to the browser (with server-sent events) is a natural extension and one of the exercises.
17.9 Orchestrating the Stack
MedKB now has six services:
┌──────────────┐ |
browser ─────► │ frontend │ :5173 React + Vite |
└──────┬───────┘ |
│ /api/* |
┌──────▼───────┐ |
│ backend │ :8000 FastAPI |
└──┬───┬───┬───┬┘ |
┌────────────────┘ │ │ └──────────────────┐ |
▼ ▼ ▼ ▼ |
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ |
│ mysql │ │ chromadb │ │ ollama │ │ infinity │ |
│ :3306 │ │ :8000 │ │ :11434 │ │ :7997 │ |
│ keyword │ │ vectors │ │ generate │ │ embed │ |
│ search │ │ │ │ │ │ │ |
└────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ |
▼ ▼ ▼ ▼ |
mysql_data chromadb_data ollama_data infinity_data |
(named volumes: survive restarts) |
Three Compose features make the stack start reliably.
Health checks plus depends_on: condition: service_healthy. A container being started is not the same as the service inside it being ready. Infinity has to download and load its model before it can answer; that takes up to a minute on first run, which is why its health check allows a 60-second start_period and 20 retries. The backend waits for all four dependencies to report healthy:
infinity:
image: michaelf34/infinity:latest
command: ["v2", "--model-id", "sentence-transformers/all-MiniLM-L6-v2", "--port", "7997"]
volumes:
- infinity_data:/app/.cache # downloaded model weights
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:7997/health || exit 1"]
interval: 10s
retries: 20
start_period: 60s
backend:
build: ./backend
env_file: .env
depends_on:
mysql: {condition: service_healthy}
chromadb: {condition: service_healthy}
ollama: {condition: service_healthy}
infinity: {condition: service_healthy}Note a subtle gap: Ollama’s health check (ollama list) passes as soon as the server is up, even if no model has been pulled. “Healthy” here means “the server answers,” not “the model is ready.” If you forget the pull step, the stack comes up green and the first question fails with a “model not found” error from Ollama.
Configuration through the environment. Service addresses are settings, not constants:
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
CHROMADB_HOST: str = "chromadb"
CHROMADB_PORT: int = 8000
OLLAMA_BASE_URL: str = "http://ollama:11434"
INFINITY_BASE_URL: str = "http://infinity:7997"The defaults are the Compose service names. To run the backend outside Docker against a native Ollama app, you change one environment variable (OLLAMA_BASE_URL=http://localhost:11434) and no code.
Lazy ingestion. The /api/ask endpoint calls ingest_corpus on every request, which is a no-op after the first:
@router.post("", response_model=AskResponse)
async def ask(payload: AskRequest, request: Request) -> AskResponse:
collection = request.app.state.rag_collection
# Ingests once (on the first request after startup) and is a no-op after that.
ingest_corpus(collection, infinity_base_url=settings.INFINITY_BASE_URL)
result = answer_question(
payload.question,
collection=collection,
infinity_base_url=settings.INFINITY_BASE_URL,
ollama_base_url=settings.OLLAMA_BASE_URL,
)
return AskResponse(question=payload.question, **result)Why not ingest at startup, in the lifespan handler? Because then a backend whose embedding client is broken (or not yet written, as in the activity) could not start at all, and the keyword-search tab would go down with it. Lazy ingestion keeps the rest of the app available. The cost is that the first question waits for the whole corpus to be embedded.
One more detail for later: ask is declared async, but ingest_corpus and answer_question make blocking HTTP calls. While one request waits several seconds for Ollama, the event loop is blocked and no other request is served. That is acceptable for a single-user classroom demo and a real problem for a multi-user service (see the advanced exercises).
17.10 Testing Without a Model
A test suite that needs a 1 GB model, a running Ollama server, and a minute of startup is a test suite nobody runs. MedKB’s tests run in about a second with no Docker and no model, using two techniques.
Mock the HTTP boundary. The client functions are thin wrappers around httpx.post. Tests replace httpx.post with a fake that records the request and returns a canned response:
def test_embed_texts_returns_vectors_in_input_order(monkeypatch):
def fake_post(url, json=None, **kwargs):
return _FakeResponse(
{
"data": [
{"embedding": [9.0, 9.0], "index": 1},
{"embedding": [1.0, 1.0], "index": 0},
]
}
)
monkeypatch.setattr(httpx, "post", fake_post)
result = embed_texts(["first", "second"], base_url="http://infinity:7997")
# index 0 -> "first", index 1 -> "second", regardless of response order
assert result == [[1.0, 1.0], [9.0, 9.0]]The fake returns the vectors in reverse order on purpose. This test pins down exactly the ordering bug from Local Embeddings with Infinity, which a live server might trigger only rarely. A mocked boundary lets you produce the rare case on every run.
Use a real vector store in-process. ChromaDB has an EphemeralClient that runs entirely in memory, so the pipeline test uses a real collection with hand-made 3-dimensional vectors and fakes only the two model calls:
def test_answer_question_returns_no_match_for_out_of_scope_question(monkeypatch):
collection = _seeded_collection() # one diabetes chunk at [1, 0, 0]
monkeypatch.setattr(pipeline, "embed_texts", lambda texts, base_url: [[0.0, 0.0, 1.0]])
def fail_if_called(prompt, base_url):
raise AssertionError("generate() should not be called below the similarity threshold")
monkeypatch.setattr(pipeline, "generate", fail_if_called)
result = pipeline.answer_question(
"What is the airspeed velocity of an unladen swallow?",
collection=collection,
infinity_base_url="http://infinity:7997",
ollama_base_url="http://ollama:11434",
)
assert result["matched"] is False
assert result["answer"] == pipeline.NO_MATCH_ANSWERThe query vector [0, 0, 1] is perpendicular to the only stored vector [1, 0, 0], so the distance is exactly 1.0, above the 0.6 threshold. The fake generate raises if it is called, so the test proves not just that the answer is “no match” but that the model was never invoked. This is the threshold gate’s contract, checked mechanically.
Note the patch target: pipeline.embed_texts, not infinity_client.embed_texts. pipeline.py did from .infinity_client import embed_texts, which copied the function into pipeline’s namespace at import time. You patch the name where it is used, not where it is defined.
What these tests do not check is whether the real model gives good answers. That needs the live stack and an evaluation set, which is the subject of the next lecture.
17.11 Local or Hosted?
With the whole system in view, we can compare honestly:
Local (Ollama + Infinity) | Hosted API | |
Data leaves your network | no | yes |
Cost per request | about zero after hardware | per token, forever |
Answer quality | good with RAG on narrow tasks; weak on hard reasoning | state of the art |
Speed on a laptop | 10–30 tokens/s on CPU | fast, independent of your hardware |
Concurrent users | a handful per machine | scales on demand |
Model can change under you | no (pinned file) | yes (versions retired) |
Operational burden | yours: memory, updates, GPUs | the provider’s |
A useful pattern in practice is to use both: embed locally (cheap,
high volume, private), develop and test against a local generator, and
keep a configuration switch to a hosted model for the cases where
quality matters most. MedKB’s design—
17.12 The MedKB Project
The complete project is given as the lecture exercise 12, in the repository at https://gitlab.cs.umd.edu/cmsc389a/fall2026/student-repos/ex12
project/ |
├── docker-compose.yml 6 services: mysql, chromadb, ollama, infinity, backend, frontend |
├── .env.example copy to .env |
├── backend/ |
│ ├── app/ |
│ │ ├── main.py FastAPI app; opens the ChromaDB collection at startup |
│ │ ├── config.py service URLs from the environment |
│ │ ├── rag/ |
│ │ │ ├── data/ qa_pairs.json, health_topic.html, fact_sheets.json |
│ │ │ ├── tokenizer_utils.py Qwen2.5 token counting + sliding window |
│ │ │ ├── chunking.py type-aware chunkers |
│ │ │ ├── prompt_builder.py XML-tagged prompt |
│ │ │ ├── infinity_client.py embed_texts() ← ACTIVITY PART 1 |
│ │ │ ├── ollama_client.py generate() ← ACTIVITY PART 2 |
│ │ │ ├── vector_store.py ChromaDB wrapper |
│ │ │ ├── ingest.py chunk + embed + store, once |
│ │ │ └── pipeline.py answer_question() |
│ │ ├── routers/ask.py POST /api/ask |
│ │ └── routers/search.py GET /api/search (old keyword search) |
│ └── tests/ 13 pytest tests, no Docker or model needed |
└── frontend/ "Ask AI (RAG)" tab + keyword-search tab |
The activity. Implement the two client functions. Each docstring specifies the exact request and response shapes, and the tests check them.
embed_texts(texts, base_url): POST to base_url + "/embeddings" with the input texts and model name, sort data by index, and return the vectors in input order. Raise on a non-2xx status.
generate(prompt, base_url, model): POST to base_url + "/api/generate" with "stream": false and return the response field. Raise on a non-2xx status.
Run the tests (no Docker needed):
cd notes/Orchestrating_Local_RAG_with_Ollama/project/backend
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pytest tests/ -vBefore you implement anything, the seven client tests fail with NotImplementedError, while the vector-store and pipeline tests already pass because they fake the clients. When you are done, all 13 pass.
Run the full stack:
cp .env.example .env
docker compose up --build -d mysql chromadb ollama infinity # dependencies first
docker compose exec ollama ollama pull qwen2.5:1.5b # first run only, ~1 GB
docker compose up --build # backend + frontendOpen http://localhost:5173. Ask Q1 (“What is type 2 diabetes?”) and expect a grounded answer with its sources listed. Ask Q2 (“What’s the capital of France?”) and expect the no-match answer. Then switch to the keyword-search tab and search for high blood sugar to see the gap RAG closes.
You are done when: all 13 tests pass; Q1 returns a grounded answer with sources on the live stack; Q2 returns “no match” instead of an invented answer; and your commit is on your fork’s main branch.
17.13 Practice Exercises
17.13.1 Basic
Using the formula in this lecture, estimate the memory needed for the weights of an 8B-parameter model at FP16, Q8_0 (about 8.5 bits), and Q4_K_M (about 4.8 bits). Which would fit on a laptop with 16 GB of RAM while leaving room for the operating system and a browser?
An Ollama response reports eval_count: 180 and eval_duration: 9000000000. Compute the generation speed. If the user is shown nothing until generation finishes, how long do they wait?
Compute the cosine distance between [3, 4] and [4, 3], and between [1, 0] and [-1, 0]. Which pair would pass MedKB’s 0.6 threshold?
Explain in your own words why the backend reaches Ollama at http://ollama:11434, while curl on your laptop uses http://localhost:11434.
Why does a vector search for “What’s the capital of France?” return three results from a medical corpus? What line of MedKB’s code keeps those results from reaching the model?
17.13.2 Intermediate
Write token_report(response: dict) -> str that takes a non-streaming Ollama response and returns a one-line summary of load time, prompt tokens per second, and generation tokens per second. Add a pytest test using the sample response from these notes.
Collect 10 in-scope and 10 out-of-scope questions for MedKB. For each, record the best cosine distance returned by ChromaDB. Plot or tabulate the two groups and choose a threshold. How many questions would your threshold misclassify? How does it compare to 0.6?
Add a num_ctx guard: before calling generate, count the prompt’s tokens with tokenizer_utils.count_tokens and, if the prompt would exceed a configured limit, drop the lowest-ranked chunks until it fits. Write a test that proves the guardrail sentence is never dropped.
Swap the generator for qwen2.5:0.5b and then qwen2.5:3b (change only configuration, not code). Ask the same five in-scope questions of each and compare answer quality and tokens per second. Report what you observe about the size/quality/speed trade-off.
Change the Compose file so Ollama and Infinity are reachable only from inside the Compose network, not from your host. Verify that the app still works and that curl http://localhost:11434/api/tags from your laptop now fails.
17.13.3 Advanced
Make ingestion detect staleness: store the embedding model name and a hash of the corpus files in the collection’s metadata, and re-ingest automatically when either changes. What has to happen to the old vectors, and why can you not simply add the new ones alongside them?
Stream the answer to the browser: add a POST /api/ask/stream endpoint that returns server-sent events, sending the sources first and then each token as Ollama produces it. Use httpx.AsyncClient so a slow generation does not block other requests, and explain why the current synchronous ask endpoint would serialize two users’ questions.
Design and run an experiment comparing two local embedding models served by Infinity (for example, all-MiniLM-L6-v2 and a larger BGE model). For a fixed set of 20 questions with known correct source chunks, measure how often the correct chunk appears in the top 3 for each model. Report the result along with each model’s embedding time for the whole corpus.
17.14 Summary
17.14.1 Key takeaways
A local LLM is a weights file plus an inference engine on hardware you control. It buys privacy, near-zero marginal cost, offline use, and a pinned model; it costs quality, speed, and operational work.
Memory needed is about parameters × bits per weight / 8, plus the KV cache. Quantization to about 4 bits is what puts useful models on laptops.
Ollama serves local models over a REST API on port 11434. /api/generate takes a prompt, /api/chat takes messages, and every response reports token counts and nanosecond timings you can use to measure speed.
The context window Ollama allocates (num_ctx) can be much smaller than what the model supports, and overflow is silent truncation, not an error.
A local RAG stack splits into services with one job each: Ollama generates, Infinity embeds, ChromaDB stores and searches.
Embedding responses must be re-ordered by index; documents and queries must use the same embedding model; changing that model means re-embedding everything.
Vector search always returns neighbors. A distance threshold, checked in code before the model is called, is what turns “nearest” into “relevant” and gives an honest “no match.”
Code that depends on model servers can be tested quickly by mocking the HTTP boundary and using an in-process vector store.
17.14.2 Terminology
Term | Meaning |
local LLM | a model whose weights and inference run on hardware you control |
open-weight model | a model whose trained weights are published for download |
inference engine | software that runs a model’s forward pass efficiently (e.g. llama.cpp) |
quantization | storing weights in fewer bits to save memory and time |
GGUF | the single-file model format used by llama.cpp and Ollama |
KV cache | stored attention keys/values for the context; grows with context length |
prefill / decode | processing the prompt (fast, parallel) / generating tokens (slow, sequential) |
num_ctx | Ollama’s context window size for a request |
keep-alive | how long Ollama keeps a model loaded after its last request |
Modelfile | a recipe packaging a base model with default parameters and system prompt |
embeddings server | a service that turns text into vectors over an API (e.g. Infinity) |
pooling | combining per-token vectors into one vector per text |
cosine distance | 1 − cosine similarity; 0 is identical, larger is less similar |
distance threshold | the cut-off above which retrieved context is treated as no match |
idempotent ingestion | ingestion that does nothing if it has already run |
17.14.3 Common mistakes
Using localhost inside a container to reach another container. Use the Compose service name.
Forgetting the named volume for /root/.ollama and re-downloading the model on every rebuild.
Forgetting to pull the model; Ollama’s health check passes anyway.
Keeping httpx’s default 5-second timeout for a model that needs 30 seconds to answer.
Trusting the order of an embeddings response instead of sorting by index.
Embedding the corpus with one model and the queries with another, or changing the embedding model without re-ingesting.
Writing chunks longer than the embedding model’s input limit, which silently embeds only their beginning.
Building prompts longer than num_ctx and losing the instructions at the start to silent truncation.
Confusing distance with similarity, so that the threshold check points the wrong way.
Leaving the model port published on all interfaces with no authentication.
Benchmarking the first request, which includes model load time.
17.14.4 Looking ahead
MedKB now answers questions, but we have no measurement of how well. Is 0.6 the right threshold? Does the answer actually follow from the retrieved chunks, or does the model add facts of its own? Did retrieval find the right chunks at all? The next lecture introduces RAG evaluation (faithfulness, answer relevance, context precision and recall, as in the Ragas metrics) and then moves the service toward deployment in containers on a cloud VM. Later, security lectures return to the XML tags in the prompt: retrieved documents are untrusted input, and prompt injection through a corpus is a real attack on exactly the system built here.