18 AI APIs: Calling LLMs in Production
18.1 Learning Objectives
By the end of this lecture you should be able to:
Create an API key and store it safely in an environment variable or a git-ignored .env file, load it with python-dotenv, and explain why a key must never reach a browser, a mobile app, or a git repository.
Trace a request through its five-step lifecycle (client → your server → provider → model → back) and describe the anatomy of an LLM API call—
endpoint, headers, JSON body, response, and the usage object— making one with nothing but curl. Build a multi-turn conversation over a stateless API by maintaining and resending the message history, and shape behavior with a system prompt.
Explain what temperature does to token sampling, choose an appropriate value for a task, and explain why current Claude models no longer expose it.
Stream responses over server-sent events (SSE), and explain why streaming improves perceived latency and prevents timeouts.
Get clean machine-readable output using stop sequences and schema-validated structured outputs.
Classify API errors as retryable or not, implement exponential backoff with jitter, and compute the dollar cost of a request from its token usage.
Use LiteLLM to keep several providers’ keys in one .env and switch between OpenAI, Gemini, Anthropic, the native Claude SDK, and a local Ollama model with a single flag.
18.2 Motivation and Intuition
18.2.1 From “it works on my laptop” to production
In the Ollama lecture we happily called client.messages.create(...) and moved on. That one line hides an entire engineering discipline. The moment your class project becomes a real service, five questions arrive at once:
Auth—
where does the API key live, and who can steal it? Behavior—
how do I make the model act like a tutor, a grader, or a JSON generator instead of a chatty generalist? Latency—
why does the user stare at a spinner for 20 seconds? Reliability—
what happens when the provider returns 429 Too Many Requests at the worst possible moment? Money and lock-in—
who is paying for these tokens, and what happens if we want to switch to a cheaper or local model next month?
Every AI product team answers these same questions. Most of them
are not AI problems; they are distributed-systems and
operations problems wearing an AI costume—
18.2.2 The mental model: a metered, throttled, remote function call
Think of an LLM API as calling a function that happens to live in someone else’s datacenter, where:
every call must present a credential (auth),
the function has no memory between calls (you resend the conversation every time),
every call is metered in tokens and billed in dollars (cost),
the provider enforces a speed limit per customer (rate limits),
and the answer arrives over a network that can be slow or flaky (streaming, retries).
None of that is visible in a happy-path demo. All of it is visible in production.
18.2.3 Connection to earlier material
REST lecture: an LLM API is a REST-ish API—
POST a JSON body, get a JSON response, act on the status code. Everything you know about HTTP applies. LLM lecture: tokens, embeddings, and next-token sampling are exactly what happens inside the provider when your request lands. Temperature is that sampling step, exposed as a knob.
Ollama lecture: a local model server speaks HTTP too, so the same client code can talk to it—
which is what LiteLLM exploits.
18.3 Running Examples: TerpGrader and TerpTutor
Two services, both for a large UMD course, follow us through the lecture.
TerpGrader grades short-answer quiz submissions. Once per assignment it must grade 500 submissions, each roughly a 900-token prompt (rubric + student answer) producing a ~150-token structured verdict. It stresses the back-end topics:
It runs unattended, so auth must come from the environment, not a human.
It needs parseable JSON, not friendly prose.
500 back-to-back requests will hit rate limits.
The department pays real money: we must predict and cap the spend per run.
TerpTutor is the interactive side: a math-help chatbot that students talk to. It stresses the front-end topics:
It must remember the conversation (multi-turn).
It must behave like a tutor—
hints, not answers (system prompts). It must feel responsive (streaming).
The department wants to try it on several providers, including a free local model for development (LiteLLM).
The concepts build in order: we first get a key, then make one TerpTutor request, give it memory, give it a personality, stream it, and finally make TerpGrader’s batch job reliable, cheap, and provider-independent.
18.4 Core Concept 1: Getting and Storing API Keys
18.4.1 Definition and intuition
Authentication binds each request to an account for billing, rate limiting, and access control. LLM providers commonly support: static API keys (a secret string sent in a header), OAuth bearer tokens (short-lived, obtained by a login flow), and cloud IAM signing when the model is served through a cloud platform such as AWS Bedrock or Google Vertex AI.
In plain English: the API key is a credit card number that can also talk to an AI. Anyone holding it can spend your money, up to your rate limit, until you notice.
18.4.2 Getting an Anthropic API key
Go to https://console.anthropic.com/ and log in.
Click Get API Keys (top right of the dashboard).
Click Create Key.
Choose the workspace (Default is fine) and give the key a name you will recognize later, such as CMSC389A. The name is for you; it does not appear in requests.
Copy the key now. It is shown exactly once. If you close the window before copying it, delete that key and create a new one.
The other providers in this lecture work the same way: OpenAI keys come from https://platform.openai.com/api-keys, Gemini keys from https://aistudio.google.com/apikey. A local Ollama server needs no key at all, only a URL.
18.4.3 The lifecycle of a key, done right
create (provider console, least privilege / per-workspace) |
│ |
▼ |
store ──▶ dev: environment variable, or a .env file (git-ignored) |
──▶ prod: secret manager (AWS Secrets Manager, GCP Secret |
Manager, Vault) injected into the process env |
│ |
▼ |
use ──▶ read from the environment at startup; never hardcode, never log |
│ |
▼ |
rotate ──▶ periodically, and IMMEDIATELY on any suspected leak |
18.4.4 Option A: an environment variable
An environment variable is a named string that the operating system hands to every process started from a shell. Your code asks for it by name, so the secret never appears in your source.
macOS / Linux, for the current terminal only:
export ANTHROPIC_API_KEY="sk-ant-api03-...your key..."
export OPENAI_API_KEY="sk-proj-..."
export GEMINI_API_KEY="AIza..."
echo ${ANTHROPIC_API_KEY:0:10} # print only a prefix to check it is setTo make it permanent, append the export line to your shell’s startup file (~/.zshrc on macOS, ~/.bashrc on most Linux), then open a new terminal or run source ~/.zshrc.
Windows PowerShell:
$env:ANTHROPIC_API_KEY = "sk-ant-api03-..." # current window only
setx ANTHROPIC_API_KEY "sk-ant-api03-..." # permanent; open a new window afterwardsReading it in Python is one line—
import os
api_key = os.environ["ANTHROPIC_API_KEY"] # KeyError (a loud crash) if missingYou rarely need even that: Anthropic(), OpenAI(), and LiteLLM all look up their standard variable names (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY) on their own.
18.4.5 Option B: a .env file
Exporting variables by hand gets tedious when a project uses several keys. A .env file is a plain-text file of NAME=value lines in the project folder. The python-dotenv package reads it and copies each line into os.environ, so the rest of your code cannot tell the difference from a real environment variable.
Create .env next to your notebook or script:
# .env -- NEVER commit this file
ANTHROPIC_API_KEY="sk-ant-api03-...your key..."Then, as in every course notebook:
%pip install anthropic python-dotenv # in Jupyter; use plain `pip install` in a terminal
from dotenv import load_dotenv
load_dotenv() # finds .env and fills os.environ
from anthropic import Anthropic
client = Anthropic() # picks up ANTHROPIC_API_KEY automaticallyThree files, three jobs. The repository hygiene that goes with a .env file is always the same:
File | Committed? | Contains |
.env | never | the real secret values |
.env.example | yes | the same variable names with placeholder values, so teammates know what to fill in |
.gitignore | yes | a line reading .env, so git refuses to track it |
# .gitignore
.env
.env.*
!.env.exampleCommit the shape of the configuration, never the values. Add the .gitignore line before you create .env: once git tracks a file, ignoring it later does not untrack it.
18.4.6 Several keys in one .env
The demo project keeps every provider’s key in a single .env. This is the file the LiteLLM section uses to switch providers:
# .env (copied from .env.example, then filled in)
LLM_PROVIDER=claude-sdk # anthropic | claude-sdk | openai | gemini | ollama
ANTHROPIC_API_KEY=sk-ant-api03-...
ANTHROPIC_MODEL=claude-opus-5-5
OPENAI_API_KEY=sk-proj-...
OPENAI_MODEL=gpt-4.1-mini
GEMINI_API_KEY=AIza...
GEMINI_MODEL=gemini-2.5-flash
OLLAMA_API_BASE=http://localhost:11434 # local server: a URL, not a secret
OLLAMA_MODEL=llama3.2Notice that the model names also live in configuration. Changing from Opus to Haiku, or from Gemini Flash to Pro, should be a config change, not a code change.
18.4.7 Who wins: the shell or the .env file?
By default load_dotenv() does not override a variable that is already set. That is the right rule: in production a secret manager injects the real key into the process environment, and a stale .env file that was accidentally copied onto the server must never replace it. The demo project’s loader makes that choice explicit (apidemo/config.py):
from dotenv import find_dotenv, load_dotenv
def load_env() -> str | None:
"""Load the nearest .env (searching upward from the cwd). Returns its path."""
path = find_dotenv(usecwd=True)
if path:
load_dotenv(path, override=False) # never clobber real env vars
return path or None
def require(name: str) -> str:
"""Return the value of an env var, or fail loudly with a helpful message."""
value = os.environ.get(name, "").strip()
if not value or "REPLACE-ME" in value:
raise MissingKeyError(
f"{name} is not set. Add it to your .env file (see .env.example) "
f"or run: export {name}=...")
return value
def mask(secret: str | None) -> str:
"""Show just enough of a key to identify it -- safe to print or log."""
if not secret:
return "<unset>"
if len(secret) <= 12:
return "*" * len(secret)
return f"{secret[:7]}...{secret[-4:]}"Worked example (Demo 0). Suppose your shell exported OPENAI_API_KEY and your .env contains ANTHROPIC_API_KEY and a stale OPENAI_API_KEY. Running python demos/00_check_env.py prints, with every key masked:
.env file: /Users/you/claude-api-demos/.env |
|
ANTHROPIC_API_KEY sk-ant-...x9Qw from .env |
OPENAI_API_KEY sk-proj...1234 from shell <- shell wins |
GEMINI_API_KEY <unset> from - |
OLLAMA_API_BASE http://localhost:11434 |
LLM_PROVIDER claude-sdk |
Step by step: the script records which keys were already in the environment, calls load_env(), and compares. ANTHROPIC_API_KEY was missing, so .env filled it in. OPENAI_API_KEY was already set, so the file’s value was ignored. GEMINI_API_KEY is in neither place; any Gemini call will fail with a clear MissingKeyError before touching the network. Run this demo first whenever “my key doesn’t work.”
18.4.8 The cardinal rule: keys stay server-side
A key must never be embedded in anything you ship to
users—
WRONG RIGHT |
───── ───── |
browser ──(key!)──▶ AI API browser ──▶ YOUR backend ──(key)──▶ AI API |
(auth, rate-limit, |
log, budget here) |
The proxy pattern is also where your product’s controls live:
per-user authentication, per-user rate limits, logging, abuse
filtering, and budget caps all happen in your backend—
18.4.9 Common misconceptions
“It’s just a class project, nobody will find my key.” GitHub is scraped by bots continuously; a committed key is typically exploited within minutes. Providers auto-revoke keys they detect in public repos—
the best-case outcome is that your job breaks. “I’ll delete the commit.” Git history is forever (and already scraped). Leak = rotate. There is no other response.
“The repo is private, so a key in config.py is fine.” Private repos get forked, shared with TAs, made public at the end of the semester, and copied into Docker images. The key also cannot be rotated without a code change.
“An environment variable is secure.” It is appropriate, not magically secure—
anyone with shell access to the machine can read it. It solves the “don’t put secrets in code or git” problem; production adds secret managers, scoping, and rotation on top. “print(api_key) is fine for debugging.” Logs get shipped to dashboards and pasted into Slack. Print mask(key) instead.
18.4.10 Worked example — TerpGrader’s deployment
TerpGrader runs as a cron job in the department’s cluster. The key
is stored in the university’s Vault instance; the job’s launcher
fetches it and injects ANTHROPIC_API_KEY into the process
environment; the code never sees a file or a literal. When a TA
accidentally pasted the key into a Slack channel, the response was:
rotate in the console (30 seconds), update Vault (one command),
done—
18.5 Core Concept 2: The Request Lifecycle and Anatomy of a Call
18.5.1 The five-step request flow
Every interaction follows the same five steps, from the moment a student clicks “send” in TerpTutor to the moment the reply appears:
┌────────┐ 1. request ┌─────────────┐ 2. request + API key ┌──────────────┐ |
│ client │ ──────────▶ │ your server │ ────────────────────▶ │ provider API │ |
│ (web / │ │ (FastAPI) │ │ │ |
│ mobile)│ ◀────────── │ │ ◀──────────────────── │ 3. model │ |
└────────┘ 5. response └─────────────┘ 4. response + usage │ processing │ |
└──────────────┘ |
Steps 1 and 5 are your API; steps 2 and 4 are the provider’s. The key only ever travels on steps 2 and 4.
18.5.2 Inside step 3: what the model does
You met these stages in the LLM lecture; here is how they map onto a single API call:
Tokenization. The input text is split into tokens: whole words, word pieces, spaces, punctuation. Billing and limits are counted in these units.
Embedding. Each token becomes a long vector of numbers that encodes all of its possible meanings. “Quantum” starts out ambiguous: a unit of physics, quantum mechanics, something tiny, or quantum computing.
Contextualization. The transformer layers adjust each vector using the surrounding tokens, so “quantum” next to “computing” and “qubits” settles on the computing sense.
Generation. An output layer turns the final vector into a probability for every possible next token. The model samples one—
not always the most likely— appends it, and repeats the whole process for the next token.
After every generated token, the model checks three stopping conditions, and the response’s stop_reason tells you which one fired:
Condition | stop_reason | What it means for you |
the model produced its end-of-turn token | end_turn | normal success |
it hit your max_tokens ceiling | max_tokens | the answer is truncated---maybe mid-sentence or mid-JSON |
it generated one of your stop sequences | stop_sequence | intentional cut-off (see Core Concept 7) |
safety systems declined the request | refusal | no usable answer; handle it explicitly |
18.5.3 The request, with nothing hidden
Everything an SDK does ultimately becomes this curl:
curl https://api.anthropic.com/v1/messages \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2026-10-21" \
-H "content-type: application/json" \
-d '{
"model": "claude-opus-5-5",
"max_tokens": 1000,
"system": "You are TerpGrader. Reply with JSON only: {\"score\": 0-10, \"feedback\": \"...\"}",
"messages": [
{"role": "user", "content": "RUBRIC: name the 3 steps of the TCP handshake.\nANSWER: SYN, then SYN-ACK, then ACK."}
]
}'Note $ANTHROPIC_API_KEY: the shell substitutes the environment variable, so even a one-off curl never contains the literal key. Dissecting the parts:
Part | Purpose | Notes |
x-api-key | authentication | who to bill and rate-limit |
anthropic-version | API versioning | pins the wire format, so provider changes don't silently break you |
model | which model | exact ID strings; a typo is a 404 |
max_tokens | output cap | a hard safety ceiling, not a target length |
system | instructions | how to behave (Core Concept 4) |
messages | the transcript | stateless: the full history every time (Core Concept 3) |
And the response:
{
"id": "msg_01XFDUDYJgAACzvnptvVoYEL",
"type": "message",
"role": "assistant",
"model": "claude-opus-5-5",
"content": [
{"type": "thinking", "thinking": "", "signature": "EqQBCkgIBRAB..."},
{"type": "text", "text": "{\"score\": 10, \"feedback\": \"All three steps, in order.\"}"}
],
"stop_reason": "end_turn",
"usage": {"input_tokens": 78, "output_tokens": 61}
}Three fields deserve special attention:
content is a list of blocks, not a string. Current Claude models may think before answering, and that thinking arrives as a thinking block before the text block (its text is omitted by default, but it is billed in output_tokens). That is why these notes never write content[0].text.
stop_reason tells you why generation ended. For TerpGrader, max_tokens means truncated, unparseable JSON—
a real bug we must handle. usage is the billing ground truth. Not your estimate, not a tokenizer library—
this object is what you pay for.
18.5.4 Worked example — the first SDK request (Demo 1)
The Python SDK turns that HTTP call into a method call. client.messages.create() needs three arguments: model, max_tokens, and messages.
# demos/01_first_request.py
from dotenv import load_dotenv
from anthropic import Anthropic
load_dotenv() # copies .env into os.environ
client = Anthropic() # finds ANTHROPIC_API_KEY on its own
model = "claude-opus-5-5"
message = client.messages.create(
model=model,
max_tokens=4000, # a safety ceiling, not a target length
messages=[
{"role": "user", "content": "What is quantum computing? Answer in one sentence"}
],
)
# Just the text. Join the text blocks: content[0] may be a thinking block.
text = "".join(block.text for block in message.content if block.type == "text")
print("ANSWER:", text)
print("stop_reason:", message.stop_reason)
print("usage:", message.usage.input_tokens, "in /", message.usage.output_tokens, "out")Expected output (your wording will differ):
ANSWER: Quantum computing is a type of computation that uses quantum-mechanical |
phenomena like superposition and entanglement to process information in qubits, |
potentially solving certain problems far faster than classical computers. |
stop_reason: end_turn |
usage: 18 in / 64 out |
Reasoning through it:
load_dotenv() runs before Anthropic(), so the key is in the environment when the client looks for it. Swap the two lines and you get an authentication error.
Each message is a dictionary with a role ("user" for text a human wrote, "assistant" for text the model wrote) and content.
max_tokens=4000 is a ceiling. The model does not try to fill it; it writes what it thinks is appropriate and stops. Set it high enough to cover thinking plus the answer—
a too-small value truncates the reply. Some online examples read the answer with message.content[0].text. On current models content[0] can be a thinking block, so join the text blocks instead. The demo project wraps this in a helper:
# apidemo/chat.py
def text_of(message) -> str:
"""Join the text blocks of a response."""
if message.stop_reason == "refusal":
return "[the model declined this request]"
return "".join(b.text for b in message.content if b.type == "text")18.5.5 The three levels of abstraction
Demo 2 (demos/02_raw_http.py) sends the TerpGrader request with the httpx library and no SDK, prints the rate-limit headers, and prints the equivalent curl. Run it with –break-key and –break-model to see real 401 and 404 bodies.
response = httpx.post(
"https://api.anthropic.com/v1/messages",
headers={
"x-api-key": api_key, # authentication
"anthropic-version": "2026-10-21", # wire-format version
"content-type": "application/json",
},
json=body,
timeout=60,
)
print("HTTP", response.status_code)
print(json.dumps(response.json(), indent=2))Level | Example | You get | You give up |
raw HTTP | curl | total transparency; great for debugging | everything is manual |
HTTP library | httpx (Demo 2) | testability, connection reuse, timeouts | you write retries, types, streaming parsing |
official SDK | anthropic | typed responses, automatic retries on 429/5xx, auth resolution, streaming helpers, typed exceptions | some magic you should understand anyway |
multi-provider library | LiteLLM (Core Concept 10) | one call shape for many providers | provider-specific features, and one more layer to debug |
Rule for real projects: use the official SDK (or LiteLLM when switching providers is a requirement). We look at raw HTTP so the SDK is no longer magic.
18.5.6 Common misconceptions
Misconception | Reality |
"max_tokens is a target length." | It's a hard cap. The model stops early when it's done, and gets cut off (stop_reason: max_tokens) when it isn't. |
"The answer is content[0].text." | content is a list of blocks; on current models the first may be a thinking block. Join the text blocks. |
"I can count tokens with len(text)/4." | Fine for rough budgeting; wrong for billing. Use the usage object (after) or the count_tokens endpoint (before). |
"JSON in = JSON out." | The envelope is JSON; the generated text is whatever the model wrote. Core Concept 7 covers getting parseable output. |
18.6 Core Concept 3: Multi-Turn Conversations
18.6.1 Definition and intuition
The Messages API is stateless: the provider stores none of
your conversation. Each request is complete and independent, and the
model sees exactly the messages list in that request—
Think of it as talking to someone with no short-term memory who reads a transcript before every reply. If you hand them only your latest sentence, they have no idea what “it” refers to.
18.6.2 The problem, demonstrated (Demo 3)
# demos/03_multi_turn.py, first half
print("Q1:", chat([{"role": "user", "content": "Define quantum computing in one sentence"}]))
print("Q2:", chat([{"role": "user", "content": "Write another sentence"}]))Q1: Quantum computing is a paradigm of computation that uses qubits ... |
Q2: The morning fog lifted slowly over the quiet harbor, revealing ... |
The second request arrived with no context, so “another sentence” became a sentence about anything at all.
18.6.3 The fix: three helper functions
From apidemo/chat.py:
def add_user_message(messages, text):
user_message = {"role": "user", "content": text}
messages.append(user_message)
def add_assistant_message(messages, text):
assistant_message = {"role": "assistant", "content": text}
messages.append(assistant_message)
def chat(messages):
message = client.messages.create(
model=model,
max_tokens=4000,
messages=messages,
)
return text_of(message) # the notebook uses message.content[0].textAnd the conversation that actually works:
# Start with an empty message list
messages = []
# Add the initial user question
add_user_message(messages, "Define quantum computing in one sentence")
# Get Claude's response, and add it to the conversation history
answer = chat(messages)
add_assistant_message(messages, answer)
# Add a follow-up question; send the full history
add_user_message(messages, "Write another sentence")
final_answer = chat(messages)
add_assistant_message(messages, final_answer)Trace of the messages list. What the API receives on each call:
call 1: [ user: "Define quantum computing in one sentence" ] |
|
call 2: [ user: "Define quantum computing in one sentence", |
assistant: "Quantum computing is ... qubits ...", |
user: "Write another sentence" ] <- context included |
Now “another sentence” clearly means another sentence about quantum computing. The rules: the first message is a user message, and roles normally alternate user, assistant, user, ...
18.6.4 Worked example — a terminal chatbot (Demo 4)
The notebook’s chat exercise wraps the helpers in a loop. The demo version adds two commands and prints token usage each turn:
# demos/04_chat_loop.py (core loop)
messages = []
while True:
user_input = input("\nyou> ").strip()
if user_input == "/quit":
break
if user_input == "/reset":
messages.clear() # Claude has "forgotten" everything
continue
add_user_message(messages, user_input)
response = client.messages.create(model=MODEL, max_tokens=4000, messages=messages)
reply = text_of(response)
add_assistant_message(messages, reply)
print(f"claude> {reply}")
print(f" (turn {len(messages) // 2}: {response.usage.input_tokens} input tokens)")A sample session:
you> My name is Testudo and I am taking CMSC389A. |
claude> Nice to meet you, Testudo! ... |
(turn 1: 21 input tokens) |
you> What is my name and which class am I in? |
claude> Your name is Testudo, and you're taking CMSC389A. |
(turn 2: 96 input tokens) |
you> /reset |
you> What is my name? |
claude> I don't know your name -- you haven't told me yet. |
(turn 1: 12 input tokens) |
Watch the input-token count. It grows every turn because the
whole history is resent—
18.6.5 Common misconceptions
“The API remembers my session.” There is no session. Clear your list and the model has forgotten everything, as /reset shows.
“Storing history is the provider’s job.” It is yours—
usually in a database keyed by conversation ID (the database lecture). Your server reloads it on each request. “I can edit the history freely.” You can, and it is a powerful tool (summarizing old turns, removing a bad answer)—
but the model then believes whatever the edited transcript says. Never let users write assistant messages directly.
18.7 Core Concept 4: System Prompts
18.7.1 Definition and intuition
A system prompt is a string passed in the separate system field of a request, outside the messages list. It tells the model what role to play and how to respond for the whole conversation. The model responds the way someone in that role would.
In plain English: the messages are the conversation; the system prompt is the job description handed to the employee before the conversation starts.
18.7.2 Worked example — turning Claude into TerpTutor (notebook 002, Demo 5)
When a student asks “How do I solve 2x + 3 = 11?”, a good tutor
should give hints, walk through the steps patiently, and show
similar worked problems—
First, make the chat helper accept an optional system prompt:
def chat(messages, system=None):
params = {
"model": model,
"max_tokens": 4000,
"messages": messages,
}
if system: # the API rejects system=None,
params["system"] = system # so only include it when it is set
message = client.messages.create(**params)
return text_of(message)The params dictionary plus **params pattern matters: the API does not accept system=None, so optional fields must be left out entirely rather than set to None. The same pattern handles stop_sequences later.
Then compare with and without a system prompt:
system_prompt = """
You are a patient math tutor. Do not directly answer the user's question.
Instead, ask follow-up questions to help the user solve the problem.
"""
messages = []
add_user_message(messages, "I need to solve the equation 2x + 3 = 11. Can you help me?")
print(chat(messages)) # no system prompt
print(chat(messages, system=system_prompt)) # TerpTutorExpected output (abridged):
--- no system prompt --- |
Sure! Subtract 3 from both sides: 2x = 8. Then divide both sides by 2: x = 4. |
|
--- with the tutor system prompt --- |
Of course! Let's work through it together. Right now x has two things |
attached to it: it's multiplied by 2, and 3 is added. Which of those |
do you think we should undo first, and what operation would undo it? |
Same model, same question, completely different behavior.
18.7.3 Worked example — a code-only assistant (system prompt exercise)
The notebook exercise uses a system prompt to strip away explanation:
system_prompt = """
You are a code assistant. Return the code content only. Don't include any other text.
"""
messages = []
add_user_message(messages, "Write a Python function that checks a string for duplicate characters.")
print(chat(messages, system=system_prompt))Expected output:
def has_duplicate_characters(s):
"""Check if a string contains duplicate characters."""
return len(s) != len(set(s))Often the model still wraps the code in a markdown fence. A system prompt strongly encourages a format; it does not guarantee one. Core Concept 7 shows how to guarantee it.
18.7.4 Common misconceptions and practical implications
“A system prompt is a security boundary.” It is a strong instruction, not access control. Users can try to talk the model out of it (prompt injection, the security lecture). Enforce real rules in your code.
“Put the system prompt in the first user message.” Use the system field: the model treats it as the operator’s instructions, and it stays fixed while the conversation grows.
Practical: the system prompt is resent (and billed) with every request, like the history. A long, stable system prompt is the classic candidate for prompt caching (Core Concept 9).
18.8 Core Concept 5: Temperature
18.8.1 Definition and intuition
Recall the generation step: for each position the model computes a probability for every possible next token, then samples one. Temperature is a number (typically 0 to 1) that reshapes that probability distribution before sampling:
Low temperature (near 0) sharpens the distribution, so the most likely token almost always wins. Output is predictable and repeatable.
High temperature (near 1) flattens it, so less likely tokens get picked more often. Output is more varied and creative.
Worked example. After “What do you think” the model might assign: about 30%, would 20%, of 10%, and the rest spread over thousands of other tokens.
about would of (others) |
temperature 0.0 100% 0% 0% 0% always "about" |
temperature 0.5 ~55% ~25% ~7% ~13% usually "about" |
temperature 1.0 30% 20% 10% 40% the model's raw odds |
Repeat that choice for every token in a paragraph and the
differences compound: at 0.0 you get nearly the same sentence each
run; at 1.0 you get a different story every time. Temperature
changes probabilities, not guarantees—
18.8.2 Choosing a temperature
Range | Good for | Running example |
0.0 -- 0.3 (low) | factual answers, coding, data extraction, moderation | TerpGrader: the same answer should get the same grade |
0.4 -- 0.7 (medium) | summarization, educational content, problem-solving | TerpTutor explanations |
0.8 -- 1.0 (high) | brainstorming, creative writing, marketing, jokes | generating practice-problem story contexts |
18.8.3 The notebook code (notebook 003)
The notebook adds a temperature parameter to the chat helper and compares three settings:
def chat(messages, system=None, temperature=1.0):
params = {
"model": model,
"max_tokens": 1000,
"messages": messages,
"temperature": temperature
}
if system:
params["system"] = system
message = client.messages.create(**params)
return message.content[0].text
messages = []
add_user_message(messages, "Generate a one sentence story about a cat.")
print("temperature=0.0:", chat(messages, temperature=0.0))
print("temperature=0.5:", chat(messages, temperature=0.5))
print("temperature=1.0:", chat(messages, temperature=1.0))18.8.4 Important: current Claude models no longer take a temperature
The notebook code above sends temperature, but current Claude
APIs no longer accept it—
Current Claude models (Opus 5.5, Sonnet 5.5, Haiku 5.5, and others) reject non-default sampling settings, and API version 2026-10-21 removes temperature, top_p, and top_k from the Messages API on every model—
a request that sends them gets a 400. The current Python SDK sends that version by default, and its messages.create() no longer has a temperature parameter at all; the notebook cell fails with a TypeError before any request is sent.
How do you control variety on Claude now? Through the prompt (“give five different ideas”, “answer in exactly this format”), through structured outputs when you need consistency (Core Concept 7), and by generating several candidates and choosing when you need variety.
18.8.5 Worked example — the experiment on a provider that still has the knob (Demo 6)
The concept is still essential: OpenAI, Gemini, and every local Ollama model expose temperature. Demo 6 runs the notebook’s experiment through the LiteLLM provider switch (Core Concept 10), three samples per setting:
python demos/06_temperature.py --provider ollama # free and local
python demos/06_temperature.py --provider openaiprompt = "Generate a one sentence story about a cat."
for temperature in (0.0, 0.5, 1.0):
print(f"\n=== temperature={temperature} ===")
for i in range(args.samples):
reply = ask(prompt, provider=args.provider, temperature=temperature, max_tokens=100)
print(f" {i + 1}. {reply.text.strip()}")Typical output with a local model:
=== temperature=0.0 === |
1. The curious cat chased a sunbeam across the floor until it fell asleep in its warmth. |
2. The curious cat chased a sunbeam across the floor until it fell asleep in its warmth. |
3. The curious cat chased a sunbeam across the floor until it fell asleep in its warmth. |
|
=== temperature=1.0 === |
1. Whiskers stole the last sardine, then blamed the dog with a perfectly innocent blink. |
2. Every night at 3 a.m., Mochi sang opera to the refrigerator, hoping it would sing back. |
3. The old tabby guarded the library's rare-book room better than any alarm ever had. |
At 0.0 the three runs are identical; at 1.0 every run is new. If you pass –temperature to a Claude provider in the switch, it prints a note and drops the parameter instead of failing.
18.8.6 Common misconceptions
“Temperature 0 means deterministic.” Nearly, not perfectly: ties and hardware-level floating-point differences can still change output. Never rely on bit-for-bit repeatability.
“Higher temperature means smarter or more creative ideas.” It means less likely tokens. Past a point that is just incoherence.
“Temperature is a quality knob.” It is a variety knob. Use the prompt, the model choice, and evaluation for quality.
18.9 Core Concept 6: Streaming Responses
18.9.1 Definition and intuition
Streaming returns the response incrementally over a single long-lived HTTP response using server-sent events (SSE): a text/event-stream body where each event is an event: <type> line plus a data: <json> line, delivering pieces of the answer as the model generates them.
A TerpTutor reply can take 10–30 seconds to generate. Without streaming, your server waits for the complete response, and the student stares at a spinner with no sign that anything is happening. Non-streaming is a letter: nothing, nothing, nothing, then the whole answer. Streaming is a phone call: words as they are produced. Two latencies matter:
Time to first token (TTFT)—
how long until something appears (about a second when streamed). Total time—
until the answer is complete (identical either way).
Streaming doesn’t make the model faster; it makes the
experience faster—
18.9.2 The events
All of these arrive as part of one request:
Event | Meaning |
message_start | a new message has begun; carries input-token usage |
content_block_start | a new block (thinking, text, tool use) begins |
content_block_delta | a chunk of the block---the text you display |
content_block_stop | the current block is complete |
message_delta | end-of-message info: stop_reason and output-token usage |
message_stop | nothing more for this message |
On the wire:
event: message_start |
data: {"type":"message_start","message":{"usage":{"input_tokens":21,...}}} |
|
event: content_block_delta |
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"Rex spotted"}} |
|
event: content_block_delta |
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" the cat"}} |
|
event: message_delta |
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":212}} |
|
event: message_stop |
data: {"type":"message_stop"} |
Note where the bookkeeping lives: input tokens arrive in message_start, output tokens and stop_reason in the final message_delta. A cost meter must read both ends of the stream.
18.9.3 Worked example — the raw event stream (notebook 004, Demo 7)
Add stream=True to messages.create and iterate:
messages = []
add_user_message(messages, "Write 1 short story about a dog chasing a cat.")
stream = client.messages.create(
model=model,
max_tokens=4000,
messages=messages,
stream=True
)
for event in stream:
print(event)Run python demos/07_streaming.py events and you will see one line per event: a message_start, block starts and stops, dozens of content_block_delta events each holding a few words, then message_delta and message_stop. Only the text deltas are what the student should see.
18.9.4 Worked example — just the text, plus the final message
Rather than filtering events yourself, use the SDK’s streaming helper, which yields only text and rebuilds the complete message at the end:
with client.messages.stream(
model=model,
max_tokens=4000,
messages=messages
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True) # send each chunk to the client
final_message = stream.get_final_message() # for the databaseflush=True matters: without it the terminal buffers output and the “streaming” appears all at once.
get_final_message() returns a normal message object, with content, stop_reason, and usage, exactly as if you had not streamed. TerpTutor streams chunks to the browser and stores the final message in its conversation history.
Seeing the difference. python demos/07_streaming.py race times both approaches on the same prompt:
non-streaming: first text after 8.4s (all 1093 chars at once) |
streaming: first text after 1.1s, finished after 8.2s |
Same total time, about eight times faster to first output.
18.9.5 Common misconceptions
“Streaming is cheaper.” Same tokens, same price. It changes when you see them, not how many there are.
“Streaming is just for looks.” Also for correctness: long generations require it to avoid timeouts, which is why the SDKs insist on streaming for very large max_tokens.
“I can reassemble the message from the deltas I printed.” Use get_final_message(). Dropped events and non-text blocks make naive reassembly buggy.
“Errors arrive as events.” A 429 or 401 arrives before any events, as a normal error response; the SDK raises it when you open the stream.
18.10 Core Concept 7: Controlling Output Format
18.10.1 Definition and intuition
When a program—
```json |
{ |
"source": ["aws.ec2"], |
"detail-type": ["EC2 Instance State-change Notification"], |
"detail": { "state": ["running"] } |
} |
``` |
This rule captures EC2 instance state changes when instances start running. |
The JSON is correct, but json.loads fails on the fence and the explanation, and a user of an “EventBridge rule generator” web app cannot just copy the whole response. TerpGrader has the same problem with its verdicts. Beyond asking nicely in the system prompt, there are two tools.
18.10.2 Tool 1: stop sequences
stop_sequences is a list of strings; generation ends the moment the model produces one of them, and stop_reason becomes stop_sequence. The stop string itself is not included in the output. Add it to the chat helper the same way as system:
def chat(messages, system=None, stop_sequences=None):
params = {"model": model, "max_tokens": 4000, "messages": messages}
if system:
params["system"] = system
if stop_sequences:
params["stop_sequences"] = stop_sequences
message = client.messages.create(**params)
return text_of(message)Worked example —
messages = []
add_user_message(messages,
"RUBRIC: name the 3 steps of the TCP handshake.\n"
"ANSWER: SYN, then SYN-ACK, then ACK.\n"
"Reply with 'SCORE: <0-10>' on the first line, then your feedback.")
print(repr(chat(messages, stop_sequences=["\n"])))'SCORE: 10' |
The feedback is never generated, so it is never billed, and int(text.split(":")[1]) parses the score.
The limit shows up as soon as you try the obvious fix for fenced JSON: stop_sequences=["“‘"]. The opening fence is the first thing the model writes, so generation stops immediately and you get an empty string. A stop sequence can cut off what comes after the data, not what comes before it. For that you need Tool 2.
18.10.3 Tool 2: structured outputs
The modern replacement constrains the response to a JSON schema and validates it for you. With the Python SDK you describe the shape as a Pydantic model and call messages.parse:
from pydantic import BaseModel
class EventPattern(BaseModel):
source: list[str]
detail_type: list[str]
states: list[str]
response = client.messages.parse(
model="claude-opus-5-5",
max_tokens=4000,
messages=[{"role": "user",
"content": "Generate a very short EventBridge pattern that matches EC2 "
"instances entering the running state."}],
output_format=EventPattern,
)
rule = response.parsed_output # a validated EventPattern object
print(rule.source, rule.states) # ['aws.ec2'] ['running']Without Pydantic, pass the schema directly with output_config={"format": {"type": "json_schema", "schema": ...}} on messages.create; the text block then contains JSON that matches the schema.
For TerpGrader this is the right tool: a Verdict model with
score: int and feedback: str means every one of the 500
replies parses—
18.10.4 Comparing the three techniques
Technique | Works on | Guarantee | Use for |
system prompt ("JSON only") | all models | none---usually works | quick experiments |
stop sequences | all models | cuts output at a marker | ending lists, removing trailing text |
structured outputs | current models | valid against your schema | anything a program parses |
18.11 Core Concept 8: Errors and Rate Limits
18.11.1 Definition and intuition
LLM APIs signal failures with standard HTTP status codes plus a JSON
error body. A rate limit is a provider-enforced cap per
organization, measured along several dimensions
simultaneously—
18.11.2 The error taxonomy — retryable or not?
The single most useful classification of any failure is “will trying again help?”
Code | Meaning | Retryable? | Correct reaction |
400 | malformed request | no | fix your code (bad body, unsupported parameter) |
401 | bad/missing key | no | fix your configuration (run Demo 0) |
403 | key lacks permission | no | fix your account/scoping |
404 | bad model ID / endpoint | no | fix the model string |
413 | request too large | no | shrink the input (trim history, chunk documents) |
429 | rate limited | YES | wait (retry-after), then retry |
500 | provider bug | YES | retry with backoff |
529 | provider overloaded | YES | retry with backoff, consider shedding load |
Retrying a 400 is a bug that burns time and money forever; failing permanently on a 429 is a bug that makes your service fragile for no reason. The taxonomy is the design.
Not every failure is an exception: stop_reason: "max_tokens" and stop_reason: "refusal" both arrive as successful 200 responses. Check stop_reason before trusting content.
18.11.3 Worked example — breaking things on purpose (Demo 9)
The SDK raises a typed exception for each status code. Catch them most-specific first:
try:
msg = client.messages.create(model=model, max_tokens=max_tokens, messages=msgs)
print(f"stop_reason={msg.stop_reason!r} text={text_of(msg)!r}")
except anthropic.AuthenticationError as err: # 401
print("fix the key. retryable:", is_retryable(err))
except anthropic.NotFoundError as err: # 404
print("fix the model id. retryable:", is_retryable(err))
except anthropic.RateLimitError as err: # 429
print("wait and retry. retryable:", is_retryable(err))
except anthropic.APIStatusError as err: # anything else with a status
print(f"status {err.status_code}. retryable:", is_retryable(err))=== bad key (401) === |
AuthenticationError -- fix the key. retryable: False |
|
=== bad model (404) === |
NotFoundError -- fix the model id. retryable: False |
|
=== max_tokens=5 === |
stop_reason='max_tokens' text='' <- HTTP 200, but useless |
The third case is the subtle one: no exception, a successful response, and no usable answer (the five tokens went to thinking or to the start of a sentence).
18.11.4 Rate limits: intuition and arithmetic
Rate limits are not punishment; they are capacity allocation. Your tier might allow, say, 50 RPM and 40,000 input tokens per minute. Now the TerpGrader arithmetic:
500 submissions × ~900 input tokens = 450,000 input tokens.
At 40,000 ITPM, the job takes at least ~12 minutes—
Beyond a point, parallelism doesn’t buy throughput; your token budget per minute does. A well-behaved client therefore (a) paces itself near the limit instead of slamming into it, and (b) treats 429 as normal weather, not an emergency.
18.11.5 Exponential backoff with jitter
The standard retry algorithm, and why each ingredient exists:
Exponential (1s, 2s, 4s, 8s, ...): a stressed server needs rapidly decreasing pressure, not a fixed drumbeat.
Jitter (randomize each delay): if 500 grading requests all fail at once and all retry after exactly 2s, they arrive as a synchronized stampede and fail again (the thundering herd). Randomness spreads them out.
Honor retry-after: when the server tells you exactly how long to wait, obey it instead of guessing.
Give up eventually: after N attempts, surface the error. Infinite retries hide outages.
From apidemo/retry.py—
RETRYABLE_STATUS = {408, 409, 429, 500, 502, 503, 504, 529}
def is_retryable(err: Exception) -> bool:
"""Will trying again help? 429/5xx/network: yes. 400/401/403/404: no."""
if isinstance(err, anthropic.APIConnectionError): # includes timeouts
return True
if isinstance(err, anthropic.APIStatusError):
return err.status_code in RETRYABLE_STATUS
return False
def with_retries(fn, *, max_attempts=4, base_delay=1.0, max_delay=30.0,
is_retryable=is_retryable, retry_after=retry_after_seconds,
sleep=time.sleep, rng=random.random, on_retry=None):
"""Call fn(); on a retryable error, back off and try again."""
for attempt in range(1, max_attempts + 1):
try:
return fn()
except Exception as err:
if not is_retryable(err) or attempt == max_attempts:
raise # fail fast / give up
delay = retry_after(err) # server said so...
if delay is None: # ...else backoff + jitter
delay = min(max_delay, base_delay * 2 ** (attempt - 1)) * (0.5 + rng())
if on_retry:
on_retry(attempt, delay, err)
sleep(delay)Trace—
attempt 1 ──▶ 429 (retry-after: 7) sleep 7.0s (server said so) |
attempt 2 ──▶ 429 (no header) sleep 2 × (0.5 + r) ≈ 1.6s (jittered) |
attempt 3 ──▶ 200 ✓ return response |
A 401 on attempt 1 raises immediately—
The official SDK already does this (by default 2 retries on 408, 409, 429, 5xx, and connection errors; configure with Anthropic(max_retries=...)). Demo 9 creates its clients with max_retries=0 so you see the raw errors, and its second half runs with_retries against a simulated flaky call with no network at all.
18.11.6 Common misconceptions
“429 means I’m being blocked.” It means slow down for a minute; it’s routine at any real scale.
“Catch Exception and retry everything.” Retrying non-retryable errors is the classic infinite loop that costs money. Catch typed errors and consult the taxonomy.
“More threads = faster batch job.” Only until ITPM saturates. Then more threads just means more 429s.
18.12 Core Concept 9: Cost Management
18.12.1 Definition and intuition
LLM APIs bill per token, with separate input and output prices per model, quoted in dollars per million tokens (MTok). Request cost = input_tokens × in_price + output_tokens × out_price, with discounted rates for cache reads, a premium for cache writes (about 1.25× the input price for the default 5-minute cache), and typically 50% off everything via an asynchronous batch API.
Reference prices for current Claude models (October 2026; check the provider’s pricing page, these change):
Model | Model ID | Input $/MTok | Output $/MTok | Character |
Claude Opus 5.5 | claude-opus-5-5 | $4.00 | $20.00 | flagship quality |
Claude Sonnet 5.5 | claude-sonnet-5-5 | $2.00 | $10.00 | balanced |
Claude Haiku 5.5 | claude-haiku-5-5 | $0.10 | $0.50 | fast + cheap (prompts up to 100K tokens) |
Tokens are the taxi meter. Input tokens are cheaper and you
control them (your prompt, history, and system prompt); output
tokens cost 5× more and you only cap them. On models that think,
the thinking is billed as output. The receipt (usage) arrives
with every response—
18.12.2 Worked example — pricing a TerpGrader run
One assignment: 500 submissions × (900 input + 150 output tokens).
Input: 500 × 900 = 450,000 tokens = 0.45 MTok
Output: 500 × 150 = 75,000 tokens = 0.075 MTok
Model | Input cost | Output cost | Per run | Semester (10 runs) |
Opus 5.5 | 0.45 × $4 = $1.80 | 0.075 × $20 = $1.50 | $3.30 | ~$33 |
Sonnet 5.5 | 0.45 × $2 = $0.90 | 0.075 × $10 = $0.75 | $1.65 | ~$16.50 |
Haiku 5.5 | 0.45 × $0.10 = $0.045 | 0.075 × $0.50 = $0.04 | $0.08 | ~$0.83 |
Three lessons fall out of this table:
Model choice is the biggest single lever—
40× between flagship and small. The engineering question is “what’s the cheapest model that passes our quality bar?”, answered by evaluation (grade 50 submissions with each, compare to TA grades), not by vibes. The 150 output tokens is also an estimate: a model that thinks more uses more output tokens. Measure with usage. The dollar amounts are small because the prompt is small. Paste each student’s full 20-page lab report instead of their answer, and input jumps ~50×—
prompt design is cost design. The rubric—
identical in all 500 requests— is a prompt-caching opportunity: cache it once, and later requests re-read it at a small fraction of the input price (for Opus 5.5, $0.20 instead of $4.00 per MTok).
The other standing levers:
max_tokens caps worst-case output spend.
History trimming—
resent history is billed input, every turn (Demo 4 shows it growing). Batch API—
TerpGrader is the perfect customer: submit all 500 as an asynchronous batch, get results within hours, pay 50% of everything. A local model—
for development and testing, an Ollama model costs nothing per token (Core Concept 10).
18.12.3 Worked example — a cost meter with a budget guard (Demo 10)
Reading usage is table stakes; acting on it is the point. From apidemo/cost.py:
# model: (input $/MTok, output $/MTok, cache-read $/MTok)
PRICING_PER_MTOK = {
"claude-opus-5-5": (4.00, 20.00, 0.20),
"claude-sonnet-5-5": (2.00, 10.00, 0.20),
"claude-haiku-5-5": (0.10, 0.50, 0.01),
}
CACHE_WRITE_MULTIPLIER = 1.25 # 5-minute cache writes
def cost_of(model: str, usage) -> float:
in_price, out_price, cache_read_price = PRICING_PER_MTOK[model]
per = 1_000_000
cost = _get(usage, "input_tokens") / per * in_price
cost += _get(usage, "output_tokens") / per * out_price
cost += _get(usage, "cache_read_input_tokens") / per * cache_read_price
cost += _get(usage, "cache_creation_input_tokens") / per * in_price * CACHE_WRITE_MULTIPLIER
return cost
class CostMeter:
"""Accumulates spend across calls; raises BudgetExceeded instead of overspending."""
def __init__(self, budget_usd: float | None = None):
self.budget_usd, self.total_usd, self.calls = budget_usd, 0.0, 0
def record(self, model: str, usage) -> float:
cost = cost_of(model, usage)
if self.budget_usd is not None and self.total_usd + cost > self.budget_usd:
raise BudgetExceeded(
f"${self.total_usd + cost:.4f} would exceed budget ${self.budget_usd:.4f}")
self.total_usd += cost
self.calls += 1
return costTerpGrader wraps every call in retries and the meter:
meter = CostMeter(budget_usd=args.budget)
try:
for i, answer in enumerate(submissions, 1):
msg = with_retries(lambda: client.messages.create(
model=MODEL, max_tokens=2000, system=RUBRIC,
messages=[{"role": "user", "content": f"ANSWER: {answer}"}]))
cost = meter.record(MODEL, msg.usage) # raises before overspending
verdict = json.loads(text_of(msg))
print(f"#{i}: score={verdict['score']:>2} ${cost:.5f} {verdict['feedback']}")
except BudgetExceeded as err:
print(f"STOPPED: {err}")$ python demos/10_cost_meter.py |
#1: score=10 $0.00290 Correct steps in the correct order. |
#2: score= 2 $0.00342 Steps are out of order and SYN-ACK is missing. |
#3: score= 0 $0.00301 Describes a greeting, not the TCP handshake. |
#4: score=10 $0.00296 All three steps, correctly ordered. |
#5: score= 0 $0.00255 No attempt at an answer. |
|
graded 5 answers for $0.01484 on claude-opus-5-5 |
projected 500-submission run: $3.30 |
|
$ python demos/10_cost_meter.py --budget 0.005 |
#1: score=10 $0.00290 Correct steps in the correct order. |
STOPPED: $0.0063 would exceed budget $0.0050 |
A runaway prompt bug (say, accidentally including the whole course
textbook in each request) now stops at the budget instead of being
discovered on the monthly invoice. Budget guards convert
billing surprises into exceptions—
18.12.4 Common misconceptions
“Input and output cost the same.” Output is 5× input. Verbose models and prompts are expensive on the wrong side.
“Costs are too small to matter.” Per request, yes. Per product, they compound: 10k users × 20 turns/day × growing history is how startups get five-digit surprise invoices.
“The cheapest model is always right / the best model is always right.” Both wrong—
the right model is the cheapest one that passes your measured quality bar, per task.
18.13 Core Concept 10: One Interface, Many Providers with LiteLLM
18.13.1 Motivation
So far every line of code has been Claude-specific. Real teams rarely stay with one provider:
Cost and quality comparisons. The “cheapest model that passes the bar” question from Core Concept 9 should include other providers’ models too.
Local development. Running TerpTutor against a free local Ollama model while you debug the UI saves money and works offline.
Privacy. Some data (student records under FERPA, for example) may only be processed by a local model.
Resilience. When one provider has an outage, a fallback to another keeps the service up.
Each provider has its own SDK, its own message format, and its own response shape. Writing if provider == ... branches through your whole app does not scale.
18.13.2 Definition and intuition
LiteLLM is an open-source Python library that exposes one
function, litellm.completion(), in the OpenAI chat-completions
format, and translates each call to the native API of 100+
providers. You choose the provider with a prefix on the model
string—
In plain English: it is a universal power adapter. Your appliance (your app) has one plug; the adapter has a socket for every country.
litellm.completion(model="<prefix>/<model>", messages=[...]) |
│ |
┌───────────────┬─────────────────┼──────────────────┬──────────────────┐ |
▼ ▼ ▼ ▼ ▼ |
"openai/..." "gemini/..." "anthropic/..." "ollama_chat/..." (100+ more) |
OPENAI_API_KEY GEMINI_API_KEY ANTHROPIC_API_KEY OLLAMA_API_BASE |
│ │ │ │ |
api.openai.com Google AI Studio api.anthropic.com localhost:11434 |
18.13.3 The minimal example
# pip install litellm python-dotenv
from dotenv import load_dotenv
import litellm
load_dotenv() # all keys from one .env
messages = [{"role": "system", "content": "Answer in one sentence."},
{"role": "user", "content": "What is the TCP handshake?"}]
for model in ["openai/gpt-4.1-mini",
"gemini/gemini-2.5-flash",
"anthropic/claude-opus-5-5",
"ollama_chat/llama3.2"]:
kwargs = {"api_base": "http://localhost:11434"} if model.startswith("ollama") else {}
response = litellm.completion(model=model, messages=messages, max_tokens=2000, **kwargs)
print(model, "->", response.choices[0].message.content)
print(" tokens:", response.usage.prompt_tokens, "in /",
response.usage.completion_tokens, "out")Only the model string changes. Notice what LiteLLM translated for you: the system message became Anthropic’s separate system field; Anthropic’s content blocks became choices[0].message.content; input_tokens became prompt_tokens.
Concept | Anthropic SDK | LiteLLM / OpenAI format |
system prompt | system="..." | {"role": "system", ...} message |
reply text | join text blocks of message.content | response.choices[0].message.content |
token usage | usage.input_tokens / output_tokens | usage.prompt_tokens / completion_tokens |
why it stopped | stop_reason | choices[0].finish_reason |
streaming text | stream.text_stream | chunk.choices[0].delta.content |
cost | compute from usage yourself | litellm.completion_cost(completion_response=r) |
18.13.4 The five providers in the demo project
The demo project’s switch (apidemo/llm.py) supports the five providers you will use in this course, plus an optional sixth:
–provider | Goes through | Model string sent | Needs |
openai | LiteLLM | openai/$OPENAI_MODEL | OPENAI_API_KEY |
gemini | LiteLLM | gemini/$GEMINI_MODEL | GEMINI_API_KEY |
anthropic | LiteLLM | anthropic/$ANTHROPIC_MODEL | ANTHROPIC_API_KEY |
claude-sdk | the official anthropic SDK, no translation | $ANTHROPIC_MODEL | ANTHROPIC_API_KEY |
ollama | LiteLLM to a local server | ollama_chat/$OLLAMA_MODEL | no key; OLLAMA_API_BASE |
claude-agent | Claude Agent SDK (optional) | $ANTHROPIC_MODEL | claude-agent-sdk + Claude Code CLI |
Why both anthropic and claude-sdk? They reach the same model by different roads, and comparing them is instructive. anthropic shows that Claude is just one more plug in LiteLLM. claude-sdk uses the official SDK directly, which is what you want when you need Claude-specific features the moment they ship: structured outputs via messages.parse, prompt-caching controls, typed exceptions, and the streaming helpers from Core Concept 6. A translation layer always lags the native API a little.
The optional claude-agent provider uses the Claude Agent
SDK, which packages the Claude Code agent loop as a library. It is
more than a completion call—
18.13.5 Worked example — the switch, step by step
Three ideas make the switch work. First, a table describes each provider: which variable names its model, its LiteLLM prefix, which key it needs, and whether it accepts temperature.
# apidemo/llm.py
@dataclass(frozen=True)
class Provider:
name: str
model_env: str # env var naming the model
default_model: str
litellm_prefix: str | None # None => not routed through LiteLLM
key_env: str | None # None => no key required
sampling: bool = True # does it accept temperature?
PROVIDERS = {
"openai": Provider("openai", "OPENAI_MODEL", "gpt-4.1-mini", "openai", "OPENAI_API_KEY"),
"gemini": Provider("gemini", "GEMINI_MODEL", "gemini-2.5-flash", "gemini", "GEMINI_API_KEY"),
# Current Claude models reject temperature/top_p/top_k, so sampling=False.
"anthropic": Provider("anthropic", "ANTHROPIC_MODEL", "claude-opus-5-5", "anthropic",
"ANTHROPIC_API_KEY", sampling=False),
"claude-sdk": Provider("claude-sdk", "ANTHROPIC_MODEL", "claude-opus-5-5", None,
"ANTHROPIC_API_KEY", sampling=False),
"ollama": Provider("ollama", "OLLAMA_MODEL", "llama3.2", "ollama_chat", None),
# ... claude-agent
}Second, one function resolves the flag. The order of precedence is the command-line flag, then LLM_PROVIDER from the environment or .env, then a default. It checks the key before any network call:
def resolve(provider: str | None = None) -> tuple[Provider, str]:
"""Pick the provider (flag > LLM_PROVIDER > claude-sdk) and its model name."""
name = (provider or os.environ.get("LLM_PROVIDER") or "claude-sdk").lower()
if name not in PROVIDERS:
raise ValueError(f"unknown provider {name!r}; choose from {', '.join(PROVIDERS)}")
p = PROVIDERS[name]
model = os.environ.get(p.model_env) or p.default_model
if p.key_env:
require(p.key_env) # fail fast, before any network call
return p, modelThird, ask() takes OpenAI-style messages, routes them, and returns one Reply type no matter who answered:
@dataclass
class Reply:
provider: str
model: str
text: str
input_tokens: int = 0
output_tokens: int = 0
cost_usd: float | None = None
seconds: float = 0.0
def ask(messages, *, provider=None, max_tokens=4000, temperature=None, **extra) -> Reply:
p, model = resolve(provider)
temperature = _check_sampling(p, temperature) # drop it for Claude, with a note
start = time.perf_counter()
if p.name == "claude-sdk": # native SDK path
system, rest = _split_system(messages) # system message -> system field
params = {"model": model, "max_tokens": max_tokens, "messages": rest}
if system:
params["system"] = system
msg = _anthropic_client().messages.create(**params)
return Reply(p.name, model, text_of(msg), msg.usage.input_tokens,
msg.usage.output_tokens, cost_of(model, msg.usage),
time.perf_counter() - start)
if p.name == "claude-agent":
return _ask_agent(messages, model, start)
import litellm # everything else: LiteLLM
resp = litellm.completion(**_litellm_kwargs(p, model, messages, max_tokens,
temperature, extra))
try:
cost = litellm.completion_cost(completion_response=resp)
except Exception: # unknown/local model: no price
cost = None
usage = resp.usage
return Reply(p.name, model, resp.choices[0].message.content or "",
usage.prompt_tokens, usage.completion_tokens, cost,
time.perf_counter() - start)_litellm_kwargs builds "<prefix>/<model>" and, for Ollama, adds api_base=OLLAMA_API_BASE. A matching stream() generator yields text chunks from any provider: text_stream for the native SDK, chunk.choices[0].delta.content for LiteLLM.
18.13.6 Running it
From the command line:
python -m apidemo.llm --provider openai "Explain TCP in one sentence"
python -m apidemo.llm --provider gemini "Explain TCP in one sentence"
python -m apidemo.llm --provider anthropic "Explain TCP in one sentence"
python -m apidemo.llm --provider claude-sdk "Explain TCP in one sentence"
python -m apidemo.llm --provider ollama "Explain TCP in one sentence" --stream
LLM_PROVIDER=gemini python -m apidemo.llm "Explain TCP in one sentence" # env instead of flag
python -m apidemo.llm --all "Explain TCP in one sentence" # every providerDemo 11 runs TerpTutor’s tutor prompt on every provider and prints a comparison (numbers are illustrative):
$ python demos/11_switch_providers.py --compare |
provider model secs in out cost |
openai gpt-4.1-mini 1.2 41 27 $0.000060 |
-> What could you do to both sides of the equation to get the 5x term by itself? |
gemini gemini-2.5-flash 1.9 38 24 $0.000072 |
-> Good question! What operation would remove the "+ 2" from the left side? |
anthropic claude-opus-5-5 3.1 44 96 $0.002096 |
-> Let's start by isolating the term with x. What could you do to both sides to ... |
claude-sdk claude-opus-5-5 2.9 44 91 $0.001996 |
-> What's the first step you'd take to get the 5x term alone on one side? |
ollama llama3.2 0.8 52 31 n/a |
-> What do you think we should do first to get x by itself? |
claude-agent skipped: RuntimeError |
Read the table like an engineer: every provider followed the system prompt; the local model was free and fastest on this short prompt; the flagship model cost the most and used the most output tokens (thinking is billed as output). Whether its answers are better enough is an evaluation question.
Because the transcript is plain dictionaries, a conversation can even change providers midway: demos/11_switch_providers.py keeps a messages list and passes it to whichever provider you chose. Start a session on ollama, then replay the same list against claude-sdk.
18.13.7 Local models with Ollama
Ollama (see the Ollama lecture) serves open models over HTTP on localhost:11434. To use it as a provider:
ollama serve # in one terminal (the desktop app does this for you)
ollama pull llama3.2 # download a small model once
python -m apidemo.llm --provider ollama "Explain TCP in one sentence"There is no key to leak, and no per-token bill. But the URL becomes the thing to protect: an Ollama port exposed on a public network is an unauthenticated model anyone can use. Keep it on localhost or behind your backend, just like an API key.
18.13.8 Common misconceptions and practical implications
“LiteLLM makes models interchangeable.” It makes the call interchangeable. The same prompt can behave differently on each model; re-evaluate whenever you switch.
“Every parameter works everywhere.” Feature parity is not guaranteed. Temperature is the example in this lecture: OpenAI, Gemini, and Ollama take it; current Claude models reject it, which is why the switch records sampling=False for Claude and drops the value. Structured outputs, caching, and tool use also differ.
“One .env means one secret.” It now holds several secrets, which makes the .gitignore rule more important, not less.
“The cost column is exact.” LiteLLM’s cost comes from its own price table, which can lag the providers’. The usage token counts are the ground truth; prices are an estimate.
Practical: LiteLLM also ships a proxy server that puts one OpenAI-compatible endpoint, with per-team keys and budgets, in front of all providers—
the proxy pattern from Core Concept 1, at organization scale.
18.14 Core Concept 11: Testing an API Integration Offline
CI has no API key, no budget, and sometimes no network. Every network boundary in your architecture should therefore have an injectable fake. The demo project uses three seams:
A fake SDK client. chat() accepts a client argument, so a test passes an object whose messages.create records its arguments and returns a canned response.
LiteLLM’s mock_response. Passing mock_response="..." to litellm.completion returns that text in a normal response object, with no network call—
including when streaming. Injected sleep and rng for the retry loop, so backoff tests run instantly and deterministically.
# tests/test_chat.py -- the API rejects system=None; prove we never send it
def test_chat_omits_unset_optional_params():
client = fake_client(text("ok"))
assert chat([{"role": "user", "content": "hi"}], client=client, model="m") == "ok"
sent = client.messages.calls[0]
assert "system" not in sent and "stop_sequences" not in sent
# tests/test_llm.py -- every LiteLLM provider, for free
@pytest.mark.parametrize("provider", ["openai", "gemini", "anthropic", "ollama"])
def test_litellm_providers_return_a_reply(provider):
reply = llm.ask("hi", provider=provider, mock_response="hello from mock")
assert reply.provider == provider
assert reply.text == "hello from mock"
# tests/test_retry.py -- 429 with retry-after, then 429 without, then success
def test_honors_retry_after_then_backs_off_with_jitter():
sleeps = []
fn = Flaky(api_error(429, {"retry-after": "7"}), api_error(429))
assert with_retries(fn, sleep=sleeps.append, rng=lambda: 0.3) == "ok"
assert sleeps == [7.0, pytest.approx(2.0 * 0.8)]$ pytest -q |
............................................. [100%] |
45 passed in 2.03s |
The suite also checks that a missing key fails before any network call, that .env never overrides a real environment variable, that mask() never reveals the middle of a key, that text_of skips thinking blocks, and that the budget guard fires.
18.15 The Complete Picture
.env / secret manager ──▶ os.environ ──▶ keys + LLM_PROVIDER + model names |
│ |
TerpTutor (browser) ──▶ FastAPI backend ─────┤ |
▲ streamed chunks │ messages list │ --provider flag |
│ │ + system prompt ▼ |
│ │ apidemo.llm.resolve() |
│ │ │ |
│ ▼ ▼ |
│ ask() / stream() ──┬── claude-sdk ──▶ anthropic SDK ─┐ |
│ ├── anthropic ─┐ │ |
│ ├── openai ────┼─ LiteLLM ──▶ ... ├─▶ providers |
│ ├── gemini ────┤ │ |
│ └── ollama ────┘ ──▶ localhost │ |
│ │ |
└──────────── text deltas ◀── SSE ◀──────── stop_reason, usage ◀───────┘ |
│ |
TerpGrader batch ──▶ with_retries(backoff+jitter) ──▶ CostMeter.record(usage) |
structured outputs (schema) (budget guard) |
Keys enter once, from the environment. The conversation lives in your code and is resent every call. The system prompt sets behavior; structured outputs guarantee format. Every response feeds the meter. Every call is wrapped in retries that respect the error taxonomy. And the provider is a configuration value, not a code change.
18.16 Live Demos
All demos run from the project folder after the setup below. Each takes under a minute. Demos 0 and 6 (with Ollama) and the test suite cost nothing; the rest make a few cheap API calls.
Command | Concept | What to watch for |
python demos/00_check_env.py | 1 | which key came from the shell and which from .env; all masked |
python demos/01_first_request.py | 2 | the full response object: content blocks, stop_reason, usage |
python demos/02_raw_http.py –break-key | 2, 8 | the same call as plain HTTP; a real 401 error body |
python demos/03_multi_turn.py | 3 | "another sentence" fails without history, works with it |
python demos/04_chat_loop.py | 3 | input tokens grow each turn; /reset erases memory |
python demos/05_system_prompt.py | 4 | the same question answered vs. tutored |
python demos/06_temperature.py -p ollama | 5 | identical lines at 0.0, different ones at 1.0 |
python demos/07_streaming.py race | 6 | time to first token vs. total time |
python demos/08_structured_output.py | 7 | fenced JSON; a stop sequence keeps just the score but empties fenced JSON; validated object |
python demos/09_errors_and_retries.py | 8 | typed errors; max_tokens is an HTTP 200; backoff log |
python demos/10_cost_meter.py –budget 0.005 | 9 | the budget guard stops the batch |
python demos/11_switch_providers.py –compare | 10 | one prompt, every provider: latency, tokens, cost |
pytest -q | 11 | 45 tests pass with no keys and no network |
18.17 GitHub-Ready Project: claude-api-demos
The companion project lives in notes/ai-api/code/claude-api-demos/ and can be dropped directly into a GitHub repository.
claude-api-demos/ |
├── README.md |
├── pyproject.toml # package metadata, pytest config, `llm` console script |
├── requirements.txt # anthropic, litellm, python-dotenv, httpx, pydantic, pytest |
├── .gitignore # .env is ignored; .env.example is not |
├── .env.example # every provider's variables, with placeholders |
├── apidemo/ |
│ ├── __init__.py |
│ ├── config.py # load_env (no override), require, mask |
│ ├── chat.py # add_user_message, add_assistant_message, chat, text_of |
│ ├── retry.py # is_retryable taxonomy, with_retries (backoff + jitter) |
│ ├── cost.py # pricing table, cost_of, CostMeter, BudgetExceeded |
│ └── llm.py # provider switch: LiteLLM + anthropic SDK + Agent SDK |
├── demos/ # 00_check_env.py ... 11_switch_providers.py |
└── tests/ # all offline |
├── test_config.py # .env precedence, masking, missing keys |
├── test_chat.py # helpers with a fake client |
├── test_retry.py # taxonomy, retry-after, jitter, give-up |
├── test_cost.py # pricing math, cache rates, budget guard |
└── test_llm.py # provider resolution, mock_response, system-prompt routing |
Quick start:
cd notes/ai-api/code/claude-api-demos
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
pip install -e . # makes `apidemo` importable
pytest -q # offline -- no key needed
cp .env.example .env # then paste your keys into .env
python demos/00_check_env.py # verify (masked)
python demos/01_first_request.py
python -m apidemo.llm --provider ollama "Explain the TCP handshake in one sentence."Reading order: config.py → chat.py → retry.py → cost.py → llm.py. Each file maps to a section of these notes.
18.18 Practice Exercises
18.18.1 Basic
Write the curl command that asks claude-haiku-5-5 for a haiku about College Park with at most 200 output tokens, reading the key from the environment. Label which part is authentication, which is versioning, and which is the payload.
Your teammate commits a .env file to a public repository, notices an hour later, and deletes it in the next commit. List everything that must happen now, in order, and the one change to the repository that would have prevented it.
A response comes back with stop_reason: "max_tokens". What happened, why is it a problem for TerpGrader specifically, and what are two different fixes?
Explain, with a two-message example, why “Write another sentence” produces an unrelated sentence when sent alone. Then write the messages list that fixes it.
Compute the cost of a single TerpTutor turn on Claude Sonnet 5.5 with 6,000 input tokens and 400 output tokens. Which side dominates, and what (from Core Concept 3) controls it?
18.18.2 Intermediate
Extend demos/04_chat_loop.py with a /system <text> command that changes the system prompt mid-conversation, and a /cost command that prints the running total from a CostMeter. Use streaming for the replies.
Rewrite TerpGrader (Demo 10) to use structured outputs with a Verdict Pydantic model (score: int, feedback: str) instead of json.loads. Write an offline test with a fake client that checks malformed input is never parsed by hand.
Add a fallback function to apidemo/llm.py: ask_with_fallback(messages, providers=["claude-sdk", "openai", "ollama"]) tries each in order, moving on only for retryable failures and missing keys, and returns the first successful Reply. Test it offline with mock_response and a provider whose key is missing.
Add a count_tokens pre-flight check to TerpGrader using the SDK’s client.messages.count_tokens(...): refuse to send any request whose estimated input cost would exceed a per-request cap.
Run Demo 6 on two different providers and write a one-paragraph comparison: at temperature 0.0, are the outputs identical across runs? Across providers? What does this tell you about using temperature 0 for reproducible grading?
18.18.3 Advanced
Implement a token-bucket throttle in front of the TerpGrader loop that enforces both RPM and ITPM limits client-side. Show with fake-clock tests that a 500-request burst never exceeds either budget in any 60-second window.
Build a small evaluation harness on top of apidemo.llm: grade 20 submissions with every configured provider, compare each provider’s scores to a TA’s scores, and report agreement, cost, and latency per provider. Recommend a provider and justify it with the numbers.
Design a multi-tenant budget system for TerpTutor: per-user daily budgets, a global monthly cap, and graceful degradation (switch to a cheaper provider at 80% of a user’s budget, refuse politely at 100%). What race conditions arise with concurrent requests against a shared budget, and how do you make the guard atomic?
18.19 Summary
18.19.1 Key takeaways
Keys are money. Store them in environment variables or a git-ignored .env loaded with python-dotenv; commit .env.example, never .env; keep keys on the server; rotate on any leak.
An LLM API call is ordinary HTTP: headers for auth and versioning, a JSON body naming model, max_tokens, and messages, and a JSON response whose usage object is the billing ground truth. Read the text by joining text blocks, not content[0].
The API is stateless: your code keeps the transcript and resends it every turn—
and pays for it every turn. A system prompt sets the role and behavior for the whole conversation; pass it only when set, never as None.
Temperature reshapes the next-token distribution: low for consistency, high for variety. Current Claude models no longer accept it; other providers still do.
Streaming costs the same and feels far faster; use text_stream for display and get_final_message() for storage.
For machine-readable output use structured outputs; stop sequences only cut output at a marker.
Classify every failure as retryable or not; back off exponentially with jitter; honor retry-after; give up eventually. Check stop_reason even on a 200.
Cost is an engineering variable: model tier, prompt size, history, caching, batching—
and a budget guard turns invoices into exceptions. LiteLLM puts many providers behind one call; with all keys in one .env and a –provider flag, the provider becomes configuration. The calls are interchangeable; the models’ behavior and features are not.
18.19.2 Terminology
Term | Meaning |
API key | secret credential identifying the paying account |
environment variable | named string the OS passes to a process; where keys live |
.env file | git-ignored NAME=value file loaded into the environment by python-dotenv |
anthropic-version | header pinning the wire-format version |
stateless | the server keeps no conversation; the client resends history |
system prompt | instructions in the separate system field that set role and behavior |
temperature | sampling setting that sharpens or flattens next-token probabilities |
stop sequence | string that ends generation when produced |
structured outputs | responses constrained to and validated against a JSON schema |
usage | per-response token counts --- the billing record |
stop_reason | why generation ended (end_turn, max_tokens, stop_sequence, refusal, ...) |
SSE | server-sent events; the streaming wire format |
TTFT | time to first token --- perceived latency |
RPM / ITPM / OTPM | rate-limit dimensions: requests / input tokens / output tokens per minute |
backoff + jitter | retry delays that grow and are randomized |
MTok | one million tokens --- the pricing unit |
budget guard | code that raises before spend exceeds a cap |
LiteLLM | library exposing one OpenAI-style call for 100+ providers |
provider prefix | the openai/, gemini/, anthropic/, ollama_chat/ part of a LiteLLM model string |
18.19.3 Common mistakes
Committing .env, or adding it to .gitignore only after git already tracks it.
Calling Anthropic() before load_dotenv().
Reading message.content[0].text on a model that returns a thinking block first.
Forgetting to append the assistant’s reply to the history, so the next turn loses context.
Passing system=None or stop_sequences=None instead of omitting them.
Copying notebook code that sends temperature to a current Claude model, and getting an error.
Retrying non-retryable errors; retrying without jitter.
Ignoring stop_reason and parsing truncated output.
Calling the provider directly from browser or mobile code.
Assuming a prompt tuned for one provider behaves the same on another after switching with LiteLLM.
18.19.4 Where this leads
Prompt engineering and evaluation: the system prompts and structured outputs here are the raw material; the evaluation lecture answers “is this prompt, model, or provider good enough?”
Tool use and agents: every agent loop multiplies today’s concerns—
one user turn may be dozens of API calls, so retries, budgets, and caching stop being optional. RAG: retrieval inflates input tokens; cost-aware chunking and caching decide feasibility, and a local model may be required for private documents.
MCP: auth and rate limiting for your tools, now that you have seen the provider’s side.
Deployment: secret managers, the backend proxy, and per-user budgets are where these patterns meet cloud deployment.