On this page:
18.1 Learning Objectives
18.2 Motivation and Intuition
18.2.1 From “it works on my laptop” to production
18.2.2 The mental model:   a metered, throttled, remote function call
18.2.3 Connection to earlier material
18.3 Running Examples:   Terp  Grader and Terp  Tutor
18.4 Core Concept 1:   Getting and Storing API Keys
18.4.1 Definition and intuition
18.4.2 Getting an Anthropic API key
18.4.3 The lifecycle of a key, done right
18.4.4 Option A:   an environment variable
18.4.5 Option B:   a .env file
18.4.6 Several keys in one .env
18.4.7 Who wins:   the shell or the .env file?
18.4.8 The cardinal rule:   keys stay server-side
18.4.9 Common misconceptions
18.4.10 Worked example — Terp  Grader’s deployment
18.5 Core Concept 2:   The Request Lifecycle and Anatomy of a Call
18.5.1 The five-step request flow
18.5.2 Inside step 3:   what the model does
18.5.3 The request, with nothing hidden
18.5.4 Worked example — the first SDK request (Demo 1)
18.5.5 The three levels of abstraction
18.5.6 Common misconceptions
18.6 Core Concept 3:   Multi-Turn Conversations
18.6.1 Definition and intuition
18.6.2 The problem, demonstrated (Demo 3)
18.6.3 The fix:   three helper functions
18.6.4 Worked example — a terminal chatbot (Demo 4)
18.6.5 Common misconceptions
18.7 Core Concept 4:   System Prompts
18.7.1 Definition and intuition
18.7.2 Worked example — turning Claude into Terp  Tutor (notebook 002, Demo 5)
18.7.3 Worked example — a code-only assistant (system prompt exercise)
18.7.4 Common misconceptions and practical implications
18.8 Core Concept 5:   Temperature
18.8.1 Definition and intuition
18.8.2 Choosing a temperature
18.8.3 The notebook code (notebook 003)
18.8.4 Important:   current Claude models no longer take a temperature
18.8.5 Worked example — the experiment on a provider that still has the knob (Demo 6)
18.8.6 Common misconceptions
18.9 Core Concept 6:   Streaming Responses
18.9.1 Definition and intuition
18.9.2 The events
18.9.3 Worked example — the raw event stream (notebook 004, Demo 7)
18.9.4 Worked example — just the text, plus the final message
18.9.5 Common misconceptions
18.10 Core Concept 7:   Controlling Output Format
18.10.1 Definition and intuition
18.10.2 Tool 1:   stop sequences
18.10.3 Tool 2:   structured outputs
18.10.4 Comparing the three techniques
18.11 Core Concept 8:   Errors and Rate Limits
18.11.1 Definition and intuition
18.11.2 The error taxonomy — retryable or not?
18.11.3 Worked example — breaking things on purpose (Demo 9)
18.11.4 Rate limits:   intuition and arithmetic
18.11.5 Exponential backoff with jitter
18.11.6 Common misconceptions
18.12 Core Concept 9:   Cost Management
18.12.1 Definition and intuition
18.12.2 Worked example — pricing a Terp  Grader run
18.12.3 Worked example — a cost meter with a budget guard (Demo 10)
18.12.4 Common misconceptions
18.13 Core Concept 10:   One Interface, Many Providers with Lite  LLM
18.13.1 Motivation
18.13.2 Definition and intuition
18.13.3 The minimal example
18.13.4 The five providers in the demo project
18.13.5 Worked example — the switch, step by step
18.13.6 Running it
18.13.7 Local models with Ollama
18.13.8 Common misconceptions and practical implications
18.14 Core Concept 11:   Testing an API Integration Offline
18.15 The Complete Picture
18.16 Live Demos
18.17 Git  Hub-Ready Project:   claude-api-demos
18.18 Practice Exercises
18.18.1 Basic
18.18.2 Intermediate
18.18.3 Advanced
18.19 Summary
18.19.1 Key takeaways
18.19.2 Terminology
18.19.3 Common mistakes
18.19.4 Where this leads
9.1

18 AI APIs: Calling LLMs in Production🔗

    18.1 Learning Objectives

    18.2 Motivation and Intuition

      18.2.1 From “it works on my laptop” to production

      18.2.2 The mental model: a metered, throttled, remote function call

      18.2.3 Connection to earlier material

    18.3 Running Examples: TerpGrader and TerpTutor

    18.4 Core Concept 1: Getting and Storing API Keys

      18.4.1 Definition and intuition

      18.4.2 Getting an Anthropic API key

      18.4.3 The lifecycle of a key, done right

      18.4.4 Option A: an environment variable

      18.4.5 Option B: a .env file

      18.4.6 Several keys in one .env

      18.4.7 Who wins: the shell or the .env file?

      18.4.8 The cardinal rule: keys stay server-side

      18.4.9 Common misconceptions

      18.4.10 Worked example — TerpGrader’s deployment

    18.5 Core Concept 2: The Request Lifecycle and Anatomy of a Call

      18.5.1 The five-step request flow

      18.5.2 Inside step 3: what the model does

      18.5.3 The request, with nothing hidden

      18.5.4 Worked example — the first SDK request (Demo 1)

      18.5.5 The three levels of abstraction

      18.5.6 Common misconceptions

    18.6 Core Concept 3: Multi-Turn Conversations

      18.6.1 Definition and intuition

      18.6.2 The problem, demonstrated (Demo 3)

      18.6.3 The fix: three helper functions

      18.6.4 Worked example — a terminal chatbot (Demo 4)

      18.6.5 Common misconceptions

    18.7 Core Concept 4: System Prompts

      18.7.1 Definition and intuition

      18.7.2 Worked example — turning Claude into TerpTutor (notebook 002, Demo 5)

      18.7.3 Worked example — a code-only assistant (system prompt exercise)

      18.7.4 Common misconceptions and practical implications

    18.8 Core Concept 5: Temperature

      18.8.1 Definition and intuition

      18.8.2 Choosing a temperature

      18.8.3 The notebook code (notebook 003)

      18.8.4 Important: current Claude models no longer take a temperature

      18.8.5 Worked example — the experiment on a provider that still has the knob (Demo 6)

      18.8.6 Common misconceptions

    18.9 Core Concept 6: Streaming Responses

      18.9.1 Definition and intuition

      18.9.2 The events

      18.9.3 Worked example — the raw event stream (notebook 004, Demo 7)

      18.9.4 Worked example — just the text, plus the final message

      18.9.5 Common misconceptions

    18.10 Core Concept 7: Controlling Output Format

      18.10.1 Definition and intuition

      18.10.2 Tool 1: stop sequences

      18.10.3 Tool 2: structured outputs

      18.10.4 Comparing the three techniques

    18.11 Core Concept 8: Errors and Rate Limits

      18.11.1 Definition and intuition

      18.11.2 The error taxonomy — retryable or not?

      18.11.3 Worked example — breaking things on purpose (Demo 9)

      18.11.4 Rate limits: intuition and arithmetic

      18.11.5 Exponential backoff with jitter

      18.11.6 Common misconceptions

    18.12 Core Concept 9: Cost Management

      18.12.1 Definition and intuition

      18.12.2 Worked example — pricing a TerpGrader run

      18.12.3 Worked example — a cost meter with a budget guard (Demo 10)

      18.12.4 Common misconceptions

    18.13 Core Concept 10: One Interface, Many Providers with LiteLLM

      18.13.1 Motivation

      18.13.2 Definition and intuition

      18.13.3 The minimal example

      18.13.4 The five providers in the demo project

      18.13.5 Worked example — the switch, step by step

      18.13.6 Running it

      18.13.7 Local models with Ollama

      18.13.8 Common misconceptions and practical implications

    18.14 Core Concept 11: Testing an API Integration Offline

    18.15 The Complete Picture

    18.16 Live Demos

    18.17 GitHub-Ready Project: claude-api-demos

    18.18 Practice Exercises

      18.18.1 Basic

      18.18.2 Intermediate

      18.18.3 Advanced

    18.19 Summary

      18.19.1 Key takeaways

      18.19.2 Terminology

      18.19.3 Common mistakes

      18.19.4 Where this leads

18.1 Learning Objectives🔗

By the end of this lecture you should be able to:

  1. Create an API key and store it safely in an environment variable or a git-ignored .env file, load it with python-dotenv, and explain why a key must never reach a browser, a mobile app, or a git repository.

  2. Trace a request through its five-step lifecycle (client → your server → provider → model → back) and describe the anatomy of an LLM API call—endpoint, headers, JSON body, response, and the usage object—making one with nothing but curl.

  3. Build a multi-turn conversation over a stateless API by maintaining and resending the message history, and shape behavior with a system prompt.

  4. Explain what temperature does to token sampling, choose an appropriate value for a task, and explain why current Claude models no longer expose it.

  5. Stream responses over server-sent events (SSE), and explain why streaming improves perceived latency and prevents timeouts.

  6. Get clean machine-readable output using stop sequences and schema-validated structured outputs.

  7. Classify API errors as retryable or not, implement exponential backoff with jitter, and compute the dollar cost of a request from its token usage.

  8. Use LiteLLM to keep several providers’ keys in one .env and switch between OpenAI, Gemini, Anthropic, the native Claude SDK, and a local Ollama model with a single flag.

18.2 Motivation and Intuition🔗

18.2.1 From “it works on my laptop” to production🔗

In the Ollama lecture we happily called client.messages.create(...) and moved on. That one line hides an entire engineering discipline. The moment your class project becomes a real service, five questions arrive at once:

  • Auth—where does the API key live, and who can steal it?

  • Behavior—how do I make the model act like a tutor, a grader, or a JSON generator instead of a chatty generalist?

  • Latency—why does the user stare at a spinner for 20 seconds?

  • Reliability—what happens when the provider returns 429 Too Many Requests at the worst possible moment?

  • Money and lock-in—who is paying for these tokens, and what happens if we want to switch to a cheaper or local model next month?

Every AI product team answers these same questions. Most of them are not AI problems; they are distributed-systems and operations problems wearing an AI costume—which is good news, because you already have the tools from the REST, FastAPI, and testing lectures.

18.2.2 The mental model: a metered, throttled, remote function call🔗

Think of an LLM API as calling a function that happens to live in someone else’s datacenter, where:

  • every call must present a credential (auth),

  • the function has no memory between calls (you resend the conversation every time),

  • every call is metered in tokens and billed in dollars (cost),

  • the provider enforces a speed limit per customer (rate limits),

  • and the answer arrives over a network that can be slow or flaky (streaming, retries).

None of that is visible in a happy-path demo. All of it is visible in production.

18.2.3 Connection to earlier material🔗
  • REST lecture: an LLM API is a REST-ish API—POST a JSON body, get a JSON response, act on the status code. Everything you know about HTTP applies.

  • LLM lecture: tokens, embeddings, and next-token sampling are exactly what happens inside the provider when your request lands. Temperature is that sampling step, exposed as a knob.

  • Ollama lecture: a local model server speaks HTTP too, so the same client code can talk to it—which is what LiteLLM exploits.

18.3 Running Examples: TerpGrader and TerpTutor🔗

Two services, both for a large UMD course, follow us through the lecture.

TerpGrader grades short-answer quiz submissions. Once per assignment it must grade 500 submissions, each roughly a 900-token prompt (rubric + student answer) producing a ~150-token structured verdict. It stresses the back-end topics:

  • It runs unattended, so auth must come from the environment, not a human.

  • It needs parseable JSON, not friendly prose.

  • 500 back-to-back requests will hit rate limits.

  • The department pays real money: we must predict and cap the spend per run.

TerpTutor is the interactive side: a math-help chatbot that students talk to. It stresses the front-end topics:

  • It must remember the conversation (multi-turn).

  • It must behave like a tutor—hints, not answers (system prompts).

  • It must feel responsive (streaming).

  • The department wants to try it on several providers, including a free local model for development (LiteLLM).

The concepts build in order: we first get a key, then make one TerpTutor request, give it memory, give it a personality, stream it, and finally make TerpGrader’s batch job reliable, cheap, and provider-independent.

18.4 Core Concept 1: Getting and Storing API Keys🔗

18.4.1 Definition and intuition🔗

Authentication binds each request to an account for billing, rate limiting, and access control. LLM providers commonly support: static API keys (a secret string sent in a header), OAuth bearer tokens (short-lived, obtained by a login flow), and cloud IAM signing when the model is served through a cloud platform such as AWS Bedrock or Google Vertex AI.

In plain English: the API key is a credit card number that can also talk to an AI. Anyone holding it can spend your money, up to your rate limit, until you notice.

18.4.2 Getting an Anthropic API key🔗
  1. Go to https://console.anthropic.com/ and log in.

  2. Click Get API Keys (top right of the dashboard).

  3. Click Create Key.

  4. Choose the workspace (Default is fine) and give the key a name you will recognize later, such as CMSC389A. The name is for you; it does not appear in requests.

  5. Copy the key now. It is shown exactly once. If you close the window before copying it, delete that key and create a new one.

The other providers in this lecture work the same way: OpenAI keys come from https://platform.openai.com/api-keys, Gemini keys from https://aistudio.google.com/apikey. A local Ollama server needs no key at all, only a URL.

18.4.3 The lifecycle of a key, done right🔗

create (provider console, least privilege / per-workspace)

   │

   ▼

store  ──▶ dev:  environment variable, or a .env file (git-ignored)

       ──▶ prod: secret manager (AWS Secrets Manager, GCP Secret

                 Manager, Vault) injected into the process env

   │

   ▼

use    ──▶ read from the environment at startup; never hardcode, never log

   │

   ▼

rotate ──▶ periodically, and IMMEDIATELY on any suspected leak

18.4.4 Option A: an environment variable🔗

An environment variable is a named string that the operating system hands to every process started from a shell. Your code asks for it by name, so the secret never appears in your source.

macOS / Linux, for the current terminal only:

export ANTHROPIC_API_KEY="sk-ant-api03-...your key..."
export OPENAI_API_KEY="sk-proj-..."
export GEMINI_API_KEY="AIza..."

echo ${ANTHROPIC_API_KEY:0:10}     # print only a prefix to check it is set

To make it permanent, append the export line to your shell’s startup file (~/.zshrc on macOS, ~/.bashrc on most Linux), then open a new terminal or run source ~/.zshrc.

Windows PowerShell:

$env:ANTHROPIC_API_KEY = "sk-ant-api03-..."          # current window only
setx ANTHROPIC_API_KEY "sk-ant-api03-..."            # permanent; open a new window afterwards

Reading it in Python is one line—and it is the only acceptable line:

import os
api_key = os.environ["ANTHROPIC_API_KEY"]     # KeyError (a loud crash) if missing

You rarely need even that: Anthropic(), OpenAI(), and LiteLLM all look up their standard variable names (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY) on their own.

18.4.5 Option B: a .env file🔗

Exporting variables by hand gets tedious when a project uses several keys. A .env file is a plain-text file of NAME=value lines in the project folder. The python-dotenv package reads it and copies each line into os.environ, so the rest of your code cannot tell the difference from a real environment variable.

Create .env next to your notebook or script:

# .env  -- NEVER commit this file
ANTHROPIC_API_KEY="sk-ant-api03-...your key..."

Then, as in every course notebook:

%pip install anthropic python-dotenv     # in Jupyter; use plain `pip install` in a terminal

from dotenv import load_dotenv
load_dotenv()                            # finds .env and fills os.environ

from anthropic import Anthropic
client = Anthropic()                     # picks up ANTHROPIC_API_KEY automatically

Three files, three jobs. The repository hygiene that goes with a .env file is always the same:

File

Committed?

Contains

.env

never

the real secret values

.env.example

yes

the same variable names with placeholder values, so teammates know what to fill in

.gitignore

yes

a line reading .env, so git refuses to track it

# .gitignore
.env
.env.*
!.env.example

Commit the shape of the configuration, never the values. Add the .gitignore line before you create .env: once git tracks a file, ignoring it later does not untrack it.

18.4.6 Several keys in one .env🔗

The demo project keeps every provider’s key in a single .env. This is the file the LiteLLM section uses to switch providers:

# .env  (copied from .env.example, then filled in)
LLM_PROVIDER=claude-sdk          # anthropic | claude-sdk | openai | gemini | ollama

ANTHROPIC_API_KEY=sk-ant-api03-...
ANTHROPIC_MODEL=claude-opus-5-5

OPENAI_API_KEY=sk-proj-...
OPENAI_MODEL=gpt-4.1-mini

GEMINI_API_KEY=AIza...
GEMINI_MODEL=gemini-2.5-flash

OLLAMA_API_BASE=http://localhost:11434   # local server: a URL, not a secret
OLLAMA_MODEL=llama3.2

Notice that the model names also live in configuration. Changing from Opus to Haiku, or from Gemini Flash to Pro, should be a config change, not a code change.

18.4.7 Who wins: the shell or the .env file?🔗

By default load_dotenv() does not override a variable that is already set. That is the right rule: in production a secret manager injects the real key into the process environment, and a stale .env file that was accidentally copied onto the server must never replace it. The demo project’s loader makes that choice explicit (apidemo/config.py):

from dotenv import find_dotenv, load_dotenv

def load_env() -> str | None:
    """Load the nearest .env (searching upward from the cwd). Returns its path."""
    path = find_dotenv(usecwd=True)
    if path:
        load_dotenv(path, override=False)   # never clobber real env vars
    return path or None

def require(name: str) -> str:
    """Return the value of an env var, or fail loudly with a helpful message."""
    value = os.environ.get(name, "").strip()
    if not value or "REPLACE-ME" in value:
        raise MissingKeyError(
            f"{name} is not set. Add it to your .env file (see .env.example) "
            f"or run:  export {name}=...")
    return value

def mask(secret: str | None) -> str:
    """Show just enough of a key to identify it -- safe to print or log."""
    if not secret:
        return "<unset>"
    if len(secret) <= 12:
        return "*" * len(secret)
    return f"{secret[:7]}...{secret[-4:]}"

Worked example (Demo 0). Suppose your shell exported OPENAI_API_KEY and your .env contains ANTHROPIC_API_KEY and a stale OPENAI_API_KEY. Running python demos/00_check_env.py prints, with every key masked:

.env file: /Users/you/claude-api-demos/.env

 

ANTHROPIC_API_KEY    sk-ant-...x9Qw         from .env

OPENAI_API_KEY       sk-proj...1234         from shell     <- shell wins

GEMINI_API_KEY       <unset>                from -

OLLAMA_API_BASE      http://localhost:11434

LLM_PROVIDER         claude-sdk

Step by step: the script records which keys were already in the environment, calls load_env(), and compares. ANTHROPIC_API_KEY was missing, so .env filled it in. OPENAI_API_KEY was already set, so the file’s value was ignored. GEMINI_API_KEY is in neither place; any Gemini call will fail with a clear MissingKeyError before touching the network. Run this demo first whenever “my key doesn’t work.”

18.4.8 The cardinal rule: keys stay server-side🔗

A key must never be embedded in anything you ship to users—browser JavaScript, a mobile app binary, a desktop app. All of those can be decompiled or inspected in the network tab; “hidden” is not a thing on the client.

WRONG                              RIGHT

─────                              ─────

browser ──(key!)──▶ AI API        browser ──▶ YOUR backend ──(key)──▶ AI API

                                            (auth, rate-limit,

                                             log, budget here)

The proxy pattern is also where your product’s controls live: per-user authentication, per-user rate limits, logging, abuse filtering, and budget caps all happen in your backend—none of which is possible if clients talk to the provider directly. TerpGrader follows this automatically (it is a batch job on a server); TerpTutor must route every chat message through its FastAPI backend.

18.4.9 Common misconceptions🔗
  • “It’s just a class project, nobody will find my key.” GitHub is scraped by bots continuously; a committed key is typically exploited within minutes. Providers auto-revoke keys they detect in public repos—the best-case outcome is that your job breaks.

  • “I’ll delete the commit.” Git history is forever (and already scraped). Leak = rotate. There is no other response.

  • “The repo is private, so a key in config.py is fine.” Private repos get forked, shared with TAs, made public at the end of the semester, and copied into Docker images. The key also cannot be rotated without a code change.

  • “An environment variable is secure.” It is appropriate, not magically secure—anyone with shell access to the machine can read it. It solves the “don’t put secrets in code or git” problem; production adds secret managers, scoping, and rotation on top.

  • “print(api_key) is fine for debugging.” Logs get shipped to dashboards and pasted into Slack. Print mask(key) instead.

18.4.10 Worked example — TerpGrader’s deployment🔗

TerpGrader runs as a cron job in the department’s cluster. The key is stored in the university’s Vault instance; the job’s launcher fetches it and injects ANTHROPIC_API_KEY into the process environment; the code never sees a file or a literal. When a TA accidentally pasted the key into a Slack channel, the response was: rotate in the console (30 seconds), update Vault (one command), done—because nothing was hardcoded, nothing else changed. On a developer laptop the very same code reads the key from .env, because load_dotenv(override=False) only fills in what the environment did not already provide.

18.5 Core Concept 2: The Request Lifecycle and Anatomy of a Call🔗

18.5.1 The five-step request flow🔗

Every interaction follows the same five steps, from the moment a student clicks “send” in TerpTutor to the moment the reply appears:

┌────────┐ 1. request  ┌─────────────┐ 2. request + API key  ┌──────────────┐

│ client │ ──────────▶ │ your server │ ────────────────────▶ │ provider API │

│ (web / │             │  (FastAPI)  │                       │              │

│ mobile)│ ◀────────── │             │ ◀──────────────────── │  3. model    │

└────────┘ 5. response └─────────────┘ 4. response + usage   │  processing  │

                                                             └──────────────┘

Steps 1 and 5 are your API; steps 2 and 4 are the provider’s. The key only ever travels on steps 2 and 4.

18.5.2 Inside step 3: what the model does🔗

You met these stages in the LLM lecture; here is how they map onto a single API call:

  1. Tokenization. The input text is split into tokens: whole words, word pieces, spaces, punctuation. Billing and limits are counted in these units.

  2. Embedding. Each token becomes a long vector of numbers that encodes all of its possible meanings. “Quantum” starts out ambiguous: a unit of physics, quantum mechanics, something tiny, or quantum computing.

  3. Contextualization. The transformer layers adjust each vector using the surrounding tokens, so “quantum” next to “computing” and “qubits” settles on the computing sense.

  4. Generation. An output layer turns the final vector into a probability for every possible next token. The model samples one—not always the most likely—appends it, and repeats the whole process for the next token.

After every generated token, the model checks three stopping conditions, and the response’s stop_reason tells you which one fired:

Condition

stop_reason

What it means for you

the model produced its end-of-turn token

end_turn

normal success

it hit your max_tokens ceiling

max_tokens

the answer is truncated---maybe mid-sentence or mid-JSON

it generated one of your stop sequences

stop_sequence

intentional cut-off (see Core Concept 7)

safety systems declined the request

refusal

no usable answer; handle it explicitly

18.5.3 The request, with nothing hidden🔗

Everything an SDK does ultimately becomes this curl:

curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2026-10-21" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-opus-5-5",
    "max_tokens": 1000,
    "system": "You are TerpGrader. Reply with JSON only: {\"score\": 0-10, \"feedback\": \"...\"}",
    "messages": [
      {"role": "user", "content": "RUBRIC: name the 3 steps of the TCP handshake.\nANSWER: SYN, then SYN-ACK, then ACK."}
    ]
  }'

Note $ANTHROPIC_API_KEY: the shell substitutes the environment variable, so even a one-off curl never contains the literal key. Dissecting the parts:

Part

Purpose

Notes

x-api-key

authentication

who to bill and rate-limit

anthropic-version

API versioning

pins the wire format, so provider changes don't silently break you

model

which model

exact ID strings; a typo is a 404

max_tokens

output cap

a hard safety ceiling, not a target length

system

instructions

how to behave (Core Concept 4)

messages

the transcript

stateless: the full history every time (Core Concept 3)

And the response:

{
  "id": "msg_01XFDUDYJgAACzvnptvVoYEL",
  "type": "message",
  "role": "assistant",
  "model": "claude-opus-5-5",
  "content": [
    {"type": "thinking", "thinking": "", "signature": "EqQBCkgIBRAB..."},
    {"type": "text", "text": "{\"score\": 10, \"feedback\": \"All three steps, in order.\"}"}
  ],
  "stop_reason": "end_turn",
  "usage": {"input_tokens": 78, "output_tokens": 61}
}

Three fields deserve special attention:

  • content is a list of blocks, not a string. Current Claude models may think before answering, and that thinking arrives as a thinking block before the text block (its text is omitted by default, but it is billed in output_tokens). That is why these notes never write content[0].text.

  • stop_reason tells you why generation ended. For TerpGrader, max_tokens means truncated, unparseable JSON—a real bug we must handle.

  • usage is the billing ground truth. Not your estimate, not a tokenizer library—this object is what you pay for.

18.5.4 Worked example — the first SDK request (Demo 1)🔗

The Python SDK turns that HTTP call into a method call. client.messages.create() needs three arguments: model, max_tokens, and messages.

# demos/01_first_request.py
from dotenv import load_dotenv
from anthropic import Anthropic

load_dotenv()                      # copies .env into os.environ
client = Anthropic()               # finds ANTHROPIC_API_KEY on its own
model = "claude-opus-5-5"

message = client.messages.create(
    model=model,
    max_tokens=4000,               # a safety ceiling, not a target length
    messages=[
        {"role": "user", "content": "What is quantum computing? Answer in one sentence"}
    ],
)

# Just the text.  Join the text blocks: content[0] may be a thinking block.
text = "".join(block.text for block in message.content if block.type == "text")
print("ANSWER:", text)
print("stop_reason:", message.stop_reason)
print("usage:", message.usage.input_tokens, "in /", message.usage.output_tokens, "out")

Expected output (your wording will differ):

ANSWER: Quantum computing is a type of computation that uses quantum-mechanical

phenomena like superposition and entanglement to process information in qubits,

potentially solving certain problems far faster than classical computers.

stop_reason: end_turn

usage: 18 in / 64 out

Reasoning through it:

  1. load_dotenv() runs before Anthropic(), so the key is in the environment when the client looks for it. Swap the two lines and you get an authentication error.

  2. Each message is a dictionary with a role ("user" for text a human wrote, "assistant" for text the model wrote) and content.

  3. max_tokens=4000 is a ceiling. The model does not try to fill it; it writes what it thinks is appropriate and stops. Set it high enough to cover thinking plus the answer—a too-small value truncates the reply.

  4. Some online examples read the answer with message.content[0].text. On current models content[0] can be a thinking block, so join the text blocks instead. The demo project wraps this in a helper:

# apidemo/chat.py
def text_of(message) -> str:
    """Join the text blocks of a response."""
    if message.stop_reason == "refusal":
        return "[the model declined this request]"
    return "".join(b.text for b in message.content if b.type == "text")

18.5.5 The three levels of abstraction🔗

Demo 2 (demos/02_raw_http.py) sends the TerpGrader request with the httpx library and no SDK, prints the rate-limit headers, and prints the equivalent curl. Run it with –break-key and –break-model to see real 401 and 404 bodies.

response = httpx.post(
    "https://api.anthropic.com/v1/messages",
    headers={
        "x-api-key": api_key,                  # authentication
        "anthropic-version": "2026-10-21",     # wire-format version
        "content-type": "application/json",
    },
    json=body,
    timeout=60,
)
print("HTTP", response.status_code)
print(json.dumps(response.json(), indent=2))

Level

Example

You get

You give up

raw HTTP

curl

total transparency; great for debugging

everything is manual

HTTP library

httpx (Demo 2)

testability, connection reuse, timeouts

you write retries, types, streaming parsing

official SDK

anthropic

typed responses, automatic retries on 429/5xx, auth resolution, streaming helpers, typed exceptions

some magic you should understand anyway

multi-provider library

LiteLLM (Core Concept 10)

one call shape for many providers

provider-specific features, and one more layer to debug

Rule for real projects: use the official SDK (or LiteLLM when switching providers is a requirement). We look at raw HTTP so the SDK is no longer magic.

18.5.6 Common misconceptions🔗

Misconception

Reality

"max_tokens is a target length."

It's a hard cap. The model stops early when it's done, and gets cut off (stop_reason: max_tokens) when it isn't.

"The answer is content[0].text."

content is a list of blocks; on current models the first may be a thinking block. Join the text blocks.

"I can count tokens with len(text)/4."

Fine for rough budgeting; wrong for billing. Use the usage object (after) or the count_tokens endpoint (before).

"JSON in = JSON out."

The envelope is JSON; the generated text is whatever the model wrote. Core Concept 7 covers getting parseable output.

18.6 Core Concept 3: Multi-Turn Conversations🔗

18.6.1 Definition and intuition🔗

The Messages API is stateless: the provider stores none of your conversation. Each request is complete and independent, and the model sees exactly the messages list in that request—nothing more. A multi-turn conversation is therefore something your code builds, by keeping a list of all messages and sending the whole list every time.

Think of it as talking to someone with no short-term memory who reads a transcript before every reply. If you hand them only your latest sentence, they have no idea what “it” refers to.

18.6.2 The problem, demonstrated (Demo 3)🔗

# demos/03_multi_turn.py, first half
print("Q1:", chat([{"role": "user", "content": "Define quantum computing in one sentence"}]))
print("Q2:", chat([{"role": "user", "content": "Write another sentence"}]))

Q1: Quantum computing is a paradigm of computation that uses qubits ...

Q2: The morning fog lifted slowly over the quiet harbor, revealing ...

The second request arrived with no context, so “another sentence” became a sentence about anything at all.

18.6.3 The fix: three helper functions🔗

From apidemo/chat.py:

def add_user_message(messages, text):
    user_message = {"role": "user", "content": text}
    messages.append(user_message)

def add_assistant_message(messages, text):
    assistant_message = {"role": "assistant", "content": text}
    messages.append(assistant_message)

def chat(messages):
    message = client.messages.create(
        model=model,
        max_tokens=4000,
        messages=messages,
    )
    return text_of(message)       # the notebook uses message.content[0].text

And the conversation that actually works:

# Start with an empty message list
messages = []

# Add the initial user question
add_user_message(messages, "Define quantum computing in one sentence")

# Get Claude's response, and add it to the conversation history
answer = chat(messages)
add_assistant_message(messages, answer)

# Add a follow-up question; send the full history
add_user_message(messages, "Write another sentence")
final_answer = chat(messages)
add_assistant_message(messages, final_answer)

Trace of the messages list. What the API receives on each call:

call 1: [ user: "Define quantum computing in one sentence" ]

 

call 2: [ user:      "Define quantum computing in one sentence",

          assistant: "Quantum computing is ... qubits ...",

          user:      "Write another sentence" ]          <- context included

Now “another sentence” clearly means another sentence about quantum computing. The rules: the first message is a user message, and roles normally alternate user, assistant, user, ...

18.6.4 Worked example — a terminal chatbot (Demo 4)🔗

The notebook’s chat exercise wraps the helpers in a loop. The demo version adds two commands and prints token usage each turn:

# demos/04_chat_loop.py (core loop)
messages = []
while True:
    user_input = input("\nyou> ").strip()
    if user_input == "/quit":
        break
    if user_input == "/reset":
        messages.clear()          # Claude has "forgotten" everything
        continue

    add_user_message(messages, user_input)
    response = client.messages.create(model=MODEL, max_tokens=4000, messages=messages)
    reply = text_of(response)
    add_assistant_message(messages, reply)

    print(f"claude> {reply}")
    print(f"  (turn {len(messages) // 2}: {response.usage.input_tokens} input tokens)")

A sample session:

you> My name is Testudo and I am taking CMSC389A.

claude> Nice to meet you, Testudo! ...

  (turn 1: 21 input tokens)

you> What is my name and which class am I in?

claude> Your name is Testudo, and you're taking CMSC389A.

  (turn 2: 96 input tokens)

you> /reset

you> What is my name?

claude> I don't know your name -- you haven't told me yet.

  (turn 1: 12 input tokens)

Watch the input-token count. It grows every turn because the whole history is resent—and billed—every turn. A 50-turn chat pays for turn 1 fifty times. This is why we trim history, and why it matters for cost (Core Concept 9).

18.6.5 Common misconceptions🔗
  • “The API remembers my session.” There is no session. Clear your list and the model has forgotten everything, as /reset shows.

  • “Storing history is the provider’s job.” It is yours—usually in a database keyed by conversation ID (the database lecture). Your server reloads it on each request.

  • “I can edit the history freely.” You can, and it is a powerful tool (summarizing old turns, removing a bad answer)—but the model then believes whatever the edited transcript says. Never let users write assistant messages directly.

18.7 Core Concept 4: System Prompts🔗

18.7.1 Definition and intuition🔗

A system prompt is a string passed in the separate system field of a request, outside the messages list. It tells the model what role to play and how to respond for the whole conversation. The model responds the way someone in that role would.

In plain English: the messages are the conversation; the system prompt is the job description handed to the employee before the conversation starts.

18.7.2 Worked example — turning Claude into TerpTutor (notebook 002, Demo 5)🔗

When a student asks “How do I solve 2x + 3 = 11?”, a good tutor should give hints, walk through the steps patiently, and show similar worked problems—not just state x = 4 or say “use a calculator”.

First, make the chat helper accept an optional system prompt:

def chat(messages, system=None):
    params = {
        "model": model,
        "max_tokens": 4000,
        "messages": messages,
    }
    if system:                      # the API rejects system=None,
        params["system"] = system   # so only include it when it is set

    message = client.messages.create(**params)
    return text_of(message)

The params dictionary plus **params pattern matters: the API does not accept system=None, so optional fields must be left out entirely rather than set to None. The same pattern handles stop_sequences later.

Then compare with and without a system prompt:

system_prompt = """
You are a patient math tutor. Do not directly answer the user's question.
Instead, ask follow-up questions to help the user solve the problem.
"""

messages = []
add_user_message(messages, "I need to solve the equation 2x + 3 = 11. Can you help me?")

print(chat(messages))                        # no system prompt
print(chat(messages, system=system_prompt))  # TerpTutor

Expected output (abridged):

--- no system prompt ---

Sure! Subtract 3 from both sides: 2x = 8. Then divide both sides by 2: x = 4.

 

--- with the tutor system prompt ---

Of course! Let's work through it together. Right now x has two things

attached to it: it's multiplied by 2, and 3 is added. Which of those

do you think we should undo first, and what operation would undo it?

Same model, same question, completely different behavior.

18.7.3 Worked example — a code-only assistant (system prompt exercise)🔗

The notebook exercise uses a system prompt to strip away explanation:

system_prompt = """
You are a code assistant. Return the code content only. Don't include any other text.
"""

messages = []
add_user_message(messages, "Write a Python function that checks a string for duplicate characters.")
print(chat(messages, system=system_prompt))

Expected output:

def has_duplicate_characters(s):
    """Check if a string contains duplicate characters."""
    return len(s) != len(set(s))

Often the model still wraps the code in a markdown fence. A system prompt strongly encourages a format; it does not guarantee one. Core Concept 7 shows how to guarantee it.

18.7.4 Common misconceptions and practical implications🔗
  • “A system prompt is a security boundary.” It is a strong instruction, not access control. Users can try to talk the model out of it (prompt injection, the security lecture). Enforce real rules in your code.

  • “Put the system prompt in the first user message.” Use the system field: the model treats it as the operator’s instructions, and it stays fixed while the conversation grows.

  • Practical: the system prompt is resent (and billed) with every request, like the history. A long, stable system prompt is the classic candidate for prompt caching (Core Concept 9).

18.8 Core Concept 5: Temperature🔗

18.8.1 Definition and intuition🔗

Recall the generation step: for each position the model computes a probability for every possible next token, then samples one. Temperature is a number (typically 0 to 1) that reshapes that probability distribution before sampling:

  • Low temperature (near 0) sharpens the distribution, so the most likely token almost always wins. Output is predictable and repeatable.

  • High temperature (near 1) flattens it, so less likely tokens get picked more often. Output is more varied and creative.

Worked example. After “What do you think” the model might assign: about 30%, would 20%, of 10%, and the rest spread over thousands of other tokens.

                 about   would    of    (others)

temperature 0.0  100%      0%     0%       0%      always "about"

temperature 0.5   ~55%   ~25%    ~7%     ~13%      usually "about"

temperature 1.0    30%    20%    10%      40%      the model's raw odds

Repeat that choice for every token in a paragraph and the differences compound: at 0.0 you get nearly the same sentence each run; at 1.0 you get a different story every time. Temperature changes probabilities, not guarantees—even at 1.0 two runs can come out similar.

18.8.2 Choosing a temperature🔗

Range

Good for

Running example

0.0 -- 0.3 (low)

factual answers, coding, data extraction, moderation

TerpGrader: the same answer should get the same grade

0.4 -- 0.7 (medium)

summarization, educational content, problem-solving

TerpTutor explanations

0.8 -- 1.0 (high)

brainstorming, creative writing, marketing, jokes

generating practice-problem story contexts

18.8.3 The notebook code (notebook 003)🔗

The notebook adds a temperature parameter to the chat helper and compares three settings:

def chat(messages, system=None, temperature=1.0):
    params = {
        "model": model,
        "max_tokens": 1000,
        "messages": messages,
        "temperature": temperature
    }
    if system:
        params["system"] = system
    message = client.messages.create(**params)
    return message.content[0].text

messages = []
add_user_message(messages, "Generate a one sentence story about a cat.")

print("temperature=0.0:", chat(messages, temperature=0.0))
print("temperature=0.5:", chat(messages, temperature=0.5))
print("temperature=1.0:", chat(messages, temperature=1.0))

18.8.4 Important: current Claude models no longer take a temperature🔗

The notebook code above sends temperature, but current Claude APIs no longer accept it—a good lesson in why the anthropic-version header exists:

  • Current Claude models (Opus 5.5, Sonnet 5.5, Haiku 5.5, and others) reject non-default sampling settings, and API version 2026-10-21 removes temperature, top_p, and top_k from the Messages API on every model—a request that sends them gets a 400.

  • The current Python SDK sends that version by default, and its messages.create() no longer has a temperature parameter at all; the notebook cell fails with a TypeError before any request is sent.

How do you control variety on Claude now? Through the prompt (“give five different ideas”, “answer in exactly this format”), through structured outputs when you need consistency (Core Concept 7), and by generating several candidates and choosing when you need variety.

18.8.5 Worked example — the experiment on a provider that still has the knob (Demo 6)🔗

The concept is still essential: OpenAI, Gemini, and every local Ollama model expose temperature. Demo 6 runs the notebook’s experiment through the LiteLLM provider switch (Core Concept 10), three samples per setting:

python demos/06_temperature.py --provider ollama     # free and local
python demos/06_temperature.py --provider openai

prompt = "Generate a one sentence story about a cat."
for temperature in (0.0, 0.5, 1.0):
    print(f"\n=== temperature={temperature} ===")
    for i in range(args.samples):
        reply = ask(prompt, provider=args.provider, temperature=temperature, max_tokens=100)
        print(f"  {i + 1}. {reply.text.strip()}")

Typical output with a local model:

=== temperature=0.0 ===

  1. The curious cat chased a sunbeam across the floor until it fell asleep in its warmth.

  2. The curious cat chased a sunbeam across the floor until it fell asleep in its warmth.

  3. The curious cat chased a sunbeam across the floor until it fell asleep in its warmth.

 

=== temperature=1.0 ===

  1. Whiskers stole the last sardine, then blamed the dog with a perfectly innocent blink.

  2. Every night at 3 a.m., Mochi sang opera to the refrigerator, hoping it would sing back.

  3. The old tabby guarded the library's rare-book room better than any alarm ever had.

At 0.0 the three runs are identical; at 1.0 every run is new. If you pass –temperature to a Claude provider in the switch, it prints a note and drops the parameter instead of failing.

18.8.6 Common misconceptions🔗
  • “Temperature 0 means deterministic.” Nearly, not perfectly: ties and hardware-level floating-point differences can still change output. Never rely on bit-for-bit repeatability.

  • “Higher temperature means smarter or more creative ideas.” It means less likely tokens. Past a point that is just incoherence.

  • “Temperature is a quality knob.” It is a variety knob. Use the prompt, the model choice, and evaluation for quality.

18.9 Core Concept 6: Streaming Responses🔗

18.9.1 Definition and intuition🔗

Streaming returns the response incrementally over a single long-lived HTTP response using server-sent events (SSE): a text/event-stream body where each event is an event: <type> line plus a data: <json> line, delivering pieces of the answer as the model generates them.

A TerpTutor reply can take 10–30 seconds to generate. Without streaming, your server waits for the complete response, and the student stares at a spinner with no sign that anything is happening. Non-streaming is a letter: nothing, nothing, nothing, then the whole answer. Streaming is a phone call: words as they are produced. Two latencies matter:

  • Time to first token (TTFT)—how long until something appears (about a second when streamed).

  • Total time—until the answer is complete (identical either way).

Streaming doesn’t make the model faster; it makes the experience faster—and for long outputs it is not optional: a response that takes minutes to generate will blow through HTTP timeouts unless it is streamed.

18.9.2 The events🔗

All of these arrive as part of one request:

Event

Meaning

message_start

a new message has begun; carries input-token usage

content_block_start

a new block (thinking, text, tool use) begins

content_block_delta

a chunk of the block---the text you display

content_block_stop

the current block is complete

message_delta

end-of-message info: stop_reason and output-token usage

message_stop

nothing more for this message

On the wire:

event: message_start

data: {"type":"message_start","message":{"usage":{"input_tokens":21,...}}}

 

event: content_block_delta

data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"Rex spotted"}}

 

event: content_block_delta

data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":" the cat"}}

 

event: message_delta

data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":212}}

 

event: message_stop

data: {"type":"message_stop"}

Note where the bookkeeping lives: input tokens arrive in message_start, output tokens and stop_reason in the final message_delta. A cost meter must read both ends of the stream.

18.9.3 Worked example — the raw event stream (notebook 004, Demo 7)🔗

Add stream=True to messages.create and iterate:

messages = []
add_user_message(messages, "Write 1 short story about a dog chasing a cat.")

stream = client.messages.create(
    model=model,
    max_tokens=4000,
    messages=messages,
    stream=True
)

for event in stream:
    print(event)

Run python demos/07_streaming.py events and you will see one line per event: a message_start, block starts and stops, dozens of content_block_delta events each holding a few words, then message_delta and message_stop. Only the text deltas are what the student should see.

18.9.4 Worked example — just the text, plus the final message🔗

Rather than filtering events yourself, use the SDK’s streaming helper, which yields only text and rebuilds the complete message at the end:

with client.messages.stream(
    model=model,
    max_tokens=4000,
    messages=messages
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)    # send each chunk to the client

    final_message = stream.get_final_message()   # for the database

  • flush=True matters: without it the terminal buffers output and the “streaming” appears all at once.

  • get_final_message() returns a normal message object, with content, stop_reason, and usage, exactly as if you had not streamed. TerpTutor streams chunks to the browser and stores the final message in its conversation history.

Seeing the difference. python demos/07_streaming.py race times both approaches on the same prompt:

non-streaming: first text after 8.4s (all 1093 chars at once)

streaming:     first text after 1.1s, finished after 8.2s

Same total time, about eight times faster to first output.

18.9.5 Common misconceptions🔗
  • “Streaming is cheaper.” Same tokens, same price. It changes when you see them, not how many there are.

  • “Streaming is just for looks.” Also for correctness: long generations require it to avoid timeouts, which is why the SDKs insist on streaming for very large max_tokens.

  • “I can reassemble the message from the deltas I printed.” Use get_final_message(). Dropped events and non-text blocks make naive reassembly buggy.

  • “Errors arrive as events.” A 429 or 401 arrives before any events, as a normal error response; the SDK raises it when you open the stream.

18.10 Core Concept 7: Controlling Output Format🔗

18.10.1 Definition and intuition🔗

When a program—not a person—consumes the output, you need exactly the data and nothing else. The model’s default helpfulness gets in the way: ask for JSON and you typically get

```json

{

  "source": ["aws.ec2"],

  "detail-type": ["EC2 Instance State-change Notification"],

  "detail": { "state": ["running"] }

}

```

This rule captures EC2 instance state changes when instances start running.

The JSON is correct, but json.loads fails on the fence and the explanation, and a user of an “EventBridge rule generator” web app cannot just copy the whole response. TerpGrader has the same problem with its verdicts. Beyond asking nicely in the system prompt, there are two tools.

18.10.2 Tool 1: stop sequences🔗

stop_sequences is a list of strings; generation ends the moment the model produces one of them, and stop_reason becomes stop_sequence. The stop string itself is not included in the output. Add it to the chat helper the same way as system:

def chat(messages, system=None, stop_sequences=None):
    params = {"model": model, "max_tokens": 4000, "messages": messages}
    if system:
        params["system"] = system
    if stop_sequences:
        params["stop_sequences"] = stop_sequences
    message = client.messages.create(**params)
    return text_of(message)

Worked example — just the score. Ask TerpGrader to put the score on the first line, then stop at the first newline:

messages = []
add_user_message(messages,
    "RUBRIC: name the 3 steps of the TCP handshake.\n"
    "ANSWER: SYN, then SYN-ACK, then ACK.\n"
    "Reply with 'SCORE: <0-10>' on the first line, then your feedback.")

print(repr(chat(messages, stop_sequences=["\n"])))

'SCORE: 10'

The feedback is never generated, so it is never billed, and int(text.split(":")[1]) parses the score.

The limit shows up as soon as you try the obvious fix for fenced JSON: stop_sequences=["“‘"]. The opening fence is the first thing the model writes, so generation stops immediately and you get an empty string. A stop sequence can cut off what comes after the data, not what comes before it. For that you need Tool 2.

18.10.3 Tool 2: structured outputs🔗

The modern replacement constrains the response to a JSON schema and validates it for you. With the Python SDK you describe the shape as a Pydantic model and call messages.parse:

from pydantic import BaseModel

class EventPattern(BaseModel):
    source: list[str]
    detail_type: list[str]
    states: list[str]

response = client.messages.parse(
    model="claude-opus-5-5",
    max_tokens=4000,
    messages=[{"role": "user",
               "content": "Generate a very short EventBridge pattern that matches EC2 "
                          "instances entering the running state."}],
    output_format=EventPattern,
)
rule = response.parsed_output        # a validated EventPattern object
print(rule.source, rule.states)      # ['aws.ec2'] ['running']

Without Pydantic, pass the schema directly with output_config={"format": {"type": "json_schema", "schema": ...}} on messages.create; the text block then contains JSON that matches the schema.

For TerpGrader this is the right tool: a Verdict model with score: int and feedback: str means every one of the 500 replies parses—no fences, no apologies, no missing fields. (It still needs a large enough max_tokens: a truncated reply is still truncated.)

18.10.4 Comparing the three techniques🔗

Technique

Works on

Guarantee

Use for

system prompt ("JSON only")

all models

none---usually works

quick experiments

stop sequences

all models

cuts output at a marker

ending lists, removing trailing text

structured outputs

current models

valid against your schema

anything a program parses

18.11 Core Concept 8: Errors and Rate Limits🔗

18.11.1 Definition and intuition🔗

LLM APIs signal failures with standard HTTP status codes plus a JSON error body. A rate limit is a provider-enforced cap per organization, measured along several dimensions simultaneously—typically requests per minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM)—signaled by 429 responses carrying a retry-after header.

18.11.2 The error taxonomy — retryable or not?🔗

The single most useful classification of any failure is “will trying again help?”

Code

Meaning

Retryable?

Correct reaction

400

malformed request

no

fix your code (bad body, unsupported parameter)

401

bad/missing key

no

fix your configuration (run Demo 0)

403

key lacks permission

no

fix your account/scoping

404

bad model ID / endpoint

no

fix the model string

413

request too large

no

shrink the input (trim history, chunk documents)

429

rate limited

YES

wait (retry-after), then retry

500

provider bug

YES

retry with backoff

529

provider overloaded

YES

retry with backoff, consider shedding load

Retrying a 400 is a bug that burns time and money forever; failing permanently on a 429 is a bug that makes your service fragile for no reason. The taxonomy is the design.

Not every failure is an exception: stop_reason: "max_tokens" and stop_reason: "refusal" both arrive as successful 200 responses. Check stop_reason before trusting content.

18.11.3 Worked example — breaking things on purpose (Demo 9)🔗

The SDK raises a typed exception for each status code. Catch them most-specific first:

try:
    msg = client.messages.create(model=model, max_tokens=max_tokens, messages=msgs)
    print(f"stop_reason={msg.stop_reason!r}  text={text_of(msg)!r}")
except anthropic.AuthenticationError as err:     # 401
    print("fix the key.  retryable:", is_retryable(err))
except anthropic.NotFoundError as err:           # 404
    print("fix the model id.  retryable:", is_retryable(err))
except anthropic.RateLimitError as err:          # 429
    print("wait and retry.  retryable:", is_retryable(err))
except anthropic.APIStatusError as err:          # anything else with a status
    print(f"status {err.status_code}.  retryable:", is_retryable(err))

=== bad key (401) ===

AuthenticationError -- fix the key.  retryable: False

 

=== bad model (404) ===

NotFoundError -- fix the model id.  retryable: False

 

=== max_tokens=5 ===

stop_reason='max_tokens'  text=''        <- HTTP 200, but useless

The third case is the subtle one: no exception, a successful response, and no usable answer (the five tokens went to thinking or to the start of a sentence).

18.11.4 Rate limits: intuition and arithmetic🔗

Rate limits are not punishment; they are capacity allocation. Your tier might allow, say, 50 RPM and 40,000 input tokens per minute. Now the TerpGrader arithmetic:

500 submissions × ~900 input tokens = 450,000 input tokens. At 40,000 ITPM, the job takes at least ~12 minutes—no matter how parallel your code is.

Beyond a point, parallelism doesn’t buy throughput; your token budget per minute does. A well-behaved client therefore (a) paces itself near the limit instead of slamming into it, and (b) treats 429 as normal weather, not an emergency.

18.11.5 Exponential backoff with jitter🔗

The standard retry algorithm, and why each ingredient exists:

  • Exponential (1s, 2s, 4s, 8s, ...): a stressed server needs rapidly decreasing pressure, not a fixed drumbeat.

  • Jitter (randomize each delay): if 500 grading requests all fail at once and all retry after exactly 2s, they arrive as a synchronized stampede and fail again (the thundering herd). Randomness spreads them out.

  • Honor retry-after: when the server tells you exactly how long to wait, obey it instead of guessing.

  • Give up eventually: after N attempts, surface the error. Infinite retries hide outages.

From apidemo/retry.py—sleep and rng are injected parameters, which is what makes the tests instant:

RETRYABLE_STATUS = {408, 409, 429, 500, 502, 503, 504, 529}

def is_retryable(err: Exception) -> bool:
    """Will trying again help?  429/5xx/network: yes.  400/401/403/404: no."""
    if isinstance(err, anthropic.APIConnectionError):    # includes timeouts
        return True
    if isinstance(err, anthropic.APIStatusError):
        return err.status_code in RETRYABLE_STATUS
    return False

def with_retries(fn, *, max_attempts=4, base_delay=1.0, max_delay=30.0,
                 is_retryable=is_retryable, retry_after=retry_after_seconds,
                 sleep=time.sleep, rng=random.random, on_retry=None):
    """Call fn(); on a retryable error, back off and try again."""
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except Exception as err:
            if not is_retryable(err) or attempt == max_attempts:
                raise                                   # fail fast / give up
            delay = retry_after(err)                    # server said so...
            if delay is None:                           # ...else backoff + jitter
                delay = min(max_delay, base_delay * 2 ** (attempt - 1)) * (0.5 + rng())
            if on_retry:
                on_retry(attempt, delay, err)
            sleep(delay)

Trace—a 429 storm during grading, base_delay=1.0:

attempt 1 ──▶ 429 (retry-after: 7)   sleep 7.0s                 (server said so)

attempt 2 ──▶ 429 (no header)        sleep 2 × (0.5 + r) ≈ 1.6s  (jittered)

attempt 3 ──▶ 200 ✓                  return response

A 401 on attempt 1 raises immediately—is_retryable is False—and that’s correct: no amount of waiting fixes a bad key.

The official SDK already does this (by default 2 retries on 408, 409, 429, 5xx, and connection errors; configure with Anthropic(max_retries=...)). Demo 9 creates its clients with max_retries=0 so you see the raw errors, and its second half runs with_retries against a simulated flaky call with no network at all.

18.11.6 Common misconceptions🔗
  • “429 means I’m being blocked.” It means slow down for a minute; it’s routine at any real scale.

  • “Catch Exception and retry everything.” Retrying non-retryable errors is the classic infinite loop that costs money. Catch typed errors and consult the taxonomy.

  • “More threads = faster batch job.” Only until ITPM saturates. Then more threads just means more 429s.

18.12 Core Concept 9: Cost Management🔗

18.12.1 Definition and intuition🔗

LLM APIs bill per token, with separate input and output prices per model, quoted in dollars per million tokens (MTok). Request cost = input_tokens × in_price + output_tokens × out_price, with discounted rates for cache reads, a premium for cache writes (about 1.25× the input price for the default 5-minute cache), and typically 50% off everything via an asynchronous batch API.

Reference prices for current Claude models (October 2026; check the provider’s pricing page, these change):

Model

Model ID

Input $/MTok

Output $/MTok

Character

Claude Opus 5.5

claude-opus-5-5

$4.00

$20.00

flagship quality

Claude Sonnet 5.5

claude-sonnet-5-5

$2.00

$10.00

balanced

Claude Haiku 5.5

claude-haiku-5-5

$0.10

$0.50

fast + cheap (prompts up to 100K tokens)

Tokens are the taxi meter. Input tokens are cheaper and you control them (your prompt, history, and system prompt); output tokens cost 5× more and you only cap them. On models that think, the thinking is billed as output. The receipt (usage) arrives with every response—a system that doesn’t read it doesn’t know what it spends.

18.12.2 Worked example — pricing a TerpGrader run🔗

One assignment: 500 submissions × (900 input + 150 output tokens).

  • Input: 500 × 900 = 450,000 tokens = 0.45 MTok

  • Output: 500 × 150 = 75,000 tokens = 0.075 MTok

Model

Input cost

Output cost

Per run

Semester (10 runs)

Opus 5.5

0.45 × $4 = $1.80

0.075 × $20 = $1.50

$3.30

~$33

Sonnet 5.5

0.45 × $2 = $0.90

0.075 × $10 = $0.75

$1.65

~$16.50

Haiku 5.5

0.45 × $0.10 = $0.045

0.075 × $0.50 = $0.04

$0.08

~$0.83

Three lessons fall out of this table:

  1. Model choice is the biggest single lever—40× between flagship and small. The engineering question is “what’s the cheapest model that passes our quality bar?”, answered by evaluation (grade 50 submissions with each, compare to TA grades), not by vibes. The 150 output tokens is also an estimate: a model that thinks more uses more output tokens. Measure with usage.

  2. The dollar amounts are small because the prompt is small. Paste each student’s full 20-page lab report instead of their answer, and input jumps ~50×—prompt design is cost design.

  3. The rubric—identical in all 500 requests—is a prompt-caching opportunity: cache it once, and later requests re-read it at a small fraction of the input price (for Opus 5.5, $0.20 instead of $4.00 per MTok).

The other standing levers:

  • max_tokens caps worst-case output spend.

  • History trimming—resent history is billed input, every turn (Demo 4 shows it growing).

  • Batch API—TerpGrader is the perfect customer: submit all 500 as an asynchronous batch, get results within hours, pay 50% of everything.

  • A local model—for development and testing, an Ollama model costs nothing per token (Core Concept 10).

18.12.3 Worked example — a cost meter with a budget guard (Demo 10)🔗

Reading usage is table stakes; acting on it is the point. From apidemo/cost.py:

# model: (input $/MTok, output $/MTok, cache-read $/MTok)
PRICING_PER_MTOK = {
    "claude-opus-5-5":   (4.00, 20.00, 0.20),
    "claude-sonnet-5-5": (2.00, 10.00, 0.20),
    "claude-haiku-5-5":  (0.10, 0.50, 0.01),
}
CACHE_WRITE_MULTIPLIER = 1.25                  # 5-minute cache writes

def cost_of(model: str, usage) -> float:
    in_price, out_price, cache_read_price = PRICING_PER_MTOK[model]
    per = 1_000_000
    cost  = _get(usage, "input_tokens") / per * in_price
    cost += _get(usage, "output_tokens") / per * out_price
    cost += _get(usage, "cache_read_input_tokens") / per * cache_read_price
    cost += _get(usage, "cache_creation_input_tokens") / per * in_price * CACHE_WRITE_MULTIPLIER
    return cost

class CostMeter:
    """Accumulates spend across calls; raises BudgetExceeded instead of overspending."""
    def __init__(self, budget_usd: float | None = None):
        self.budget_usd, self.total_usd, self.calls = budget_usd, 0.0, 0

    def record(self, model: str, usage) -> float:
        cost = cost_of(model, usage)
        if self.budget_usd is not None and self.total_usd + cost > self.budget_usd:
            raise BudgetExceeded(
                f"${self.total_usd + cost:.4f} would exceed budget ${self.budget_usd:.4f}")
        self.total_usd += cost
        self.calls += 1
        return cost

TerpGrader wraps every call in retries and the meter:

meter = CostMeter(budget_usd=args.budget)
try:
    for i, answer in enumerate(submissions, 1):
        msg = with_retries(lambda: client.messages.create(
            model=MODEL, max_tokens=2000, system=RUBRIC,
            messages=[{"role": "user", "content": f"ANSWER: {answer}"}]))
        cost = meter.record(MODEL, msg.usage)           # raises before overspending
        verdict = json.loads(text_of(msg))
        print(f"#{i}: score={verdict['score']:>2}  ${cost:.5f}  {verdict['feedback']}")
except BudgetExceeded as err:
    print(f"STOPPED: {err}")

$ python demos/10_cost_meter.py

#1: score=10  $0.00290  Correct steps in the correct order.

#2: score= 2  $0.00342  Steps are out of order and SYN-ACK is missing.

#3: score= 0  $0.00301  Describes a greeting, not the TCP handshake.

#4: score=10  $0.00296  All three steps, correctly ordered.

#5: score= 0  $0.00255  No attempt at an answer.

 

graded 5 answers for $0.01484 on claude-opus-5-5

projected 500-submission run: $3.30

 

$ python demos/10_cost_meter.py --budget 0.005

#1: score=10  $0.00290  Correct steps in the correct order.

STOPPED: $0.0063 would exceed budget $0.0050

A runaway prompt bug (say, accidentally including the whole course textbook in each request) now stops at the budget instead of being discovered on the monthly invoice. Budget guards convert billing surprises into exceptions—and exceptions are something you already know how to handle. (Strictly, the meter records a cost after the call returns, so it stops the next call; a pre-flight check with the count_tokens endpoint is an exercise.)

18.12.4 Common misconceptions🔗
  • “Input and output cost the same.” Output is 5× input. Verbose models and prompts are expensive on the wrong side.

  • “Costs are too small to matter.” Per request, yes. Per product, they compound: 10k users × 20 turns/day × growing history is how startups get five-digit surprise invoices.

  • “The cheapest model is always right / the best model is always right.” Both wrong—the right model is the cheapest one that passes your measured quality bar, per task.

18.13 Core Concept 10: One Interface, Many Providers with LiteLLM🔗

18.13.1 Motivation🔗

So far every line of code has been Claude-specific. Real teams rarely stay with one provider:

  • Cost and quality comparisons. The “cheapest model that passes the bar” question from Core Concept 9 should include other providers’ models too.

  • Local development. Running TerpTutor against a free local Ollama model while you debug the UI saves money and works offline.

  • Privacy. Some data (student records under FERPA, for example) may only be processed by a local model.

  • Resilience. When one provider has an outage, a fallback to another keeps the service up.

Each provider has its own SDK, its own message format, and its own response shape. Writing if provider == ... branches through your whole app does not scale.

18.13.2 Definition and intuition🔗

LiteLLM is an open-source Python library that exposes one function, litellm.completion(), in the OpenAI chat-completions format, and translates each call to the native API of 100+ providers. You choose the provider with a prefix on the model string—"openai/gpt-4.1-mini", "gemini/gemini-2.5-flash", "anthropic/claude-opus-5-5", "ollama_chat/llama3.2"—and LiteLLM reads that provider’s key from its standard environment variable. Every response comes back in the same OpenAI shape.

In plain English: it is a universal power adapter. Your appliance (your app) has one plug; the adapter has a socket for every country.

                     litellm.completion(model="<prefix>/<model>", messages=[...])

                                        │

      ┌───────────────┬─────────────────┼──────────────────┬──────────────────┐

      ▼               ▼                 ▼                  ▼                  ▼

"openai/..."    "gemini/..."      "anthropic/..."   "ollama_chat/..."   (100+ more)

OPENAI_API_KEY  GEMINI_API_KEY    ANTHROPIC_API_KEY  OLLAMA_API_BASE

      │               │                 │                  │

 api.openai.com  Google AI Studio  api.anthropic.com  localhost:11434

18.13.3 The minimal example🔗

# pip install litellm python-dotenv
from dotenv import load_dotenv
import litellm

load_dotenv()                                   # all keys from one .env

messages = [{"role": "system", "content": "Answer in one sentence."},
            {"role": "user",   "content": "What is the TCP handshake?"}]

for model in ["openai/gpt-4.1-mini",
              "gemini/gemini-2.5-flash",
              "anthropic/claude-opus-5-5",
              "ollama_chat/llama3.2"]:
    kwargs = {"api_base": "http://localhost:11434"} if model.startswith("ollama") else {}
    response = litellm.completion(model=model, messages=messages, max_tokens=2000, **kwargs)
    print(model, "->", response.choices[0].message.content)
    print("   tokens:", response.usage.prompt_tokens, "in /",
          response.usage.completion_tokens, "out")

Only the model string changes. Notice what LiteLLM translated for you: the system message became Anthropic’s separate system field; Anthropic’s content blocks became choices[0].message.content; input_tokens became prompt_tokens.

Concept

Anthropic SDK

LiteLLM / OpenAI format

system prompt

system="..."

{"role": "system", ...} message

reply text

join text blocks of message.content

response.choices[0].message.content

token usage

usage.input_tokens / output_tokens

usage.prompt_tokens / completion_tokens

why it stopped

stop_reason

choices[0].finish_reason

streaming text

stream.text_stream

chunk.choices[0].delta.content

cost

compute from usage yourself

litellm.completion_cost(completion_response=r)

18.13.4 The five providers in the demo project🔗

The demo project’s switch (apidemo/llm.py) supports the five providers you will use in this course, plus an optional sixth:

–provider

Goes through

Model string sent

Needs

openai

LiteLLM

openai/$OPENAI_MODEL

OPENAI_API_KEY

gemini

LiteLLM

gemini/$GEMINI_MODEL

GEMINI_API_KEY

anthropic

LiteLLM

anthropic/$ANTHROPIC_MODEL

ANTHROPIC_API_KEY

claude-sdk

the official anthropic SDK, no translation

$ANTHROPIC_MODEL

ANTHROPIC_API_KEY

ollama

LiteLLM to a local server

ollama_chat/$OLLAMA_MODEL

no key; OLLAMA_API_BASE

claude-agent

Claude Agent SDK (optional)

$ANTHROPIC_MODEL

claude-agent-sdk + Claude Code CLI

Why both anthropic and claude-sdk? They reach the same model by different roads, and comparing them is instructive. anthropic shows that Claude is just one more plug in LiteLLM. claude-sdk uses the official SDK directly, which is what you want when you need Claude-specific features the moment they ship: structured outputs via messages.parse, prompt-caching controls, typed exceptions, and the streaming helpers from Core Concept 6. A translation layer always lags the native API a little.

The optional claude-agent provider uses the Claude Agent SDK, which packages the Claude Code agent loop as a library. It is more than a completion call—it can read files and run tools—so the switch disables its tools and limits it to one turn to make it behave like the others. Install it with pip install claude-agent-sdk and install the Claude Code CLI.

18.13.5 Worked example — the switch, step by step🔗

Three ideas make the switch work. First, a table describes each provider: which variable names its model, its LiteLLM prefix, which key it needs, and whether it accepts temperature.

# apidemo/llm.py
@dataclass(frozen=True)
class Provider:
    name: str
    model_env: str              # env var naming the model
    default_model: str
    litellm_prefix: str | None  # None => not routed through LiteLLM
    key_env: str | None         # None => no key required
    sampling: bool = True       # does it accept temperature?

PROVIDERS = {
    "openai":     Provider("openai", "OPENAI_MODEL", "gpt-4.1-mini", "openai", "OPENAI_API_KEY"),
    "gemini":     Provider("gemini", "GEMINI_MODEL", "gemini-2.5-flash", "gemini", "GEMINI_API_KEY"),
    # Current Claude models reject temperature/top_p/top_k, so sampling=False.
    "anthropic":  Provider("anthropic", "ANTHROPIC_MODEL", "claude-opus-5-5", "anthropic",
                           "ANTHROPIC_API_KEY", sampling=False),
    "claude-sdk": Provider("claude-sdk", "ANTHROPIC_MODEL", "claude-opus-5-5", None,
                           "ANTHROPIC_API_KEY", sampling=False),
    "ollama":     Provider("ollama", "OLLAMA_MODEL", "llama3.2", "ollama_chat", None),
    # ... claude-agent
}

Second, one function resolves the flag. The order of precedence is the command-line flag, then LLM_PROVIDER from the environment or .env, then a default. It checks the key before any network call:

def resolve(provider: str | None = None) -> tuple[Provider, str]:
    """Pick the provider (flag > LLM_PROVIDER > claude-sdk) and its model name."""
    name = (provider or os.environ.get("LLM_PROVIDER") or "claude-sdk").lower()
    if name not in PROVIDERS:
        raise ValueError(f"unknown provider {name!r}; choose from {', '.join(PROVIDERS)}")
    p = PROVIDERS[name]
    model = os.environ.get(p.model_env) or p.default_model
    if p.key_env:
        require(p.key_env)                     # fail fast, before any network call
    return p, model

Third, ask() takes OpenAI-style messages, routes them, and returns one Reply type no matter who answered:

@dataclass
class Reply:
    provider: str
    model: str
    text: str
    input_tokens: int = 0
    output_tokens: int = 0
    cost_usd: float | None = None
    seconds: float = 0.0

def ask(messages, *, provider=None, max_tokens=4000, temperature=None, **extra) -> Reply:
    p, model = resolve(provider)
    temperature = _check_sampling(p, temperature)     # drop it for Claude, with a note
    start = time.perf_counter()

    if p.name == "claude-sdk":                        # native SDK path
        system, rest = _split_system(messages)        # system message -> system field
        params = {"model": model, "max_tokens": max_tokens, "messages": rest}
        if system:
            params["system"] = system
        msg = _anthropic_client().messages.create(**params)
        return Reply(p.name, model, text_of(msg), msg.usage.input_tokens,
                     msg.usage.output_tokens, cost_of(model, msg.usage),
                     time.perf_counter() - start)

    if p.name == "claude-agent":
        return _ask_agent(messages, model, start)

    import litellm                                    # everything else: LiteLLM
    resp = litellm.completion(**_litellm_kwargs(p, model, messages, max_tokens,
                                                temperature, extra))
    try:
        cost = litellm.completion_cost(completion_response=resp)
    except Exception:                                 # unknown/local model: no price
        cost = None
    usage = resp.usage
    return Reply(p.name, model, resp.choices[0].message.content or "",
                 usage.prompt_tokens, usage.completion_tokens, cost,
                 time.perf_counter() - start)

_litellm_kwargs builds "<prefix>/<model>" and, for Ollama, adds api_base=OLLAMA_API_BASE. A matching stream() generator yields text chunks from any provider: text_stream for the native SDK, chunk.choices[0].delta.content for LiteLLM.

18.13.6 Running it🔗

From the command line:

python -m apidemo.llm --provider openai     "Explain TCP in one sentence"
python -m apidemo.llm --provider gemini     "Explain TCP in one sentence"
python -m apidemo.llm --provider anthropic  "Explain TCP in one sentence"
python -m apidemo.llm --provider claude-sdk "Explain TCP in one sentence"
python -m apidemo.llm --provider ollama     "Explain TCP in one sentence" --stream

LLM_PROVIDER=gemini python -m apidemo.llm "Explain TCP in one sentence"   # env instead of flag
python -m apidemo.llm --all "Explain TCP in one sentence"                 # every provider

Demo 11 runs TerpTutor’s tutor prompt on every provider and prints a comparison (numbers are illustrative):

$ python demos/11_switch_providers.py --compare

provider      model                 secs    in   out       cost

openai        gpt-4.1-mini           1.2    41    27  $0.000060

   -> What could you do to both sides of the equation to get the 5x term by itself?

gemini        gemini-2.5-flash       1.9    38    24  $0.000072

   -> Good question! What operation would remove the "+ 2" from the left side?

anthropic     claude-opus-5-5        3.1    44    96  $0.002096

   -> Let's start by isolating the term with x. What could you do to both sides to ...

claude-sdk    claude-opus-5-5        2.9    44    91  $0.001996

   -> What's the first step you'd take to get the 5x term alone on one side?

ollama        llama3.2               0.8    52    31        n/a

   -> What do you think we should do first to get x by itself?

claude-agent  skipped: RuntimeError

Read the table like an engineer: every provider followed the system prompt; the local model was free and fastest on this short prompt; the flagship model cost the most and used the most output tokens (thinking is billed as output). Whether its answers are better enough is an evaluation question.

Because the transcript is plain dictionaries, a conversation can even change providers midway: demos/11_switch_providers.py keeps a messages list and passes it to whichever provider you chose. Start a session on ollama, then replay the same list against claude-sdk.

18.13.7 Local models with Ollama🔗

Ollama (see the Ollama lecture) serves open models over HTTP on localhost:11434. To use it as a provider:

ollama serve                 # in one terminal (the desktop app does this for you)
ollama pull llama3.2         # download a small model once
python -m apidemo.llm --provider ollama "Explain TCP in one sentence"

There is no key to leak, and no per-token bill. But the URL becomes the thing to protect: an Ollama port exposed on a public network is an unauthenticated model anyone can use. Keep it on localhost or behind your backend, just like an API key.

18.13.8 Common misconceptions and practical implications🔗
  • “LiteLLM makes models interchangeable.” It makes the call interchangeable. The same prompt can behave differently on each model; re-evaluate whenever you switch.

  • “Every parameter works everywhere.” Feature parity is not guaranteed. Temperature is the example in this lecture: OpenAI, Gemini, and Ollama take it; current Claude models reject it, which is why the switch records sampling=False for Claude and drops the value. Structured outputs, caching, and tool use also differ.

  • “One .env means one secret.” It now holds several secrets, which makes the .gitignore rule more important, not less.

  • “The cost column is exact.” LiteLLM’s cost comes from its own price table, which can lag the providers’. The usage token counts are the ground truth; prices are an estimate.

  • Practical: LiteLLM also ships a proxy server that puts one OpenAI-compatible endpoint, with per-team keys and budgets, in front of all providers—the proxy pattern from Core Concept 1, at organization scale.

18.14 Core Concept 11: Testing an API Integration Offline🔗

CI has no API key, no budget, and sometimes no network. Every network boundary in your architecture should therefore have an injectable fake. The demo project uses three seams:

  • A fake SDK client. chat() accepts a client argument, so a test passes an object whose messages.create records its arguments and returns a canned response.

  • LiteLLM’s mock_response. Passing mock_response="..." to litellm.completion returns that text in a normal response object, with no network call—including when streaming.

  • Injected sleep and rng for the retry loop, so backoff tests run instantly and deterministically.

# tests/test_chat.py -- the API rejects system=None; prove we never send it
def test_chat_omits_unset_optional_params():
    client = fake_client(text("ok"))
    assert chat([{"role": "user", "content": "hi"}], client=client, model="m") == "ok"
    sent = client.messages.calls[0]
    assert "system" not in sent and "stop_sequences" not in sent

# tests/test_llm.py -- every LiteLLM provider, for free
@pytest.mark.parametrize("provider", ["openai", "gemini", "anthropic", "ollama"])
def test_litellm_providers_return_a_reply(provider):
    reply = llm.ask("hi", provider=provider, mock_response="hello from mock")
    assert reply.provider == provider
    assert reply.text == "hello from mock"

# tests/test_retry.py -- 429 with retry-after, then 429 without, then success
def test_honors_retry_after_then_backs_off_with_jitter():
    sleeps = []
    fn = Flaky(api_error(429, {"retry-after": "7"}), api_error(429))
    assert with_retries(fn, sleep=sleeps.append, rng=lambda: 0.3) == "ok"
    assert sleeps == [7.0, pytest.approx(2.0 * 0.8)]

$ pytest -q

.............................................                     [100%]

45 passed in 2.03s

The suite also checks that a missing key fails before any network call, that .env never overrides a real environment variable, that mask() never reveals the middle of a key, that text_of skips thinking blocks, and that the budget guard fires.

18.15 The Complete Picture🔗

.env / secret manager ──▶ os.environ ──▶ keys + LLM_PROVIDER + model names

                                             │

TerpTutor (browser) ──▶ FastAPI backend ─────┤

   ▲   streamed chunks     │ messages list   │      --provider flag

   │                       │ + system prompt ▼

   │                       │           apidemo.llm.resolve()

   │                       │                 │

   │                       ▼                 ▼

   │                 ask() / stream() ──┬── claude-sdk ──▶ anthropic SDK ─┐

   │                                    ├── anthropic ─┐                  │

   │                                    ├── openai ────┼─ LiteLLM ──▶ ... ├─▶ providers

   │                                    ├── gemini ────┤                  │

   │                                    └── ollama ────┘ ──▶ localhost    │

   │                                                                      │

   └──────────── text deltas ◀── SSE ◀──────── stop_reason, usage ◀───────┘

                                                      │

TerpGrader batch ──▶ with_retries(backoff+jitter) ──▶ CostMeter.record(usage)

                     structured outputs (schema)       (budget guard)

Keys enter once, from the environment. The conversation lives in your code and is resent every call. The system prompt sets behavior; structured outputs guarantee format. Every response feeds the meter. Every call is wrapped in retries that respect the error taxonomy. And the provider is a configuration value, not a code change.

18.16 Live Demos🔗

All demos run from the project folder after the setup below. Each takes under a minute. Demos 0 and 6 (with Ollama) and the test suite cost nothing; the rest make a few cheap API calls.

Command

Concept

What to watch for

python demos/00_check_env.py

1

which key came from the shell and which from .env; all masked

python demos/01_first_request.py

2

the full response object: content blocks, stop_reason, usage

python demos/02_raw_http.py –break-key

2, 8

the same call as plain HTTP; a real 401 error body

python demos/03_multi_turn.py

3

"another sentence" fails without history, works with it

python demos/04_chat_loop.py

3

input tokens grow each turn; /reset erases memory

python demos/05_system_prompt.py

4

the same question answered vs. tutored

python demos/06_temperature.py -p ollama

5

identical lines at 0.0, different ones at 1.0

python demos/07_streaming.py race

6

time to first token vs. total time

python demos/08_structured_output.py

7

fenced JSON; a stop sequence keeps just the score but empties fenced JSON; validated object

python demos/09_errors_and_retries.py

8

typed errors; max_tokens is an HTTP 200; backoff log

python demos/10_cost_meter.py –budget 0.005

9

the budget guard stops the batch

python demos/11_switch_providers.py –compare

10

one prompt, every provider: latency, tokens, cost

pytest -q

11

45 tests pass with no keys and no network

18.17 GitHub-Ready Project: claude-api-demos🔗

The companion project lives in notes/ai-api/code/claude-api-demos/ and can be dropped directly into a GitHub repository.

claude-api-demos/

├── README.md

├── pyproject.toml           # package metadata, pytest config, `llm` console script

├── requirements.txt         # anthropic, litellm, python-dotenv, httpx, pydantic, pytest

├── .gitignore               # .env is ignored; .env.example is not

├── .env.example             # every provider's variables, with placeholders

├── apidemo/

│   ├── __init__.py

│   ├── config.py            # load_env (no override), require, mask

│   ├── chat.py              # add_user_message, add_assistant_message, chat, text_of

│   ├── retry.py             # is_retryable taxonomy, with_retries (backoff + jitter)

│   ├── cost.py              # pricing table, cost_of, CostMeter, BudgetExceeded

│   └── llm.py               # provider switch: LiteLLM + anthropic SDK + Agent SDK

├── demos/                   # 00_check_env.py ... 11_switch_providers.py

└── tests/                   # all offline

    ├── test_config.py       # .env precedence, masking, missing keys

    ├── test_chat.py         # helpers with a fake client

    ├── test_retry.py        # taxonomy, retry-after, jitter, give-up

    ├── test_cost.py         # pricing math, cache rates, budget guard

    └── test_llm.py          # provider resolution, mock_response, system-prompt routing

Quick start:

cd notes/ai-api/code/claude-api-demos
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -r requirements.txt
pip install -e .                                      # makes `apidemo` importable
pytest -q                                             # offline -- no key needed

cp .env.example .env                                  # then paste your keys into .env
python demos/00_check_env.py                          # verify (masked)
python demos/01_first_request.py
python -m apidemo.llm --provider ollama "Explain the TCP handshake in one sentence."

Reading order: config.py → chat.py → retry.py → cost.py → llm.py. Each file maps to a section of these notes.

18.18 Practice Exercises🔗

18.18.1 Basic🔗
  1. Write the curl command that asks claude-haiku-5-5 for a haiku about College Park with at most 200 output tokens, reading the key from the environment. Label which part is authentication, which is versioning, and which is the payload.

  2. Your teammate commits a .env file to a public repository, notices an hour later, and deletes it in the next commit. List everything that must happen now, in order, and the one change to the repository that would have prevented it.

  3. A response comes back with stop_reason: "max_tokens". What happened, why is it a problem for TerpGrader specifically, and what are two different fixes?

  4. Explain, with a two-message example, why “Write another sentence” produces an unrelated sentence when sent alone. Then write the messages list that fixes it.

  5. Compute the cost of a single TerpTutor turn on Claude Sonnet 5.5 with 6,000 input tokens and 400 output tokens. Which side dominates, and what (from Core Concept 3) controls it?

18.18.2 Intermediate🔗
  1. Extend demos/04_chat_loop.py with a /system <text> command that changes the system prompt mid-conversation, and a /cost command that prints the running total from a CostMeter. Use streaming for the replies.

  2. Rewrite TerpGrader (Demo 10) to use structured outputs with a Verdict Pydantic model (score: int, feedback: str) instead of json.loads. Write an offline test with a fake client that checks malformed input is never parsed by hand.

  3. Add a fallback function to apidemo/llm.py: ask_with_fallback(messages, providers=["claude-sdk", "openai", "ollama"]) tries each in order, moving on only for retryable failures and missing keys, and returns the first successful Reply. Test it offline with mock_response and a provider whose key is missing.

  4. Add a count_tokens pre-flight check to TerpGrader using the SDK’s client.messages.count_tokens(...): refuse to send any request whose estimated input cost would exceed a per-request cap.

  5. Run Demo 6 on two different providers and write a one-paragraph comparison: at temperature 0.0, are the outputs identical across runs? Across providers? What does this tell you about using temperature 0 for reproducible grading?

18.18.3 Advanced🔗
  1. Implement a token-bucket throttle in front of the TerpGrader loop that enforces both RPM and ITPM limits client-side. Show with fake-clock tests that a 500-request burst never exceeds either budget in any 60-second window.

  2. Build a small evaluation harness on top of apidemo.llm: grade 20 submissions with every configured provider, compare each provider’s scores to a TA’s scores, and report agreement, cost, and latency per provider. Recommend a provider and justify it with the numbers.

  3. Design a multi-tenant budget system for TerpTutor: per-user daily budgets, a global monthly cap, and graceful degradation (switch to a cheaper provider at 80% of a user’s budget, refuse politely at 100%). What race conditions arise with concurrent requests against a shared budget, and how do you make the guard atomic?

18.19 Summary🔗

18.19.1 Key takeaways🔗
  • Keys are money. Store them in environment variables or a git-ignored .env loaded with python-dotenv; commit .env.example, never .env; keep keys on the server; rotate on any leak.

  • An LLM API call is ordinary HTTP: headers for auth and versioning, a JSON body naming model, max_tokens, and messages, and a JSON response whose usage object is the billing ground truth. Read the text by joining text blocks, not content[0].

  • The API is stateless: your code keeps the transcript and resends it every turn—and pays for it every turn.

  • A system prompt sets the role and behavior for the whole conversation; pass it only when set, never as None.

  • Temperature reshapes the next-token distribution: low for consistency, high for variety. Current Claude models no longer accept it; other providers still do.

  • Streaming costs the same and feels far faster; use text_stream for display and get_final_message() for storage.

  • For machine-readable output use structured outputs; stop sequences only cut output at a marker.

  • Classify every failure as retryable or not; back off exponentially with jitter; honor retry-after; give up eventually. Check stop_reason even on a 200.

  • Cost is an engineering variable: model tier, prompt size, history, caching, batching—and a budget guard turns invoices into exceptions.

  • LiteLLM puts many providers behind one call; with all keys in one .env and a –provider flag, the provider becomes configuration. The calls are interchangeable; the models’ behavior and features are not.

18.19.2 Terminology🔗

Term

Meaning

API key

secret credential identifying the paying account

environment variable

named string the OS passes to a process; where keys live

.env file

git-ignored NAME=value file loaded into the environment by python-dotenv

anthropic-version

header pinning the wire-format version

stateless

the server keeps no conversation; the client resends history

system prompt

instructions in the separate system field that set role and behavior

temperature

sampling setting that sharpens or flattens next-token probabilities

stop sequence

string that ends generation when produced

structured outputs

responses constrained to and validated against a JSON schema

usage

per-response token counts --- the billing record

stop_reason

why generation ended (end_turn, max_tokens, stop_sequence, refusal, ...)

SSE

server-sent events; the streaming wire format

TTFT

time to first token --- perceived latency

RPM / ITPM / OTPM

rate-limit dimensions: requests / input tokens / output tokens per minute

backoff + jitter

retry delays that grow and are randomized

MTok

one million tokens --- the pricing unit

budget guard

code that raises before spend exceeds a cap

LiteLLM

library exposing one OpenAI-style call for 100+ providers

provider prefix

the openai/, gemini/, anthropic/, ollama_chat/ part of a LiteLLM model string

18.19.3 Common mistakes🔗
  1. Committing .env, or adding it to .gitignore only after git already tracks it.

  2. Calling Anthropic() before load_dotenv().

  3. Reading message.content[0].text on a model that returns a thinking block first.

  4. Forgetting to append the assistant’s reply to the history, so the next turn loses context.

  5. Passing system=None or stop_sequences=None instead of omitting them.

  6. Copying notebook code that sends temperature to a current Claude model, and getting an error.

  7. Retrying non-retryable errors; retrying without jitter.

  8. Ignoring stop_reason and parsing truncated output.

  9. Calling the provider directly from browser or mobile code.

  10. Assuming a prompt tuned for one provider behaves the same on another after switching with LiteLLM.

18.19.4 Where this leads🔗
  • Prompt engineering and evaluation: the system prompts and structured outputs here are the raw material; the evaluation lecture answers “is this prompt, model, or provider good enough?”

  • Tool use and agents: every agent loop multiplies today’s concerns—one user turn may be dozens of API calls, so retries, budgets, and caching stop being optional.

  • RAG: retrieval inflates input tokens; cost-aware chunking and caching decide feasibility, and a local model may be required for private documents.

  • MCP: auth and rate limiting for your tools, now that you have seen the provider’s side.

  • Deployment: secret managers, the backend proxy, and per-user budgets are where these patterns meet cloud deployment.