On this page:
13.1 Learning Objectives
13.2 Why Look Under the Hood?
13.3 The Running Example:   Terp  LM
13.4 Stage 1:   Pre-training
13.4.1 Step one:   download the internet
13.4.2 Tokenization:   what the model actually reads
13.4.3 The neural network:   input, output, and training
13.4.4 Inference:   sampling, stochasticity, and temperature
13.5 The Base Model:   a Token Simulator
13.6 Stage 2:   Supervised Fine-Tuning—Programming by Example
13.6.1 Post-training Terp  LM
13.7 LLM Psychology:   Predictable Failure Modes
13.7.1 Hallucinations, and the two kinds of memory
13.7.2 Models need tokens to think
13.8 Stage 3:   Reinforcement Learning—Practice Problems
13.8.1 RLHF:   reinforcement learning when you can’t check the answer
13.9 Grand Summary:   What Are You Talking To?
13.10 Trying These Systems Yourself
13.11 The Terp  LM Project
13.12 Practice Exercises
13.12.1 Basic
13.12.2 Intermediate
13.12.3 Advanced
13.13 Summary
13.13.1 Key takeaways
13.13.2 Terminology
13.13.3 Common mistakes
13.13.4 Looking ahead
9.1

13 How LLMs Work: From Internet Text to ChatGPT🔗

    13.1 Learning Objectives

    13.2 Why Look Under the Hood?

    13.3 The Running Example: TerpLM

    13.4 Stage 1: Pre-training

      13.4.1 Step one: download the internet

      13.4.2 Tokenization: what the model actually reads

      13.4.3 The neural network: input, output, and training

      13.4.4 Inference: sampling, stochasticity, and temperature

    13.5 The Base Model: a Token Simulator

    13.6 Stage 2: Supervised Fine-Tuning—Programming by Example

      13.6.1 Post-training TerpLM

    13.7 LLM Psychology: Predictable Failure Modes

      13.7.1 Hallucinations, and the two kinds of memory

      13.7.2 Models need tokens to think

    13.8 Stage 3: Reinforcement Learning—Practice Problems

      13.8.1 RLHF: reinforcement learning when you can’t check the answer

    13.9 Grand Summary: What Are You Talking To?

    13.10 Trying These Systems Yourself

    13.11 The TerpLM Project

    13.12 Practice Exercises

      13.12.1 Basic

      13.12.2 Intermediate

      13.12.3 Advanced

    13.13 Summary

      13.13.1 Key takeaways

      13.13.2 Terminology

      13.13.3 Common mistakes

      13.13.4 Looking ahead

These notes follow the arc of Andrej Karpathy’s lecture Deep Dive into LLMs like ChatGPT (YouTube, 3.5 hours), adapted for this course and paired with code you can run. A good short companion is Anfal Mushtaq’s TL;DR write-up of the same lecture.1These notes compress the lecture, add a runnable miniature implementation (TerpLM), and connect the material to the rest of this course. Mushtaq’s write-up also hosts a copy of Karpathy’s complete excalidraw diagram (Karpathy’s own sharing link expires). Several figures in this chapter are Karpathy’s diagrams, reproduced via that write-up.

13.1 Learning Objectives🔗

By the end of this lecture you should be able to:

  1. Describe the three training stages of a modern LLM— pre-training, supervised fine-tuning, and reinforcement learning— including what data each consumes, what it produces, and roughly what it costs.

  2. Explain byte-pair encoding: why models read tokens rather than characters, and predict the practical consequences (case sensitivity, spelling failures, counting failures).

  3. State what a language model fundamentally computes (a probability distribution over the next token), what its parameters are, what the loss measures, and what changes during training versus during inference.

  4. Explain why a base model is an “internet document simulator,” and why the assistant you talk to is best understood as a simulation of a human data labeler following a company’s instructions.

  5. Use the distinction between parameters (vague recollection) and the context window (working memory) to predict when a model will hallucinate and how tool use mitigates it.

  6. Explain “models need tokens to think”: why a fixed amount of computation per token makes chains of intermediate steps necessary, and why counting and spelling fail.

  7. Distinguish supervised fine-tuning from reinforcement learning from RLHF; explain what is new about “thinking” models like DeepSeek-R1 and o1/o3, and why RLHF cannot be run indefinitely.

  8. Train, sample from, and “post-train” a complete miniature language model (tokenizer, parameters, loss, temperature) on your laptop.

13.2 Why Look Under the Hood?🔗

In Introduction: This Is Not an AI Class we said the central skill of this course is judgment over AI output, and in Software Testing we built the machinery for verifying it. This lecture supplies the missing piece: a mechanical understanding of the thing being judged. When you know what is actually behind the ChatGPT text box—what happens to your words when you press enter, where the reply comes from, and which parts of the pipeline put it there—the model’s strengths and its famous failures (hallucinated APIs, wrong arithmetic, miscounted letters, confident nonsense) stop being mysteries and become predictable consequences of how the system is built. Engineers who can predict failure modes prompt better, verify smarter, and design better systems around the model.

The plan mirrors how the systems are actually built, in three sequential stages plus the psychology that falls out of them:

  internet text          conversations           practice problems

       │                       │                        │

       ▼                       ▼                        ▼

┌──────────────┐      ┌──────────────────┐      ┌──────────────────┐

│ PRE-TRAINING │ ───► │ SUPERVISED       │ ───► │ REINFORCEMENT    │

│ (months,     │      │ FINE-TUNING      │      │ LEARNING         │

│  $millions)  │      │ (hours--days)    │      │ (the new bit)    │

└──────────────┘      └──────────────────┘      └──────────────────┘

       │                       │                        │

       ▼                       ▼                        ▼

  base model              assistant              "thinking" model

(internet document     (simulates a human       (discovers its own

   simulator)             data labeler)          reasoning steps)

13.3 The Running Example: TerpLM🔗

Reading about LLMs is one thing; training one is better. Alongside the real systems, we will build TerpLM, a complete language model pipeline small enough to train on a laptop in two minutes: a real byte-pair-encoding tokenizer, a gradient-trained next-token predictor, sampling-based inference with temperature, and even a “post-training” stage. The project lives in the course repository under notes/llm/code/terplm.

TerpLM’s neural network is deliberately the dumbest one that still works—a bigram model that predicts the next token from only the single previous token. Every concept in this lecture (tokens, parameters, loss, sampling, temperature, chat templates) appears in it faithfully; the only things missing are the transformer architecture that lets real models use thousands of tokens of context, and about eleven orders of magnitude of scale. Keeping the model dumb keeps the pipeline transparent, and its failures will turn out to be as instructive as its successes.

13.4 Stage 1: Pre-training🔗

13.4.1 Step one: download the internet🔗

Building an LLM starts with text—a lot of it, of high quality and high diversity, because everything the model will ever “know” comes from this corpus. A representative public example is Hugging Face’s FineWeb dataset; every major lab (OpenAI, Anthropic, Google) maintains an internal equivalent. The pipeline typically starts from Common Crawl (an organization that has been crawling the web since 2007; on the order of billions of pages) and then filters aggressively:

  1. URL filtering — drop blocklisted domains: malware, spam, marketing, adult content, and so on.

  2. Text extraction — crawled pages are raw HTML; strip the markup, navigation, and boilerplate, keeping just the text.

  3. Language filtering — e.g. FineWeb keeps pages that are predominantly English. A design choice with visible consequences: filter out Spanish and your model will be bad at Spanish.

  4. Quality filters — heuristic rules from the Gopher and C4 pipelines that drop boilerplate, junk, and degenerate pages.

  5. Deduplication — drop near-identical pages (FineWeb uses MinHash) so the model does not over-train on repeats.

  6. PII removal — filter pages containing personal data such as addresses and social security numbers.

The FineWeb filtering pipeline, from raw crawl to training text. Diagram: Andrej Karpathy, Deep Dive into LLMs like ChatGPT.

The punchline is how small the result is: FineWeb is about 44 terabytes—roughly one hard drive—of text, around 15 trillion tokens. The “knowledge of the internet” that ends up inside a frontier model passes through a single disk.

13.4.2 Tokenization: what the model actually reads🔗

Formal definition. A tokenizer is an invertible mapping between text and sequences of integers (tokens) drawn from a fixed, finite vocabulary. Modern LLMs learn the vocabulary with byte-pair encoding (BPE): start from the 256 possible bytes, repeatedly find the most frequent adjacent pair of symbols in the corpus, and “mint” a new symbol for that pair, until the vocabulary reaches a target size (GPT-4 uses 100,277 symbols).

In plain English: the model never sees letters or words. It sees chunk IDs. “hello world” is not eleven characters to GPT-4; it is two opaque integers.

Intuition. Neural networks want a one-dimensional sequence of symbols from a finite set, and sequence length is the scarce resource. Raw bits would give 2 symbols and absurdly long sequences; bytes give 256 symbols and sequences 8 times shorter; BPE keeps trading a bigger vocabulary for shorter sequences, settling around 100k symbols at roughly 4 characters per token of English.

Common misconception. “Tokens are words.” They are not: common words get one token, rare words get several, whitespace and capitalization change everything, and the same string tokenizes differently depending on a leading space. Sites like tiktokenizer let you see GPT-4’s actual tokenization interactively; you will be surprised.

TerpLM contains a real BPE trainer in about sixty lines (terplm/bpe.py). Training 60 merges on a small corpus about the university and then encoding a sentence:

first 6 merges learned:

  (  104,  101) -> 256   meaning 'h' + 'e' = 'he'

  (  116,  256) -> 257   meaning 't' + 'he' = 'the'

  (  116,  101) -> 258   meaning 't' + 'e' = 'te'

  (  114,   97) -> 259   meaning 'r' + 'a' = 'ra'

  (  116,  117) -> 260   meaning 't' + 'u' = 'tu'

  (  105,  110) -> 261   meaning 'i' + 'n' = 'in'

 

encode('the terrapin tested the program')

  -> [257, 32, 275, 32, 299, 258, 100, 32, 257, 32, 287, 109]

  -> [the][ ][terrapin][ ][tes][te][d][ ][the][ ][progra][m]

  31 bytes -> 12 tokens

The merges are readable: the algorithm discovered he, then the, exactly as you would expect from English statistics, and “terrapin”—frequent in this corpus—became a single token. And case sensitivity falls straight out of the mechanism:

 'Maryland' -> [Maryland]

 'maryland' -> [m][a][ry][l][and]

' Maryland' -> [ ][Maryland]

The capitalized form was frequent in the training corpus and earned a single token; the lowercase form shatters into five. To the model these are completely unrelated symbols. Hold that thought for the section on spelling failures.

Practical implications. Tokenization is a preprocessing step done once; after it, “the internet” is one enormous one-dimensional sequence of integers, and everything downstream— training, inference, conversations, even images and audio in multimodal models—is sequences of tokens.

13.4.3 The neural network: input, output, and training🔗

Formal definition. A language model is a function, parameterized by a vector of numbers (the parameters or weights), that maps a sequence of tokens (the context, up to a maximum context length—1,024 tokens for GPT-2, 128,000 and beyond for current frontier models) to a probability distribution over the vocabulary for the next token. Training adjusts the parameters so that, across the corpus, the actual next token receives high probability; the loss (cross-entropy) measures how far the predictions are from the data, and lower is better.

In plain English: the model is a guess-the-next-token machine, and training is millions of rounds of “here are the previous tokens—what comes next?” with a nudge to the knobs after every round.

One training example: a window of four tokens goes in; the network outputs a probability for each of the 100,277 possible next tokens; training nudges the probability of the actual next token (3962) upward. Diagram: Andrej Karpathy.

Intuition. Think of the parameters as knobs on a giant DJ mixing board (GPT-2 had 1.6 billion knobs; modern models hundreds of billions). At random settings the machine outputs noise. Training is an automated procedure that twiddles the knobs, a tiny step at a time, until the machine’s output statistics match the corpus statistics. Inside, the “machine” is just one big differentiable mathematical expression—matrix multiplications and simple nonlinearities arranged in the transformer architecture. The internals matter less than you would think; what matters is: fixed expression, billions of knobs, input tokens, output probabilities.

Common misconception. “The model stores its training text in a database and looks things up.” There is no database and no lookup. Whatever the model “knows” is smeared across the parameters as statistical tendencies—which is why its knowledge is a vague recollection, not a record, and why it can be wrong with perfect confidence.

Watching the loss fall is the daily life of an LLM researcher, so let’s do it. TerpLM’s model (terplm/model.py) is one matrix W: row t holds the logits—pre-softmax scores—for “what follows token t.” Its training step is the whole of deep learning in five lines:

def train_step(self, prev: np.ndarray, nxt: np.ndarray, lr: float) -> float:
    """One gradient-descent update on a batch of (prev, next) pairs."""
    probs = _softmax(self.W[prev])  # (batch, vocab)
    batch = len(prev)
    loss = float(-np.log(probs[np.arange(batch), nxt] + 1e-12).mean())
    # Gradient of cross-entropy through softmax: probs - one_hot.
    probs[np.arange(batch), nxt] -= 1.0
    np.add.at(self.W, prev, -lr * probs / batch)
    return loss

We train it on the classic “tiny shakespeare” corpus (1.1 MB of public-domain text), with a 514-token vocabulary. The run prints the loss and a 12-token sample every 250 steps:

$ bash scripts/fetch_shakespeare.sh
$ python -m terplm.train data/shakespeare.txt --merges 256 --steps 3000
training tokenizer: 256 merges ...
corpus: 1,115,394 chars -> 691,315 tokens (vocab 514, 1.61 bytes/token)
loss of uniform guessing: ln(514) = 6.242

step     0  loss 6.242  sample: 'Fet?\x15\x08Butmorekesuhaciath\x01'
step   500  loss 5.202  sample: 'Fcesheght??ight?what?love'
step  1000  loss 4.803  sample: 'Fonce?hand vomitilgo~'
step  1500  loss 4.541  sample: 'F\x1e, be dehovustheeis '
step  2000  loss 4.343  sample: 'FndIN\x0eanhis then?you a?'
step  3000  loss 4.113  sample: "F?c?Yliindup?'t"

saved model + tokenizer to out/

Read this the way a researcher does. At step 0 the parameters are random, the loss sits exactly at ln(514)—the loss of pure uniform guessing—and the sample is noise. As updates accumulate, the loss falls and samples develop local coherence: real word fragments, plausible spacing, apostrophes in sensible places. It never gets further than that, and this is the punchline of the bigram model: with one token of context, local texture is all you can learn. A transformer running the same loop with thousands of tokens of context—and a few trillion more parameters—is what turns this gibberish into fluent paragraphs. GPT-2 (2019) was exactly this recipe at 1.6 billion parameters and 100 billion training tokens; it cost about $40,000 to train then, and Karpathy’s llm.c reproduction brought that to roughly $672—headed toward $100— which tells you how much of the recent revolution is better data, better software, and above all better hardware (the GPU data centers driving the current “gold rush”) rather than new ideas.

13.4.4 Inference: sampling, stochasticity, and temperature🔗

Formal definition. Inference generates text from a trained model: feed in a token prefix, obtain the probability distribution for the next token, sample one token from that distribution, append it, and repeat. Temperature rescales the logits before the softmax; low temperature sharpens the distribution toward the most likely token, high temperature flattens it.

In plain English: the model does not “decide” on a reply; it flips a biased coin per token, forever. When you use ChatGPT, no learning happens—the parameters were frozen months ago; you are just sampling.

The inference loop: each sampled token is appended and fed back in. Note the last row: the model samples 13659 where the training document had 3962—generation is a remix of training statistics, not a recital. Diagram: Andrej Karpathy.

The consequences are immediately visible in TerpLM. The same prompt, two seeds:

$ python -m terplm.generate --prompt "ROMEO:" --tokens 40 --seed 1
ROMEO:?E:/I'll??thee?veruto thy to or say andlovespeconhe festis fornoseilme O
$ python -m terplm.generate --prompt "ROMEO:" --tokens 40 --seed 2
ROMEO:a will my h?ly,
Uall ascilo??orWhatmor?ARD?ver bjOLloile, you sh?

Different every time: these are stochastic systems, remixing the statistics of their training data rather than reciting it. Now temperature, taken to the extreme:

$ python -m terplm.generate --prompt "ROMEO:" --temperature 0.4
ROMEO:
Th?I'll the the I the of the the s s are he s so bar to I the

$ python -m terplm.generate --prompt "ROMEO:" --temperature 0.05
ROMEO:
Whow the the the the the the the the the the the the the

At temperature 0.05 the model almost always takes the single most probable next token, and in English bigram statistics the most probable follower of almost anything is the—so it falls into the the the the attractor. Real systems balance the same trade-off: low temperature for focused, deterministic-ish output (good for code), higher for creative diversity. (File away the phrase “the the the”; it makes a sinister reappearance in the RLHF section.)

13.5 The Base Model: a Token Simulator🔗

What pre-training produces is called a base model, and it is not an assistant.

Formal definition. A base model is the network plus its trained parameters, with no further conditioning: a sampler of token sequences whose statistics match its training corpus.

In plain English: it is an internet document simulator. Prompt it with “What is 2+2?” and it will not answer you—it will continue the document, perhaps with more exam questions, perhaps with a philosophy essay, differently each time. It is, in Karpathy’s phrase, a glorified, very expensive autocomplete.

Intuition. The parameters are a lossy compression of the internet—a zip file of statistical gist, hundreds of billions of numbers distilled from trillions of tokens. Sampling from the base model is “dreaming internet pages”: statistically faithful, factually unanchored remixes.

Releasing a model, by the way, means publishing exactly two artifacts: the inference code—a few hundred lines describing the forward pass (GPT-2’s fits in one Python file)—and the weights, the billions of tuned numbers, which are where all the value lives. Releases like GPT-2 (2019) or Meta’s Llama 3.1 405B are called open weights rather than open source: you get the parameters, but typically not the training data or the full recipe to reproduce them.

Common misconception. “Useless, then.” Far from it: all of the knowledge lives here, and three behaviors of base models explain most of what assistants later do:

  • Regurgitation. Prompt a strong base model with the first sentence of a famous Wikipedia article and it may recite the continuation nearly verbatim—high-quality sources are deliberately oversampled in training (the model has effectively “read” Wikipedia ten times), and like a student who has reread a text ten times, it can recite. Verbatim recitation is usually undesirable—and it should temper how you think about both the privacy and the copyright of training data.

  • Educated hallucination. Prompt it with text about events after its knowledge cutoff (the end date of its training corpus) and it confidently continues with a plausible parallel universe—naming the wrong running mate, the wrong opponent. Resample and you get a different parallel universe. Nothing is malfunctioning: it is doing exactly what it was trained to do, continue documents plausibly.

  • In-context learning. Give it ten English:Korean word pairs and an eleventh English word, and it completes the pattern with a correct translation. The pattern was learned inside the context window at inference time, not during training. This emergent ability—few-shot prompting—was the bridge that made base models practically useful, and you can even conjure a passable assistant from a pure base model by formatting your prompt as a transcript of a helpful-assistant conversation and letting the model continue it.

That last trick—the assistant exists in the prompt, not the weights—is the perfect mental bridge to stage two, which bakes the trick into the parameters.

13.6 Stage 2: Supervised Fine-Tuning—Programming by Example🔗

Formal definition. Supervised fine-tuning (SFT) continues training the base model with the same algorithm and the same objective—next-token prediction—on a different dataset: a curated corpus of multi-turn conversations between a “user” and an “assistant,” written to exemplify ideal assistant behavior.

In plain English: you cannot program a neural network’s behavior in code, so you program it by example. Want it to answer questions helpfully, decline harmful requests, admit uncertainty? Write a few hundred thousand conversations that do exactly that, and train on them. The model rapidly adopts the persona.

Intuition. Pre-training takes months on thousands of GPUs; SFT can take hours, because the conversation corpus is minuscule next to the internet. The base model already contains the knowledge; SFT teaches it a style of access.

Conversations must become token sequences like everything else, so each lab defines a chat template with new special tokens—reserved IDs that never appear in pre-training text. In the ChatML-style format used by OpenAI, one training conversation serializes roughly as:

<|im_start|>system<|im_sep|>You are a helpful assistant<|im_end|>

<|im_start|>user<|im_sep|>What is 4 + 4?<|im_end|>

<|im_start|>assistant<|im_sep|>4 + 4 = 8<|im_end|>

The system turn carries standing instructions (identity, tone, rules)—silently prepended to every conversation you have; we will meet it again under “knowledge of self.” At inference time the server appends <|im_start|>assistant<|im_sep|> after your question and simply asks the model to continue the sequence: the assistant’s reply is the continuation. When you chat with any LLM, this serialization is happening underneath every turn; the structured “conversation” you see is a UI fiction over one one-dimensional token stream. (The terms chat template and ChatML are worth remembering: every fine-tuning framework and chat API speaks them.)

Where do the conversations come from? This is the part everyone should know: from people. The InstructGPT paper (OpenAI, 2022) described hiring human labelers and handing them labeling instructions—company-written documents, often hundreds of pages, boiling down to “be helpful, truthful, and harmless”—under which the labelers write prompts and ideal responses. Today most of the volume is generated synthetically by existing LLMs and then human-edited, but the seed of the behavior is still human curation under corporate instructions. Both generations of dataset are public and browsable: OpenAssistant’s OASST1 is human-written by volunteers; UltraChat is largely synthetic, with millions of conversations.

Common misconception—and the single most useful reframe in this lecture. “I am talking to a magical AI.” What you are talking to is a statistical simulation of a data labeler following their employer’s labeling instructions. Ask ChatGPT for the top five landmarks in Paris and you are getting, in effect, an instantaneous imitation of the answer a (skilled, well-instructed) contractor would have written after twenty minutes of research— either because a very similar conversation is in the SFT set, or as an emergent blend of the labeler persona with pre-training knowledge. This reframe also explains the model’s self-knowledge: ask a model who built it, and unless its makers hardcoded identity conversations into the SFT data (some labs do, with a couple hundred examples) or put the identity in the hidden system message that silently precedes your conversation, it will guess—often answering “I am ChatGPT by OpenAI” simply because the internet is full of such answers. It has no persistent self; it boots up, processes tokens, and shuts down.

13.6.1 Post-training TerpLM🔗

The claim “SFT is the same algorithm with a different dataset” sounds too simple to be true, so TerpLM demonstrates it literally. Our train.py has a –resume flag that loads the Shakespeare-trained model and continues training on a tiny corpus of conversations in the <|im_start|> protocol (the special tokens are single IDs in TerpLM’s vocabulary, exactly as in production):

$ python -m terplm.train data/chat_corpus.txt --resume --steps 2000
resuming: swapping dataset to data/chat_corpus.txt (same algorithm)
corpus: 1,754 chars -> 850 tokens (vocab 514, 2.06 bytes/token)

step     0  loss 4.982  sample: '<|im_start|>et I\x15qumoreutbut4ciBut\x01'
step   500  loss 1.445  sample: '<|im_start|>user\nWhat con?what?love'
step  1000  loss 1.140  sample: '<|im_start|>user\nI builtant\nI'
step  2000  loss 1.064  sample: '<|im_start|>user\nWhat land.<|im_end|>\nI '

$ python -m terplm.generate --prompt "<|im_start|>user" --tokens 45
<|im_start|>user
What you the buame?<|im_end|>
<|im_start|>user
<|im_start|>assity modeland.<|im_end|>
I from you your builtant

Look closely at what two thousand updates on 850 tokens bought. The model has visibly learned the protocol: turns begin with <|im_start|>, end with <|im_end|>, roles appear after the start marker, questions trail question marks. The content is word salad—of course it is; the model conditions on one previous token and cannot remember the question while producing the answer. This is the cleanest demonstration in the whole project of what SFT does (impose conversational form on a pre-trained statistical engine) and of why context length matters (form is bigram-learnable; substance is not).

13.7 LLM Psychology: Predictable Failure Modes🔗

Everything in this section is a consequence of the pipeline above, derivable rather than memorizable.

13.7.1 Hallucinations, and the two kinds of memory🔗

Formal definition. A hallucination is fluent, confident output that is factually false—the model fabricating information in the style of knowing it.

Why does it happen? Look at the SFT data. Training conversations of the form “Who is X?” are answered confidently and correctly, because the human labeler looked X up. The model learns the form: such questions get confident answers. Ask it about a person who does not exist and the statistically appropriate continuation is a confident biography—sampled from plausible nonsense. Older models did exactly this, every time, with a different fabrication per resample.

Two mitigations are now standard, and both are instructive:

  1. Teach the model to say “I don’t know”—but only where it truly doesn’t. Meta’s Llama 3 team probes the knowledge boundary empirically: generate factual questions from documents, interrogate the model several times per question, and use an LLM judge to compare its answers with the known correct one. Where the model reliably fails, add a training conversation whose correct answer is “I’m sorry, I don’t know.” The remarkable part: a modest number of such examples generalizes. The network plausibly already contains internal activations correlated with uncertainty; the new data merely wires that signal to words.

  2. Tools: let the model look it up. Introduce special tokens such as SEARCH_START/SEARCH_END. When the sampler sees the model emit a search query between them, it pauses generation, runs the query against a real search engine, pastes the retrieved text into the context window, and resumes sampling. At the token level the whole exchange looks like this:

    <|im_start|>user<|im_sep|>Who is Orson Kovacs?<|im_end|>

    <|im_start|>assistant<|im_sep|><SEARCH_START>Who is Orson

    Kovacs?<SEARCH_END><|im_end|>

     

          [...search results pasted into the context window...]

     

    <|im_start|>assistant<|im_sep|>Orson Kovacs is ...<|im_end|>

    A few thousand SFT examples teach the model when and how to reach for the tool. The citations you see in modern chat interfaces are exactly this mechanism.

The second mitigation works because of a distinction worth engraving somewhere:

knowledge in the PARAMETERS  =  vague recollection

  (read it months ago, statistically compressed)

 

knowledge in the CONTEXT WINDOW  =  working memory

  (directly attended to on every forward pass)

This one distinction is the most practically useful fact in the lecture. “Summarize chapter 1 of Pride and Prejudice” works passably from recollection because the internet has discussed that chapter to death; “summarize chapter 1, which I am pasting below” works far better, for the same reason you would summarize better having just reread the chapter. This is the entire idea of retrieval-augmented generation (RAG), which we will meet later in the course: when you build LLM applications, the game is getting the right information into the context window instead of hoping it is in the parameters.

13.7.2 Models need tokens to think🔗

Here is the deepest of the practical insights. Recall the architecture: producing each token costs one forward pass through a network with a fixed number of layers. Per token, the model performs a fixed, finite amount of computation— roughly a hundred layers’ worth, no more, no matter how hard the question.

Now suppose you are a data labeler writing the ideal response to: “Emily buys 3 apples and 2 oranges. Each orange costs $2. The total is $13. What does an apple cost?” Two candidate responses:

BAD:   "The answer is $3.  We can see this because

        13 - 4 = 9 and 9 / 3 = 3."

 

GOOD:  "The oranges cost 2 × 2 = $4.  So the apples

        cost 13 - 4 = $9.  There are 3 apples, so

        each apple costs 9 / 3 = $3.  The answer is $3."

Both are correct. The first is a terrible training label. Reading left to right, it asks the model to produce the token 3 immediately—cramming the entire solution into the fixed compute budget of one token—and everything after is post-hoc justification of an answer already emitted. The second spreads the work across many tokens, each requiring one easy step, with every intermediate result landing in the context window (working memory!) where later tokens can build on it. Train on answers of the first shape and you get a model that guesses; of the second shape, a model that computes. This is why chat models “think out loud”—those intermediate steps are not a presentation choice for you, they are computation for the model—and why prompting a model to answer a hard question “in one word” measurably degrades accuracy.

Two corollaries you can verify in any chat window today:

  • Counting fails. “How many dots: ........” asks for all the work in one token, on top of which the dots arrive bundled into opaque multi-dot tokens. The fix is delegation: ask the model to use code—copy-pasting the dots into a Python string is easy (it is just echoing tokens), and .count(".") does the counting with an interpreter instead of neural mental arithmetic.

  • Spelling fails. The famous “how many r’s in strawberry?” The model does not see r’s; it sees one or two token IDs for the whole word (remember maryland → [m][a][ry][l][and] from our own tokenizer— characters are smeared unpredictably across symbols), and it must also count. Two known weaknesses, multiplied. Again: “use code” fixes it.

And one humbling note to file under jagged intelligence: models that solve olympiad problems have claimed that 9.11 is greater than 9.9 (one investigation found internal features associated with Bible verses activating—9:11 does follow 9:9). The capability surface is Swiss cheese: spectacular and full of holes, and the holes are not always where reasoning about the pipeline predicts. Hence the standing rule of this course: use the model as a tool, check its work, own the result.

13.8 Stage 3: Reinforcement Learning—Practice Problems🔗

The school analogy organizes everything so far:

In a textbook

Training stage

What the model gets

exposition / background prose

pre-training

knowledge

worked examples with solutions

supervised fine-tuning

imitation of experts

practice problems (answer key only)

reinforcement learning

skill discovered by trial and error

Formal definition. In reinforcement learning (RL) for LLMs, the model is given prompts with verifiable answers (math, code with tests), samples many candidate solutions per prompt—thousands, across tens of thousands of prompts—and is then trained on the solutions that reached correct answers, making such token sequences more probable.

In plain English: guess and check, at industrial scale, with “do more of what worked.”

One RL step on the very prompt from our “tokens to think” section: 15 sampled solutions, 4 reach the right answer (green), and the model trains on the best of them. Repeat across tens of thousands of prompts. Diagram: Andrej Karpathy.

Intuition—why imitation isn’t enough. When a human labeler writes a worked solution, they write token sequences that are easy for a human. But the model’s cognition is not ours: a step trivial for the labeler may demand an impossible leap of single-token computation from the model, while knowledge the model has in abundance goes unused. The ideal token sequence for the model to reason with is not something any human can write down—it has to be discovered by the model itself. SFT gets the model into the vicinity of competent solutions; RL dials in what actually works for its own cognition.

What emerges. The DeepSeek-R1 paper (2025) made the recipe public and showed two things happening as RL proceeds: accuracy on math problems climbs, and—unprogrammed by anyone—solutions get longer. The model starts writing things like “Wait, wait. That’s an aha moment. Let me re-evaluate this step by step,” trying alternative framings, backtracking, double-checking—an internal monologue, or chain of thought, that emerges because it empirically improves accuracy. No labeler wrote it; no labeler could have. This is what powers the “thinking” or “reasoning” models (DeepSeek-R1, OpenAI’s o1/o3, Gemini’s thinking variants): when the interface says “reasoning...”, the model is spending thousands of tokens of working-memory computation before answering. GPT-4o-class chat models, by contrast, are best thought of as SFT models with light RLHF polish.2Some interfaces hide the raw chain of thought and show summaries, partly over distillation concerns—a competitor could train on the visible reasoning traces and recover much of the capability.

The precedent. We have seen this movie: AlphaGo. The supervised version, trained to imitate expert human games, topped out just below the best humans—imitation cannot exceed its source. The RL version, playing against itself with only wins as the signal, blew past them, and produced move 37: a move human experts assigned probability 1-in-10,000, which proved brilliant.

From the AlphaGo paper (Silver et al., 2016): the supervised-learning version plateaus below top human strength (dashed line); the reinforcement-learning version climbs past it.

That is the promise of RL in open-ended reasoning— strategies no human would write in a training set—and the reason labs are racing to build ever-larger collections of verifiable “practice problems.” In LLM reasoning we have so far seen only the first hints of a move 37.

13.8.1 RLHF: reinforcement learning when you can’t check the answer🔗

Math has an answer key. “Write a joke about pelicans” does not.

Formal definition. Reinforcement learning from human feedback (RLHF) extends RL to unverifiable domains by training a separate reward model—another transformer that maps (prompt, candidate response) to a score—to imitate human preferences, gathered cheaply by asking humans to rank a handful of candidate responses rather than score or write them. The arithmetic is the whole argument: naive RL with humans in the loop—say 1,000 updates × 1,000 prompts × 1,000 rollouts—would need a billion human judgments of pelican jokes. RLHF instead collects about 5,000 rankings (1,000 prompts × 5 rollouts), trains the reward model on them, and then queries the simulator a billion times for free.

The RLHF indirection: replace a billion human judgments with five thousand, plus a learned simulator of the judge. Diagram: Andrej Karpathy.

Intuition—why ranking works. The discriminator–generator gap: for creative tasks it is far easier to judge which of five poems is best than to write the ideal poem. RLHF lets humans contribute the easy judgment and makes the model discover responses that would be judged well.

The catch, and it is fundamental: the reward model is a billion-parameter simulation of human preference, and RL is relentlessly good at finding inputs that fool it. Run RLHF long enough and the “best joke about pelicans” becomes—this is the canonical real example—the the the the the, an adversarial example to which the reward model assigns a top score for no humanly comprehensible reason. (Our miniature saw the same string emerge from low-temperature sampling; both pathologies are optimizers collapsing onto degenerate maxima.) Adding each discovered nonsense input to the training data with a low score just reveals infinitely many more. So in practice RLHF runs for a few hundred updates and is cropped before gaming sets in. Karpathy’s summary is the right one: RLHF is not RL in the magical sense—it is a worthwhile fine-tune, but unlike RL in verifiable domains (where the checker cannot be gamed: the answer is three or it isn’t, the game of Go is won or it isn’t), you cannot pour more compute into it and expect move 37. Verifiable domains scale; simulated judges saturate.

13.9 Grand Summary: What Are You Talking To?🔗

Trace your next ChatGPT query through everything above. Your text is wrapped in a chat protocol and tokenized into a one-dimensional sequence. A fixed mathematical expression with frozen parameters —knowledge smeared into them by months of next-token prediction on a filtered internet, persona imposed by fine-tuning on labeled conversations, polish added by RLHF—continues the sequence one sampled token at a time, each token costing one bounded forward pass. What comes back is best understood as:

  • For a standard chat model: a statistical simulation of a skilled data labeler following their employer’s instructions— instant, tireless, broadly knowledgeable, and capable of arbitrary holes.

  • For a thinking model: the same, plus reasoning strategies no labeler wrote—discovered by the model through millions of practice problems—unrolling in its working memory before it answers.

Neither is a person; neither has persistent existence between your messages; both are extraordinary tools. The operating rules this course keeps returning to all follow mechanically:

Rule of practice

Because

Put relevant material in the prompt rather than assuming the model knows it

context window = working memory; parameters = vague recollection

Let it reason in steps; never demand hard answers “in one word”

fixed compute per token

Push arithmetic, counting, and string-twiddling into tools/code

neural mental math over opaque tokens is unreliable

Reach for a thinking model on hard reasoning; a fast chat model otherwise

RL-trained deliberation costs time and money and is overkill for lookups

Verify everything that matters (see Software Testing)

stochastic Swiss-cheese capability, confident by construction

13.10 Trying These Systems Yourself🔗

Everything this chapter describes can be poked at directly, mostly for free:

  • Tokenizers. tiktokenizer shows GPT-4’s actual tokenization of anything you type—try the Maryland/maryland experiment, and switch the dropdown to a chat model to see the <|im_start|> serialization of a conversation.

  • Inside a transformer. bbycroft.net/llm is an interactive 3D visualization of a real (small) production architecture—the “giant mathematical expression” rendered walkable.

  • Base models. Hyperbolic serves Llama 3.1 405B base—one of the few places to experience a raw token simulator (try the regurgitation, parallel-universe, and few-shot demos from this chapter).

  • Open-weights chat and thinking models. together.ai hosts DeepSeek-R1, Llama, and friends; watching R1’s visible chain of thought on a math problem is the best free demonstration of the RL section.

  • Running models locally. LM Studio or Ollama run distilled, quantized open-weights models on a laptop—no data leaves your machine.

  • Keeping up. LM Arena ranks models by blind human preference votes (useful as a first pass, though increasingly gamed—try a few models on your tasks before trusting any leaderboard), and the AI News newsletter is a comprehensive, largely LLM-written digest of the field.

13.11 The TerpLM Project🔗

The complete miniature pipeline is in the course repository at notes/llm/code/terplm:

terplm/

├── terplm/

│   ├── bpe.py        # byte-pair-encoding tokenizer (train/encode/decode)

│   ├── model.py      # bigram LM: parameters, loss, gradient, sampling

│   ├── train.py      # training CLI ("post-training" via --resume)

│   └── generate.py   # inference CLI (prompt, temperature, seed)

├── data/

│   ├── tiny_corpus.txt   # bundled corpus (runs in seconds)

│   └── chat_corpus.txt   # conversations with <|im_start|> tokens

├── scripts/fetch_shakespeare.sh

└── tests/            # BPE round-trip property, distribution sanity

cd notes/llm/code/terplm
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pytest                                        # 11 tests

bash scripts/fetch_shakespeare.sh             # 1. "download the internet"
python -m terplm.train data/shakespeare.txt --merges 256 --steps 3000   # 2. pre-train
python -m terplm.generate --prompt "ROMEO:" --seed 1                    # 3. inference
python -m terplm.generate --prompt "ROMEO:" --temperature 0.05          #    temperature
python -m terplm.train data/chat_corpus.txt --resume --steps 2000       # 4. post-train
python -m terplm.generate --prompt "<|im_start|>user" --temperature 0.6 # 5. "assistant"

The test suite is a callback to Software Testing: the tokenizer’s central guarantee—decode(encode(t)) == t for any string—is checked as a Hypothesis round-trip property, and the model’s output is property-checked to be a genuine probability distribution.

13.12 Practice Exercises🔗

13.12.1 Basic🔗
  1. In your own words, why does an LLM see Maryland and maryland as unrelated symbols? Use TerpLM’s tokenizer output to illustrate.

  2. TerpLM’s untrained loss is exactly ln(514) = 6.242. Derive that number. What would it be for GPT-4’s vocabulary of 100,277?

  3. Explain the difference between what changes during training and what changes during inference. Which one is happening when you use ChatGPT?

  4. Classify each behavior as caused by (a) tokenization, (b) fixed compute per token, or (c) parameters-as-vague-recollection: miscounting the r’s in strawberry; inventing a biography for a fake name; failing “answer in one word” arithmetic.

  5. Why is the base model called a “token simulator”? Give two reasons it is still valuable without any post-training.

  6. Read a TL;DR summary of the Karpathy lecture (one is linked in the footnote at the top of this chapter). Find two places where the summary keeps the what but drops the why —for example, it may say step-by-step answers are “more reliable” without the fixed-compute-per-token reason. What does this tell you about reading AI-generated summaries?

13.12.2 Intermediate🔗
  1. Train TerpLM with –merges 16 and –merges 512 on the same corpus. Report bytes-per-token and final loss for each, and explain the trade-off in terms of vocabulary size versus sequence length. (Careful: losses over different vocabularies are not directly comparable—explain why.)

  2. Add a –top-k option to generate.py that samples only among the k most probable tokens. Compare output at –top-k 5 with temperature 1.0 against plain temperature 0.4.

  3. Write three new conversations for chat_corpus.txt that teach TerpLM a refusal (“I cannot help with that”). Post-train and report whether the refusal format appears in samples.

  4. Using the tiktokenizer website, find: a word that is one token capitalized but several lowercase; two strings that differ only in whitespace but tokenize differently; and the token count of a 280-character tweet. Explain the relevance to API pricing, which is per token.

  5. Design (on paper) the special tokens and sampler behavior for a calculator tool, in the style of the web-search example. Specify exactly when the sampler pauses, what enters the context window, and what SFT data you would need.

13.12.3 Advanced🔗
  1. Extend TerpLM to a trigram model that conditions on the previous two tokens (parameters W[t1, t2]). Report the loss improvement on Shakespeare and the parameter count, then explain why this approach cannot scale to a 1000-token context and what the transformer does instead.

  2. Implement the Llama-3-style hallucination probe at TerpLM scale—or, more practically, against a small open-weights chat model via an API: generate factual questions from a document, interrogate the model repeatedly, judge answers automatically, and measure how often it confidently fabricates versus declines.

  3. The notes claim RLHF reward models are gameable but verifiable checkers are not. Consider RL on code with unit tests as the checker. Is that checker truly ungameable? Describe two concrete ways a model could “pass the tests” while wrong, and connect them to the test-quality discussion in Software Testing.

13.13 Summary🔗

13.13.1 Key takeaways🔗
  • An LLM is a fixed mathematical expression that maps a token context to a probability distribution over the next token; training tunes its parameters to match data statistics, and inference samples from it one token at a time.

  • Pre-training compresses a filtered internet into the parameters (base model = internet document simulator); SFT programs an assistant persona by example; RL on verifiable problems lets the model discover its own reasoning.

  • Tokens, not characters: BPE trades vocabulary size for sequence length, and spelling/counting failures fall out of it.

  • Parameters are vague recollection; the context window is working memory. Put what matters in the window.

  • Fixed compute per token means models need tokens to think: intermediate steps are computation, not decoration.

  • The assistant you talk to simulates a human data labeler following labeling instructions; thinking models add emergent, RL-discovered chains of thought.

  • RLHF is a useful fine-tune, not magic: reward models are gameable simulations, so it cannot be scaled the way verifiable RL can.

  • Capability is jagged—olympiad-level and 9.11-vs-9.9-level in the same system. Verify accordingly.

13.13.2 Terminology🔗

Term

Meaning

token / tokenization

integer text-chunk IDs from a fixed vocabulary; the model’s only alphabet

byte-pair encoding

vocabulary learned by repeatedly merging the most frequent symbol pair

parameters / weights

the tuned numbers (billions) holding everything the model has learned

context window

the token sequence the model can attend to; its working memory

loss

cross-entropy gauge of prediction quality; falls during training

inference

generating tokens by repeated sampling from the model’s distribution

temperature

sampling sharpness: low → most-likely tokens, high → diversity

base model

pre-trained network only; a document simulator, not an assistant

knowledge cutoff

end date of the pre-training corpus

in-context learning

patterns picked up from the prompt at inference time (few-shot)

SFT

continued training on curated conversations; programs the assistant by example

special tokens

reserved IDs structuring conversations and tool calls

chat template (ChatML)

the special-token layout that serializes conversations

system message

hidden instructions prepended to every conversation

open weights

released parameters without training data or recipe; not fully open source

RAG

retrieval-augmented generation: fetch documents into the context window

hallucination

confident fabrication in the style of knowledge

chain of thought

intermediate reasoning tokens; computation in working memory

reasoning/thinking model

model RL-trained on verifiable problems (R1, o1/o3)

reward model

network trained on human rankings to score responses (RLHF)

distillation

training a smaller model to imitate a larger one’s outputs

13.13.3 Common mistakes🔗
  • Treating the model as a database and its answers as lookups —then being surprised by confident fabrication.

  • Asking for conclusions without letting the model show intermediate steps, or trusting its mental arithmetic instead of handing it a code tool.

  • Assuming the model can see characters, count items, or know about events after its cutoff.

  • Pasting nothing and asking it to “recall” your document instead of putting the document in the context window.

  • Anthropomorphizing identity questions (“who built you?”) that the weights cannot reliably answer.

  • Using an expensive thinking model for a simple lookup, or a fast chat model for a hard proof.

  • Forgetting that everything—including this lecture’s notes —should be verified before it ships. The model is confident by construction, not by knowledge.

13.13.4 Looking ahead🔗

Multimodal models tokenize audio (spectrogram slices) and images (patches) into the same one-dimensional streams—nothing in this lecture changes, there are just more kinds of tokens. Agents string model calls into long-running supervised jobs; computer-use models emit keyboard and mouse actions as tokens. A further open frontier is test-time training: today’s deployed models freeze their parameters and can only “learn” through the context window, while humans keep updating their parameters (not least while sleeping)—closing that gap is active research. For this course specifically: the next time we wrap a model in a FastAPI service (REST APIs and FastAPI: Designing and Building Web Interfaces), you will know exactly what is behind the endpoint; and when we evaluate LLM applications, the testing vocabulary of Software Testing returns—regression suites of prompts, properties over outputs, and red-teaming as adversarial input search against a system whose failure modes you now understand mechanically.

1These notes compress the lecture, add a runnable miniature implementation (TerpLM), and connect the material to the rest of this course. Mushtaq’s write-up also hosts a copy of Karpathy’s complete excalidraw diagram (Karpathy’s own sharing link expires). Several figures in this chapter are Karpathy’s diagrams, reproduced via that write-up.

2Some interfaces hide the raw chain of thought and show summaries, partly over distillation concerns—a competitor could train on the visible reasoning traces and recover much of the capability.