13 How LLMs Work: From Internet Text to ChatGPT
13.8.1 RLHF: reinforcement learning when you can’t check the answer |
These notes follow the arc of Andrej Karpathy’s lecture Deep Dive into LLMs like ChatGPT (YouTube, 3.5 hours), adapted for this course and paired with code you can run. A good short companion is Anfal Mushtaq’s TL;DR write-up of the same lecture.1These notes compress the lecture, add a runnable miniature implementation (TerpLM), and connect the material to the rest of this course. Mushtaq’s write-up also hosts a copy of Karpathy’s complete excalidraw diagram (Karpathy’s own sharing link expires). Several figures in this chapter are Karpathy’s diagrams, reproduced via that write-up.
13.1 Learning Objectives
By the end of this lecture you should be able to:
Describe the three training stages of a modern LLM—
pre-training, supervised fine-tuning, and reinforcement learning— including what data each consumes, what it produces, and roughly what it costs. Explain byte-pair encoding: why models read tokens rather than characters, and predict the practical consequences (case sensitivity, spelling failures, counting failures).
State what a language model fundamentally computes (a probability distribution over the next token), what its parameters are, what the loss measures, and what changes during training versus during inference.
Explain why a base model is an “internet document simulator,” and why the assistant you talk to is best understood as a simulation of a human data labeler following a company’s instructions.
Use the distinction between parameters (vague recollection) and the context window (working memory) to predict when a model will hallucinate and how tool use mitigates it.
Explain “models need tokens to think”: why a fixed amount of computation per token makes chains of intermediate steps necessary, and why counting and spelling fail.
Distinguish supervised fine-tuning from reinforcement learning from RLHF; explain what is new about “thinking” models like DeepSeek-R1 and o1/o3, and why RLHF cannot be run indefinitely.
Train, sample from, and “post-train” a complete miniature language model (tokenizer, parameters, loss, temperature) on your laptop.
13.2 Why Look Under the Hood?
In Introduction: This Is Not an AI Class we said the central skill of this course is
judgment over AI output, and in Software Testing we built
the machinery for verifying it. This lecture supplies the missing
piece: a mechanical understanding of the thing being judged. When
you know what is actually behind the ChatGPT text box—
The plan mirrors how the systems are actually built, in three sequential stages plus the psychology that falls out of them:
internet text conversations practice problems |
│ │ │ |
▼ ▼ ▼ |
┌──────────────┐ ┌──────────────────┐ ┌──────────────────┐ |
│ PRE-TRAINING │ ───► │ SUPERVISED │ ───► │ REINFORCEMENT │ |
│ (months, │ │ FINE-TUNING │ │ LEARNING │ |
│ $millions) │ │ (hours--days) │ │ (the new bit) │ |
└──────────────┘ └──────────────────┘ └──────────────────┘ |
│ │ │ |
▼ ▼ ▼ |
base model assistant "thinking" model |
(internet document (simulates a human (discovers its own |
simulator) data labeler) reasoning steps) |
13.3 The Running Example: TerpLM
Reading about LLMs is one thing; training one is better. Alongside the real systems, we will build TerpLM, a complete language model pipeline small enough to train on a laptop in two minutes: a real byte-pair-encoding tokenizer, a gradient-trained next-token predictor, sampling-based inference with temperature, and even a “post-training” stage. The project lives in the course repository under notes/llm/code/terplm.
TerpLM’s neural network is deliberately the dumbest one that still
works—
13.4 Stage 1: Pre-training
13.4.1 Step one: download the internet
Building an LLM starts with text—
URL filtering —
drop blocklisted domains: malware, spam, marketing, adult content, and so on. Text extraction —
crawled pages are raw HTML; strip the markup, navigation, and boilerplate, keeping just the text. Language filtering —
e.g. FineWeb keeps pages that are predominantly English. A design choice with visible consequences: filter out Spanish and your model will be bad at Spanish. Quality filters —
heuristic rules from the Gopher and C4 pipelines that drop boilerplate, junk, and degenerate pages. Deduplication —
drop near-identical pages (FineWeb uses MinHash) so the model does not over-train on repeats. PII removal —
filter pages containing personal data such as addresses and social security numbers.
The FineWeb filtering pipeline, from raw crawl to training text. Diagram: Andrej Karpathy, Deep Dive into LLMs like ChatGPT.
The punchline is how small the result is: FineWeb is about
44 terabytes—
13.4.2 Tokenization: what the model actually reads
Formal definition. A tokenizer is an invertible mapping between text and sequences of integers (tokens) drawn from a fixed, finite vocabulary. Modern LLMs learn the vocabulary with byte-pair encoding (BPE): start from the 256 possible bytes, repeatedly find the most frequent adjacent pair of symbols in the corpus, and “mint” a new symbol for that pair, until the vocabulary reaches a target size (GPT-4 uses 100,277 symbols).
In plain English: the model never sees letters or words. It sees chunk IDs. “hello world” is not eleven characters to GPT-4; it is two opaque integers.
Intuition. Neural networks want a one-dimensional sequence of symbols from a finite set, and sequence length is the scarce resource. Raw bits would give 2 symbols and absurdly long sequences; bytes give 256 symbols and sequences 8 times shorter; BPE keeps trading a bigger vocabulary for shorter sequences, settling around 100k symbols at roughly 4 characters per token of English.
Common misconception. “Tokens are words.” They are not: common words get one token, rare words get several, whitespace and capitalization change everything, and the same string tokenizes differently depending on a leading space. Sites like tiktokenizer let you see GPT-4’s actual tokenization interactively; you will be surprised.
TerpLM contains a real BPE trainer in about sixty lines (terplm/bpe.py). Training 60 merges on a small corpus about the university and then encoding a sentence:
first 6 merges learned: |
( 104, 101) -> 256 meaning 'h' + 'e' = 'he' |
( 116, 256) -> 257 meaning 't' + 'he' = 'the' |
( 116, 101) -> 258 meaning 't' + 'e' = 'te' |
( 114, 97) -> 259 meaning 'r' + 'a' = 'ra' |
( 116, 117) -> 260 meaning 't' + 'u' = 'tu' |
( 105, 110) -> 261 meaning 'i' + 'n' = 'in' |
|
encode('the terrapin tested the program') |
-> [257, 32, 275, 32, 299, 258, 100, 32, 257, 32, 287, 109] |
-> [the][ ][terrapin][ ][tes][te][d][ ][the][ ][progra][m] |
31 bytes -> 12 tokens |
The merges are readable: the algorithm discovered he, then
the, exactly as you would expect from English statistics, and
“terrapin”—
'Maryland' -> [Maryland] |
'maryland' -> [m][a][ry][l][and] |
' Maryland' -> [ ][Maryland] |
The capitalized form was frequent in the training corpus and earned a single token; the lowercase form shatters into five. To the model these are completely unrelated symbols. Hold that thought for the section on spelling failures.
Practical implications. Tokenization is a preprocessing
step done once; after it, “the internet” is one enormous
one-dimensional sequence of integers, and everything downstream—
13.4.3 The neural network: input, output, and training
Formal definition. A language model is a function,
parameterized by a vector of numbers (the parameters or
weights), that maps a sequence of tokens (the context,
up to a maximum context length—
In plain English: the model is a guess-the-next-token
machine, and training is millions of rounds of “here are the
previous tokens—
One training example: a window of four tokens goes in; the network outputs a probability for each of the 100,277 possible next tokens; training nudges the probability of the actual next token (3962) upward. Diagram: Andrej Karpathy.
Intuition. Think of the parameters as knobs on a giant DJ
mixing board (GPT-2 had 1.6 billion knobs; modern models hundreds of
billions). At random settings the machine outputs noise. Training
is an automated procedure that twiddles the knobs, a tiny step at a
time, until the machine’s output statistics match the corpus
statistics. Inside, the “machine” is just one big differentiable
mathematical expression—
Common misconception. “The model stores its training text
in a database and looks things up.” There is no database and no
lookup. Whatever the model “knows” is smeared across the
parameters as statistical tendencies—
Watching the loss fall is the daily life of an LLM researcher, so
let’s do it. TerpLM’s model (terplm/model.py) is one matrix
W: row t holds the logits—
def train_step(self, prev: np.ndarray, nxt: np.ndarray, lr: float) -> float:
"""One gradient-descent update on a batch of (prev, next) pairs."""
probs = _softmax(self.W[prev]) # (batch, vocab)
batch = len(prev)
loss = float(-np.log(probs[np.arange(batch), nxt] + 1e-12).mean())
# Gradient of cross-entropy through softmax: probs - one_hot.
probs[np.arange(batch), nxt] -= 1.0
np.add.at(self.W, prev, -lr * probs / batch)
return lossWe train it on the classic “tiny shakespeare” corpus (1.1 MB of public-domain text), with a 514-token vocabulary. The run prints the loss and a 12-token sample every 250 steps:
$ bash scripts/fetch_shakespeare.sh
$ python -m terplm.train data/shakespeare.txt --merges 256 --steps 3000
training tokenizer: 256 merges ...
corpus: 1,115,394 chars -> 691,315 tokens (vocab 514, 1.61 bytes/token)
loss of uniform guessing: ln(514) = 6.242
step 0 loss 6.242 sample: 'Fet?\x15\x08Butmorekesuhaciath\x01'
step 500 loss 5.202 sample: 'Fcesheght??ight?what?love'
step 1000 loss 4.803 sample: 'Fonce?hand vomitilgo~'
step 1500 loss 4.541 sample: 'F\x1e, be dehovustheeis '
step 2000 loss 4.343 sample: 'FndIN\x0eanhis then?you a?'
step 3000 loss 4.113 sample: "F?c?Yliindup?'t"
saved model + tokenizer to out/Read this the way a researcher does. At step 0 the parameters are
random, the loss sits exactly at ln(514)—
13.4.4 Inference: sampling, stochasticity, and temperature
Formal definition. Inference generates text from a trained model: feed in a token prefix, obtain the probability distribution for the next token, sample one token from that distribution, append it, and repeat. Temperature rescales the logits before the softmax; low temperature sharpens the distribution toward the most likely token, high temperature flattens it.
In plain English: the model does not “decide” on a reply;
it flips a biased coin per token, forever. When you use ChatGPT,
no learning happens—
The inference loop: each sampled token is appended and fed back in. Note the last row: the model samples 13659 where the training document had 3962—
generation is a remix of training statistics, not a recital. Diagram: Andrej Karpathy.
The consequences are immediately visible in TerpLM. The same prompt, two seeds:
$ python -m terplm.generate --prompt "ROMEO:" --tokens 40 --seed 1
ROMEO:?E:/I'll??thee?veruto thy to or say andlovespeconhe festis fornoseilme O
$ python -m terplm.generate --prompt "ROMEO:" --tokens 40 --seed 2
ROMEO:a will my h?ly,
Uall ascilo??orWhatmor?ARD?ver bjOLloile, you sh?Different every time: these are stochastic systems, remixing the statistics of their training data rather than reciting it. Now temperature, taken to the extreme:
$ python -m terplm.generate --prompt "ROMEO:" --temperature 0.4
ROMEO:
Th?I'll the the I the of the the s s are he s so bar to I the
$ python -m terplm.generate --prompt "ROMEO:" --temperature 0.05
ROMEO:
Whow the the the the the the the the the the the the theAt temperature 0.05 the model almost always takes the single most
probable next token, and in English bigram statistics the most
probable follower of almost anything is the—
13.5 The Base Model: a Token Simulator
What pre-training produces is called a base model, and it is not an assistant.
Formal definition. A base model is the network plus its trained parameters, with no further conditioning: a sampler of token sequences whose statistics match its training corpus.
In plain English: it is an internet document simulator.
Prompt it with “What is 2+2?” and it will not answer you—
Intuition. The parameters are a lossy compression of
the internet—
Releasing a model, by the way, means publishing exactly two
artifacts: the inference code—
Common misconception. “Useless, then.” Far from it: all of the knowledge lives here, and three behaviors of base models explain most of what assistants later do:
Regurgitation. Prompt a strong base model with the first sentence of a famous Wikipedia article and it may recite the continuation nearly verbatim—
high-quality sources are deliberately oversampled in training (the model has effectively “read” Wikipedia ten times), and like a student who has reread a text ten times, it can recite. Verbatim recitation is usually undesirable— and it should temper how you think about both the privacy and the copyright of training data. Educated hallucination. Prompt it with text about events after its knowledge cutoff (the end date of its training corpus) and it confidently continues with a plausible parallel universe—
naming the wrong running mate, the wrong opponent. Resample and you get a different parallel universe. Nothing is malfunctioning: it is doing exactly what it was trained to do, continue documents plausibly. In-context learning. Give it ten English:Korean word pairs and an eleventh English word, and it completes the pattern with a correct translation. The pattern was learned inside the context window at inference time, not during training. This emergent ability—
few-shot prompting— was the bridge that made base models practically useful, and you can even conjure a passable assistant from a pure base model by formatting your prompt as a transcript of a helpful-assistant conversation and letting the model continue it.
That last trick—
13.6 Stage 2: Supervised Fine-Tuning— Programming by Example
Formal definition. Supervised fine-tuning (SFT)
continues training the base model with the same algorithm and the
same objective—
In plain English: you cannot program a neural network’s behavior in code, so you program it by example. Want it to answer questions helpfully, decline harmful requests, admit uncertainty? Write a few hundred thousand conversations that do exactly that, and train on them. The model rapidly adopts the persona.
Intuition. Pre-training takes months on thousands of GPUs; SFT can take hours, because the conversation corpus is minuscule next to the internet. The base model already contains the knowledge; SFT teaches it a style of access.
Conversations must become token sequences like everything else, so
each lab defines a chat template with new special
tokens—
<|im_start|>system<|im_sep|>You are a helpful assistant<|im_end|> |
<|im_start|>user<|im_sep|>What is 4 + 4?<|im_end|> |
<|im_start|>assistant<|im_sep|>4 + 4 = 8<|im_end|> |
The system turn carries standing instructions (identity,
tone, rules)—
Where do the conversations come from? This is the part everyone
should know: from people. The InstructGPT paper (OpenAI,
2022) described hiring human labelers and handing them
labeling instructions—
Common misconception—
13.6.1 Post-training TerpLM
The claim “SFT is the same algorithm with a different dataset” sounds too simple to be true, so TerpLM demonstrates it literally. Our train.py has a –resume flag that loads the Shakespeare-trained model and continues training on a tiny corpus of conversations in the <|im_start|> protocol (the special tokens are single IDs in TerpLM’s vocabulary, exactly as in production):
$ python -m terplm.train data/chat_corpus.txt --resume --steps 2000
resuming: swapping dataset to data/chat_corpus.txt (same algorithm)
corpus: 1,754 chars -> 850 tokens (vocab 514, 2.06 bytes/token)
step 0 loss 4.982 sample: '<|im_start|>et I\x15qumoreutbut4ciBut\x01'
step 500 loss 1.445 sample: '<|im_start|>user\nWhat con?what?love'
step 1000 loss 1.140 sample: '<|im_start|>user\nI builtant\nI'
step 2000 loss 1.064 sample: '<|im_start|>user\nWhat land.<|im_end|>\nI '
$ python -m terplm.generate --prompt "<|im_start|>user" --tokens 45
<|im_start|>user
What you the buame?<|im_end|>
<|im_start|>user
<|im_start|>assity modeland.<|im_end|>
I from you your builtantLook closely at what two thousand updates on 850 tokens bought.
The model has visibly learned the protocol: turns begin with
<|im_start|>, end with <|im_end|>, roles appear after the
start marker, questions trail question marks. The content
is word salad—
13.7 LLM Psychology: Predictable Failure Modes
Everything in this section is a consequence of the pipeline above, derivable rather than memorizable.
13.7.1 Hallucinations, and the two kinds of memory
Formal definition. A hallucination is fluent,
confident output that is factually false—
Why does it happen? Look at the SFT data. Training conversations
of the form “Who is X?” are answered confidently and correctly,
because the human labeler looked X up. The model learns the
form: such questions get confident answers. Ask it about a
person who does not exist and the statistically appropriate
continuation is a confident biography—
Two mitigations are now standard, and both are instructive:
Teach the model to say “I don’t know”—
but only where it truly doesn’t. Meta’s Llama 3 team probes the knowledge boundary empirically: generate factual questions from documents, interrogate the model several times per question, and use an LLM judge to compare its answers with the known correct one. Where the model reliably fails, add a training conversation whose correct answer is “I’m sorry, I don’t know.” The remarkable part: a modest number of such examples generalizes. The network plausibly already contains internal activations correlated with uncertainty; the new data merely wires that signal to words. Tools: let the model look it up. Introduce special tokens such as SEARCH_START/SEARCH_END. When the sampler sees the model emit a search query between them, it pauses generation, runs the query against a real search engine, pastes the retrieved text into the context window, and resumes sampling. At the token level the whole exchange looks like this:
<|im_start|>user<|im_sep|>Who is Orson Kovacs?<|im_end|>
<|im_start|>assistant<|im_sep|><SEARCH_START>Who is Orson
Kovacs?<SEARCH_END><|im_end|>
[...search results pasted into the context window...]
<|im_start|>assistant<|im_sep|>Orson Kovacs is ...<|im_end|>
A few thousand SFT examples teach the model when and how to reach for the tool. The citations you see in modern chat interfaces are exactly this mechanism.
The second mitigation works because of a distinction worth engraving somewhere:
knowledge in the PARAMETERS = vague recollection |
(read it months ago, statistically compressed) |
|
knowledge in the CONTEXT WINDOW = working memory |
(directly attended to on every forward pass) |
This one distinction is the most practically useful fact in the lecture. “Summarize chapter 1 of Pride and Prejudice” works passably from recollection because the internet has discussed that chapter to death; “summarize chapter 1, which I am pasting below” works far better, for the same reason you would summarize better having just reread the chapter. This is the entire idea of retrieval-augmented generation (RAG), which we will meet later in the course: when you build LLM applications, the game is getting the right information into the context window instead of hoping it is in the parameters.
13.7.2 Models need tokens to think
Here is the deepest of the practical insights. Recall the
architecture: producing each token costs one forward pass
through a network with a fixed number of layers. Per token, the
model performs a fixed, finite amount of computation—
Now suppose you are a data labeler writing the ideal response to: “Emily buys 3 apples and 2 oranges. Each orange costs $2. The total is $13. What does an apple cost?” Two candidate responses:
BAD: "The answer is $3. We can see this because |
13 - 4 = 9 and 9 / 3 = 3." |
|
GOOD: "The oranges cost 2 × 2 = $4. So the apples |
cost 13 - 4 = $9. There are 3 apples, so |
each apple costs 9 / 3 = $3. The answer is $3." |
Both are correct. The first is a terrible training label.
Reading left to right, it asks the model to produce the token
3 immediately—
Two corollaries you can verify in any chat window today:
Counting fails. “How many dots: ........” asks for all the work in one token, on top of which the dots arrive bundled into opaque multi-dot tokens. The fix is delegation: ask the model to use code—
copy-pasting the dots into a Python string is easy (it is just echoing tokens), and .count(".") does the counting with an interpreter instead of neural mental arithmetic. Spelling fails. The famous “how many r’s in strawberry?” The model does not see r’s; it sees one or two token IDs for the whole word (remember maryland → [m][a][ry][l][and] from our own tokenizer—
characters are smeared unpredictably across symbols), and it must also count. Two known weaknesses, multiplied. Again: “use code” fixes it.
And one humbling note to file under jagged intelligence:
models that solve olympiad problems have claimed that 9.11 is
greater than 9.9 (one investigation found internal features
associated with Bible verses activating—
13.8 Stage 3: Reinforcement Learning— Practice Problems
The school analogy organizes everything so far:
In a textbook | Training stage | What the model gets |
exposition / background prose | pre-training | knowledge |
worked examples with solutions | supervised fine-tuning | imitation of experts |
practice problems (answer key only) | reinforcement learning | skill discovered by trial and error |
Formal definition. In reinforcement learning (RL)
for LLMs, the model is given prompts with verifiable answers
(math, code with tests), samples many candidate solutions per
prompt—
In plain English: guess and check, at industrial scale, with “do more of what worked.”
One RL step on the very prompt from our “tokens to think” section: 15 sampled solutions, 4 reach the right answer (green), and the model trains on the best of them. Repeat across tens of thousands of prompts. Diagram: Andrej Karpathy.
Intuition—
What emerges. The DeepSeek-R1 paper (2025) made the recipe
public and showed two things happening as RL proceeds: accuracy on
math problems climbs, and—
The precedent. We have seen this movie: AlphaGo. The
supervised version, trained to imitate expert human games, topped
out just below the best humans—
From the AlphaGo paper (Silver et al., 2016): the supervised-learning version plateaus below top human strength (dashed line); the reinforcement-learning version climbs past it.
13.8.1 RLHF: reinforcement learning when you can’t check the answer
Math has an answer key. “Write a joke about pelicans” does not.
Formal definition. Reinforcement learning from human
feedback (RLHF) extends RL to unverifiable domains by training a
separate reward model—
The RLHF indirection: replace a billion human judgments with five thousand, plus a learned simulator of the judge. Diagram: Andrej Karpathy.
Intuition—
The catch, and it is fundamental: the reward model is a
billion-parameter simulation of human preference, and RL is
relentlessly good at finding inputs that fool it. Run RLHF long
enough and the “best joke about pelicans” becomes—
13.9 Grand Summary: What Are You Talking To?
Trace your next ChatGPT query through everything above. Your text
is wrapped in a chat protocol and tokenized into a one-dimensional
sequence. A fixed mathematical expression with frozen parameters
—
For a standard chat model: a statistical simulation of a skilled data labeler following their employer’s instructions—
instant, tireless, broadly knowledgeable, and capable of arbitrary holes. For a thinking model: the same, plus reasoning strategies no labeler wrote—
discovered by the model through millions of practice problems— unrolling in its working memory before it answers.
Neither is a person; neither has persistent existence between your messages; both are extraordinary tools. The operating rules this course keeps returning to all follow mechanically:
Rule of practice | Because |
Put relevant material in the prompt rather than assuming the model knows it | context window = working memory; parameters = vague recollection |
Let it reason in steps; never demand hard answers “in one word” | fixed compute per token |
Push arithmetic, counting, and string-twiddling into tools/code | neural mental math over opaque tokens is unreliable |
Reach for a thinking model on hard reasoning; a fast chat model otherwise | RL-trained deliberation costs time and money and is overkill for lookups |
Verify everything that matters (see Software Testing) | stochastic Swiss-cheese capability, confident by construction |
13.10 Trying These Systems Yourself
Everything this chapter describes can be poked at directly, mostly for free:
Tokenizers. tiktokenizer shows GPT-4’s actual tokenization of anything you type—
try the Maryland/maryland experiment, and switch the dropdown to a chat model to see the <|im_start|> serialization of a conversation. Inside a transformer. bbycroft.net/llm is an interactive 3D visualization of a real (small) production architecture—
the “giant mathematical expression” rendered walkable. Base models. Hyperbolic serves Llama 3.1 405B base—
one of the few places to experience a raw token simulator (try the regurgitation, parallel-universe, and few-shot demos from this chapter). Open-weights chat and thinking models. together.ai hosts DeepSeek-R1, Llama, and friends; watching R1’s visible chain of thought on a math problem is the best free demonstration of the RL section.
Running models locally. LM Studio or Ollama run distilled, quantized open-weights models on a laptop—
no data leaves your machine. Keeping up. LM Arena ranks models by blind human preference votes (useful as a first pass, though increasingly gamed—
try a few models on your tasks before trusting any leaderboard), and the AI News newsletter is a comprehensive, largely LLM-written digest of the field.
13.11 The TerpLM Project
The complete miniature pipeline is in the course repository at notes/llm/code/terplm:
terplm/ |
├── terplm/ |
│ ├── bpe.py # byte-pair-encoding tokenizer (train/encode/decode) |
│ ├── model.py # bigram LM: parameters, loss, gradient, sampling |
│ ├── train.py # training CLI ("post-training" via --resume) |
│ └── generate.py # inference CLI (prompt, temperature, seed) |
├── data/ |
│ ├── tiny_corpus.txt # bundled corpus (runs in seconds) |
│ └── chat_corpus.txt # conversations with <|im_start|> tokens |
├── scripts/fetch_shakespeare.sh |
└── tests/ # BPE round-trip property, distribution sanity |
cd notes/llm/code/terplm
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pytest # 11 tests
bash scripts/fetch_shakespeare.sh # 1. "download the internet"
python -m terplm.train data/shakespeare.txt --merges 256 --steps 3000 # 2. pre-train
python -m terplm.generate --prompt "ROMEO:" --seed 1 # 3. inference
python -m terplm.generate --prompt "ROMEO:" --temperature 0.05 # temperature
python -m terplm.train data/chat_corpus.txt --resume --steps 2000 # 4. post-train
python -m terplm.generate --prompt "<|im_start|>user" --temperature 0.6 # 5. "assistant"The test suite is a callback to Software Testing: the
tokenizer’s central guarantee—
13.12 Practice Exercises
13.12.1 Basic
In your own words, why does an LLM see Maryland and maryland as unrelated symbols? Use TerpLM’s tokenizer output to illustrate.
TerpLM’s untrained loss is exactly ln(514) = 6.242. Derive that number. What would it be for GPT-4’s vocabulary of 100,277?
Explain the difference between what changes during training and what changes during inference. Which one is happening when you use ChatGPT?
Classify each behavior as caused by (a) tokenization, (b) fixed compute per token, or (c) parameters-as-vague-recollection: miscounting the r’s in strawberry; inventing a biography for a fake name; failing “answer in one word” arithmetic.
Why is the base model called a “token simulator”? Give two reasons it is still valuable without any post-training.
Read a TL;DR summary of the Karpathy lecture (one is linked in the footnote at the top of this chapter). Find two places where the summary keeps the what but drops the why —
for example, it may say step-by-step answers are “more reliable” without the fixed-compute-per-token reason. What does this tell you about reading AI-generated summaries?
13.12.2 Intermediate
Train TerpLM with –merges 16 and –merges 512 on the same corpus. Report bytes-per-token and final loss for each, and explain the trade-off in terms of vocabulary size versus sequence length. (Careful: losses over different vocabularies are not directly comparable—
explain why.) Add a –top-k option to generate.py that samples only among the k most probable tokens. Compare output at –top-k 5 with temperature 1.0 against plain temperature 0.4.
Write three new conversations for chat_corpus.txt that teach TerpLM a refusal (“I cannot help with that”). Post-train and report whether the refusal format appears in samples.
Using the tiktokenizer website, find: a word that is one token capitalized but several lowercase; two strings that differ only in whitespace but tokenize differently; and the token count of a 280-character tweet. Explain the relevance to API pricing, which is per token.
Design (on paper) the special tokens and sampler behavior for a calculator tool, in the style of the web-search example. Specify exactly when the sampler pauses, what enters the context window, and what SFT data you would need.
13.12.3 Advanced
Extend TerpLM to a trigram model that conditions on the previous two tokens (parameters W[t1, t2]). Report the loss improvement on Shakespeare and the parameter count, then explain why this approach cannot scale to a 1000-token context and what the transformer does instead.
Implement the Llama-3-style hallucination probe at TerpLM scale—
or, more practically, against a small open-weights chat model via an API: generate factual questions from a document, interrogate the model repeatedly, judge answers automatically, and measure how often it confidently fabricates versus declines. The notes claim RLHF reward models are gameable but verifiable checkers are not. Consider RL on code with unit tests as the checker. Is that checker truly ungameable? Describe two concrete ways a model could “pass the tests” while wrong, and connect them to the test-quality discussion in Software Testing.
13.13 Summary
13.13.1 Key takeaways
An LLM is a fixed mathematical expression that maps a token context to a probability distribution over the next token; training tunes its parameters to match data statistics, and inference samples from it one token at a time.
Pre-training compresses a filtered internet into the parameters (base model = internet document simulator); SFT programs an assistant persona by example; RL on verifiable problems lets the model discover its own reasoning.
Tokens, not characters: BPE trades vocabulary size for sequence length, and spelling/counting failures fall out of it.
Parameters are vague recollection; the context window is working memory. Put what matters in the window.
Fixed compute per token means models need tokens to think: intermediate steps are computation, not decoration.
The assistant you talk to simulates a human data labeler following labeling instructions; thinking models add emergent, RL-discovered chains of thought.
RLHF is a useful fine-tune, not magic: reward models are gameable simulations, so it cannot be scaled the way verifiable RL can.
Capability is jagged—
olympiad-level and 9.11-vs-9.9-level in the same system. Verify accordingly.
13.13.2 Terminology
Term | Meaning |
token / tokenization | integer text-chunk IDs from a fixed vocabulary; the model’s only alphabet |
byte-pair encoding | vocabulary learned by repeatedly merging the most frequent symbol pair |
parameters / weights | the tuned numbers (billions) holding everything the model has learned |
context window | the token sequence the model can attend to; its working memory |
loss | cross-entropy gauge of prediction quality; falls during training |
inference | generating tokens by repeated sampling from the model’s distribution |
temperature | sampling sharpness: low → most-likely tokens, high → diversity |
base model | pre-trained network only; a document simulator, not an assistant |
knowledge cutoff | end date of the pre-training corpus |
in-context learning | patterns picked up from the prompt at inference time (few-shot) |
SFT | continued training on curated conversations; programs the assistant by example |
special tokens | reserved IDs structuring conversations and tool calls |
chat template (ChatML) | the special-token layout that serializes conversations |
system message | hidden instructions prepended to every conversation |
open weights | released parameters without training data or recipe; not fully open source |
RAG | retrieval-augmented generation: fetch documents into the context window |
hallucination | confident fabrication in the style of knowledge |
chain of thought | intermediate reasoning tokens; computation in working memory |
reasoning/thinking model | model RL-trained on verifiable problems (R1, o1/o3) |
reward model | network trained on human rankings to score responses (RLHF) |
distillation | training a smaller model to imitate a larger one’s outputs |
13.13.3 Common mistakes
Treating the model as a database and its answers as lookups —
then being surprised by confident fabrication. Asking for conclusions without letting the model show intermediate steps, or trusting its mental arithmetic instead of handing it a code tool.
Assuming the model can see characters, count items, or know about events after its cutoff.
Pasting nothing and asking it to “recall” your document instead of putting the document in the context window.
Anthropomorphizing identity questions (“who built you?”) that the weights cannot reliably answer.
Using an expensive thinking model for a simple lookup, or a fast chat model for a hard proof.
Forgetting that everything—
including this lecture’s notes — should be verified before it ships. The model is confident by construction, not by knowledge.
13.13.4 Looking ahead
Multimodal models tokenize audio (spectrogram slices) and images
(patches) into the same one-dimensional streams—
1These notes compress the lecture, add a runnable miniature implementation (TerpLM), and connect the material to the rest of this course. Mushtaq’s write-up also hosts a copy of Karpathy’s complete excalidraw diagram (Karpathy’s own sharing link expires). Several figures in this chapter are Karpathy’s diagrams, reproduced via that write-up.





