On this page:
23.1 Learning Objectives
23.2 Why Testing?
23.2.1 Two warnings from the pioneers
23.2.2 Why testing matters
23.2.3 The cost of finding bugs late
23.2.4 What is a test?
23.2.5 Verification versus validation
23.2.6 Testing versus debugging
23.2.7 Testing versus formal verification
23.2.7.1 Example:   verification in Dafny
23.2.8 Why is software hard to test?
23.3 The Running Example:   Terp  Ledger
23.4 Levels and Types of Testing
23.4.1 Testing scale:   unit, integration, system
23.4.2 The V model
23.4.3 Regression testing
23.4.4 The test pyramid
23.4.5 Testing by purpose
23.4.6 When are tests written?
23.5 Designing Good Test Cases
23.5.1 A warm-up:   withdraw
23.5.2 Equivalence partitioning
23.5.3 Boundary value analysis
23.5.4 Edge cases, positive and negative tests
23.5.5 Test oracles
23.6 Testing Frameworks and Automation
23.6.1 Assertions and the Arrange–Act–Assert pattern
23.6.2 Reading a failure
23.6.3 Parameterized tests
23.6.4 Fixtures:   setup and teardown
23.6.5 Test suites and test organization
23.6.6 Mocking and stubbing:   test doubles
23.6.7 What makes a test good?
23.7 Test-Driven Development
23.8 Property-Based Testing
23.8.1 A first property:   reversing a list
23.8.2 A weak property can miss a bug
23.8.3 Hypothesis hunts a Terp  Ledger bug
23.8.4 Shrinking
23.8.5 Generators
23.8.6 A catalog of properties
23.8.7 Seeds and reproducibility
23.9 Differential Testing
23.10 Code Coverage
23.10.1 Measuring coverage
23.10.2 Statement, branch, and condition coverage compared
23.10.3 What coverage cannot tell you
23.11 Mutation Testing
23.11.1 Mutants and mutation operators
23.11.2 A tiny mutation tester
23.11.3 Worked example:   a strong suite
23.11.4 Worked example:   finding a real gap
23.11.5 Equivalent mutants and cost
23.12 Fuzz Testing
23.12.1 The harness
23.12.2 Generators:   random bytes versus structured input
23.12.3 Coverage-based randomized testing
23.12.4 Fuzzing versus property-based testing
23.13 Static Analysis
23.14 Testing in Modern Software Development
23.14.1 Continuous integration
23.14.2 Code review, test requirements, and deployment
23.15 Testing and Generative AI
23.15.1 Using LLMs to generate unit tests:   the oracle problem
23.15.2 Asking AI for missing edge cases and properties
23.15.3 Explaining failures and generating fixtures
23.15.4 Testing AI-generated code
23.15.5 Using mutation testing to evaluate AI-generated tests
23.15.6 A workflow for AI-assisted testing
23.16 Which Technique Catches Which Bug?
23.17 The Terp  Ledger Project
23.18 Practice Exercises
23.18.1 Basic
23.18.2 Intermediate
23.18.3 Advanced
23.19 Summary
23.19.1 Key takeaways
23.19.2 Terminology
23.19.3 Common mistakes
23.19.4 Looking ahead
9.1

23 Software Testing🔗

    23.1 Learning Objectives

    23.2 Why Testing?

      23.2.1 Two warnings from the pioneers

      23.2.2 Why testing matters

      23.2.3 The cost of finding bugs late

      23.2.4 What is a test?

      23.2.5 Verification versus validation

      23.2.6 Testing versus debugging

      23.2.7 Testing versus formal verification

        23.2.7.1 Example: verification in Dafny

      23.2.8 Why is software hard to test?

    23.3 The Running Example: TerpLedger

    23.4 Levels and Types of Testing

      23.4.1 Testing scale: unit, integration, system

      23.4.2 The V model

      23.4.3 Regression testing

      23.4.4 The test pyramid

      23.4.5 Testing by purpose

      23.4.6 When are tests written?

    23.5 Designing Good Test Cases

      23.5.1 A warm-up: withdraw

      23.5.2 Equivalence partitioning

      23.5.3 Boundary value analysis

      23.5.4 Edge cases, positive and negative tests

      23.5.5 Test oracles

    23.6 Testing Frameworks and Automation

      23.6.1 Assertions and the Arrange–Act–Assert pattern

      23.6.2 Reading a failure

      23.6.3 Parameterized tests

      23.6.4 Fixtures: setup and teardown

      23.6.5 Test suites and test organization

      23.6.6 Mocking and stubbing: test doubles

      23.6.7 What makes a test good?

    23.7 Test-Driven Development

    23.8 Property-Based Testing

      23.8.1 A first property: reversing a list

      23.8.2 A weak property can miss a bug

      23.8.3 Hypothesis hunts a TerpLedger bug

      23.8.4 Shrinking

      23.8.5 Generators

      23.8.6 A catalog of properties

      23.8.7 Seeds and reproducibility

    23.9 Differential Testing

    23.10 Code Coverage

      23.10.1 Measuring coverage

      23.10.2 Statement, branch, and condition coverage compared

      23.10.3 What coverage cannot tell you

    23.11 Mutation Testing

      23.11.1 Mutants and mutation operators

      23.11.2 A tiny mutation tester

      23.11.3 Worked example: a strong suite

      23.11.4 Worked example: finding a real gap

      23.11.5 Equivalent mutants and cost

    23.12 Fuzz Testing

      23.12.1 The harness

      23.12.2 Generators: random bytes versus structured input

      23.12.3 Coverage-based randomized testing

      23.12.4 Fuzzing versus property-based testing

    23.13 Static Analysis

    23.14 Testing in Modern Software Development

      23.14.1 Continuous integration

      23.14.2 Code review, test requirements, and deployment

    23.15 Testing and Generative AI

      23.15.1 Using LLMs to generate unit tests: the oracle problem

      23.15.2 Asking AI for missing edge cases and properties

      23.15.3 Explaining failures and generating fixtures

      23.15.4 Testing AI-generated code

      23.15.5 Using mutation testing to evaluate AI-generated tests

      23.15.6 A workflow for AI-assisted testing

    23.16 Which Technique Catches Which Bug?

    23.17 The TerpLedger Project

    23.18 Practice Exercises

      23.18.1 Basic

      23.18.2 Intermediate

      23.18.3 Advanced

    23.19 Summary

      23.19.1 Key takeaways

      23.19.2 Terminology

      23.19.3 Common mistakes

      23.19.4 Looking ahead

23.1 Learning Objectives🔗

By the end of this lecture you should be able to:

  1. Explain what testing can and cannot show, and tell apart verification and validation, testing and debugging, and testing and formal verification.

  2. Place unit, integration, system, acceptance, and regression tests on the V model and the test pyramid, and explain why a healthy suite has many small tests and few large ones.

  3. Design test cases systematically using equivalence partitioning, boundary value analysis, positive and negative tests, and an explicit test oracle.

  4. Write automated tests with pytest: assertions, parametrized tests, fixtures with setup and teardown, and test doubles (stubs, fakes, spies, mocks). Apply the test-driven development cycle.

  5. Write property-based tests with Hypothesis, explain how generators and shrinking work, and use differential testing when a trusted reference implementation exists.

  6. Measure function, statement, branch, and condition coverage, and explain why 100% coverage does not mean the code is correct.

  7. Explain mutation testing and fuzz testing (including coverage-guided fuzzing), and use mutation testing to grade a test suite.

  8. Run tests automatically in continuous integration, and use LLMs to generate, improve, and evaluate tests without trusting them blindly.

23.2 Why Testing?🔗

23.2.1 Two warnings from the pioneers🔗

Two quotes frame this whole lecture. The first is from Edsger Dijkstra:

Program testing can be used to show the presence of bugs, but never to show their absence!

A passing test suite means that no test you thought to write has failed—nothing more. A function that takes two 64-bit integers has 2128 possible inputs; you will test perhaps twenty. Everything in this lecture is either a way to choose those twenty inputs more wisely, a way to check many more inputs automatically, or a way to measure how much your tests actually check.

The second is from Tony Hoare:

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

Testing and design are connected. Code that is simple and has clear inputs and outputs is easy to test. Code that secretly reads global state, talks to the network, or depends on the current time is hard to test—and that difficulty is usually a sign that the design is wrong.

23.2.2 Why testing matters🔗
  • Bugs are expensive. Developers are often estimated to spend around half of their time finding and fixing bugs. Testing is not the only way industry evaluates software under development, but it is by far the most common one.

  • Software is built by changing it. Most code you write changes a system that already works. A regression is an old feature broken by a new change. A test suite is a regression alarm: it re-checks everything you have ever promised, every time anything changes.

  • Tests are executable documentation. A test states, in runnable form, what the code is supposed to do. Unlike a comment, it cannot silently become wrong: if the behavior changes, the test fails.

  • A few ideas go a long way. The same small set of concepts—partitions, boundaries, oracles, properties, coverage, mutants—lets you design tests for almost any kind of software. You will use them for your whole career.

  • Generative AI makes testing more important, not less. An LLM can write a hundred lines of plausible code in seconds. The question that decides whether you can ship those lines is: how would I know if this is wrong? Every technique in this lecture is one answer to that question.

23.2.3 The cost of finding bugs late🔗

The later a defect is found, the more it costs to fix. A commonly cited rule of thumb (the numbers vary from study to study, but the shape never does):

Phase where the bug is found

Relative cost to fix

Requirements

1×

Design / architecture

3×

Coding

7×

Testing

15×

Deployment / maintenance

30× to 100×

cost

 ▲                                                ████  30x..100x

 │                                                ████

 │                                                ████

 │                                     ████ 15x   ████

 │                         ████ 7x     ████       ████

 │             ████ 3x     ████        ████       ████

 │  ████ 1x    ████        ████        ████       ████

 └───────────────────────────────────────────────────────►  time

  requirements  design    coding     testing    deployment

A bug found by a unit test on your laptop costs a minute. The same bug found by a customer costs a bug report, a reproduction, a fix, a review, a release, and possibly lost data or trust. This is why modern teams try to shift testing left: test early, test automatically, and test on every change.

23.2.4 What is a test?🔗

Definition. A test case consists of an input (including any setup the program needs), an action that runs the code under test, and an expected result. The thing that decides what the expected result is—a person, a specification, or another program—is called the test oracle. A test suite is a collection of test cases.

In plain English: a test runs your code on an input you chose and checks the answer against what the answer should be.

Intuition. A test is a small experiment. You make a prediction (the expected result), run the experiment, and compare. A failing test means either the code is wrong or your prediction was wrong—and both are worth knowing.

23.2.5 Verification versus validation🔗

These two words sound alike but ask different questions:

  • Verification: Are we building the product right? Does the code meet its specification? Example: “split_evenly returns shares that add up to the total, as the docstring promises.”

  • Validation: Are we building the right product? Does the specification meet the user’s actual needs? Example: “Do club members actually want the leftover cent assigned to the first person, or would they prefer it rotated?”

A program can pass every verification test and still fail validation, because the specification itself was wrong. Unit tests mostly verify; acceptance tests and user studies validate.

23.2.6 Testing versus debugging🔗
  • Testing finds that something is wrong: a test fails.

  • Debugging finds why it is wrong and fixes it: you locate the faulty line, understand the cause, and change the code.

Good tests make debugging easier. A small, focused test that fails points almost directly at the bug. A large test that fails only tells you that something, somewhere, is broken. After you fix a bug, the failing test stays in the suite as a regression test so the bug can never quietly return.

23.2.7 Testing versus formal verification🔗

There is a stronger option than testing: formal verification proves mathematically that a program meets its specification for all inputs. Static analysis tools examine code without running it. Both are powerful, and we will use some lightweight static analysis later in this lecture. But in practice:

  • Formal verification of real programs is hard. It requires a precise formal specification and a lot of expertise, so it is reserved for code that absolutely must not fail (cryptography, operating system kernels, aircraft control). The Dafny example below shows why.

  • Commercial static analysis tools must handle false positives, confusing warnings, and huge numbers of warnings. Developers often ignore tools that cry wolf.

23.2.7.1 Example: verification in Dafny🔗

Dafny is a programming language with a built-in verifier. You write a method together with its specification: requires clauses (preconditions, what the caller must guarantee) and ensures clauses (postconditions, what the method promises). The verifier then tries to prove, for every possible input, that the code meets the specification. If it succeeds, there is no need to run a single test. If it fails, it reports which promise it could not prove.

That sounds like it makes testing unnecessary. Two experiments show why it does not: the proof is only as good as the specification, and writing the proof is hard work.

Experiment 1: a proof is only as good as the specification. Here are three versions of a method that should return the larger of two integers:

// Version 1: no specification at all.

method Max1(a: int, b: int) returns (r: int)

{

  return 0;            // obviously wrong -- but there is nothing to violate

}

 

// Version 2: a weak specification.

method Max2(a: int, b: int) returns (r: int)

  ensures r >= a && r >= b

{

  if a > b {

    r := a;

  } else {

    r := b + 1;        // BUG: should be b

  }

}

 

// Version 3: the complete specification.

method Max3(a: int, b: int) returns (r: int)

  ensures r >= a && r >= b

  ensures r == a || r == b

{

  if a > b {

    r := a;

  } else {

    r := b;

  }

}

$ dafny verify max.dfy
Dafny program verifier finished with 3 verified, 0 errors

All three verify, including the two wrong ones:

  • Max1 returns 0 for every input. It has no specification, so the verifier proved that it meets no promises. That proof is true and tells us nothing.

  • Max2 returns b + 1 when b is larger. The specification only says the result is at least as large as both inputs, and b + 1 is. The code is wrong, but it meets the spec. So would r := a*a + b*b + 1000, or any other value that happens to be large enough.

  • Max3 adds the missing requirement: the result must be one of the inputs. Only now does the specification say what “maximum” means.

If we put the buggy b + 1 body back under the complete specification, the verifier rejects it:

$ dafny verify max_bad.dfy
max_bad.dfy(4,0): Error: a postcondition could not be proved on this return path
max_bad.dfy(3,17): Related location: this is the postcondition that could not be proved
  |
3 |   ensures r == a || r == b
  |                  ^^
Dafny program verifier finished with 0 verified, 1 error

Notice the parallel with testing. A weak specification is like a weak test oracle. Earlier we saw that “reverse twice gives back the list” accepts a reverse that does nothing. Here, “the result is at least as large as both inputs” accepts a max that adds one. Verification moves the hard question from “did I test enough inputs?” to “did I specify the right thing?”

Experiment 2: what does “sorted” mean? Suppose a sorting routine has a bug that overwrites one element:

input:  [1, 4, 3, 4, 5]

output: [1, 3, 3, 4, 5]     -- sorted, but one 4 became a 3

The output is in order, so a specification that only says “the output is sorted” accepts it. It also accepts something much worse:

predicate Sorted(s: seq<int>)

{

  forall i, j :: 0 <= i < j < |s| ==> s[i] <= s[j]

}

 

// Weak specification: "the output is sorted."

method SortWeak(s: seq<int>) returns (r: seq<int>)

  ensures Sorted(r)

{

  r := [];             // sorted!  ...and useless

}

The verifier accepts SortWeak: the empty sequence is sorted. A correct specification needs two properties: the output is sorted, and it is a permutation of the input (it contains the same elements, the same number of times). In Dafny, “same elements with the same counts” is written by comparing multisets:

method Sort(s: seq<int>) returns (r: seq<int>)

  ensures Sorted(r)                       // 1. in order

  ensures multiset(r) == multiset(s)      // 2. a permutation of the input

With this specification, the empty-sequence body is rejected (a postcondition could not be proved ... multiset(r) == multiset(s)), and so is the bug that turned a 4 into a 3, because the multiset {1,3,3,4,5} is not {1,3,4,4,5}.

Experiment 3: proving the real sort is hard. Here is an insertion sort that Dafny verifies against the full specification:

method Sort(s: seq<int>) returns (r: seq<int>)

  ensures Sorted(r)

  ensures multiset(r) == multiset(s)

{

  r := [];

  var i := 0;

  while i < |s|

    invariant 0 <= i <= |s|

    invariant Sorted(r)

    invariant multiset(r) == multiset(s[..i])

  {

    r := Insert(r, s[i]);

    assert s[..i+1] == s[..i] + [s[i]];   // hint for the verifier

    i := i + 1;

  }

  assert s[..|s|] == s;                    // hint for the verifier

}

 

// Insert x into the sorted sequence r, keeping it sorted.

method Insert(r: seq<int>, x: int) returns (t: seq<int>)

  requires Sorted(r)

  ensures Sorted(t)

  ensures multiset(t) == multiset(r) + multiset{x}

{

  var k := 0;

  while k < |r| && r[k] < x

    invariant 0 <= k <= |r|

    invariant forall m :: 0 <= m < k ==> r[m] < x

  {

    k := k + 1;

  }

  t := r[..k] + [x] + r[k..];

  assert r == r[..k] + r[k..];             // hint for the verifier

}

$ dafny verify sort.dfy
Dafny program verifier finished with 4 verified, 0 errors

The actual algorithm is a handful of lines. Everything else—the requires and ensures clauses of the helper, five loop invariants (facts that hold before and after every iteration), and three assert hints about how sequence slices fit together—exists only to convince the verifier. Each piece is necessary:

  • Delete the invariant forall m :: 0 <= m < k ==> r[m] < x and the verifier can no longer prove that Insert keeps the sequence sorted: Error: a postcondition could not be proved.

  • Delete the three assert hints and verification fails in three places, including Error: this invariant could not be proved to be maintained by the loop—even though the code is exactly as correct as before.

This is the trade-off in a nutshell. A verified program is correct for every input, which no test suite can promise. But you pay twice: first to write a specification that really says what you mean (Experiments 1 and 2), and then to write the invariants and hints that let the verifier prove it (Experiment 3). For a sorting function that is a pleasant exercise; for a 100,000-line web service whose requirements change every week, it is rarely practical. Students who want to learn more about specification and verification can take CMSC433.

Testing is the practical middle ground: every developer can do it, it works on any code, and a failing test is concrete evidence that is hard to argue with. And the lesson of the Dafny examples carries over directly: whether you verify or test, the quality of your checks depends on how precisely you state what the program should do.

23.2.8 Why is software hard to test?🔗
  • Huge input spaces. You cannot try every input, so you must choose a small, representative set.

  • Hidden dependencies. Databases, networks, clocks, random number generators, and LLMs make results change from run to run.

  • The oracle problem. For some programs it is hard to know what the right answer is (What is the correct output of a weather simulation? Of a chatbot?).

  • Missing requirements. Writing a test often reveals that nobody decided what the program should do in some case. This is a feature, not a bug: testing exposes gaps in the specification.

23.3 The Running Example: TerpLedger🔗

Throughout these notes we test one small system, TerpLedger: a shared-expense ledger for a student club. Members record transactions—“alice paid $23.50 for pizza, split among alice, bob, and carol”—and the ledger answers who owes whom. The complete project lives in the course repository under and is described at the end of these notes.

Module

What it does

Main testing technique

app/split.py

splits money in integer cents

unit, property-based, differential, mutation

app/parser.py

parses a one-line text format

test design, fuzzing, round-trip property

app/ledger.py

in-memory and SQLite stores

differential (contract) tests

app/categorize.py

asks an LLM for an expense category

test doubles (fakes, spies)

app/main.py

FastAPI web API

integration tests

whole repository

—

coverage, static analysis, CI

A transaction record looks like this:

2026-09-12,Pizza night,2350,alice,alice;bob;carol

meaning: on September 12, alice paid 2350 cents ($23.50) for “Pizza night,” to be split among alice, bob, and carol. Amounts are integer cents. This is deliberate: floating-point numbers cannot represent 0.10 exactly (try 0.1 + 0.2 == 0.3 in Python), and money code written with floats is a classic source of bugs.

The heart of the system is one function. Splitting 2350 cents three ways cannot give everyone the same share—someone has to absorb the extra cent—so the specification says: shares differ by at most one cent, sum exactly to the total, and earlier participants absorb the leftovers.

def split_evenly(total_cents: int, n_people: int) -> list[int]:
    """Split `total_cents` among `n_people` as evenly as possible.

    Guarantees:
      * the shares sum to exactly `total_cents`,
      * no two shares differ by more than one cent,
      * the people listed first absorb the leftover cents.

    Raises ValueError if `total_cents` is negative or `n_people` < 1.
    """
    if n_people < 1:
        raise ValueError("need at least one person to split between")
    if total_cents < 0:
        raise ValueError("total must be non-negative")
    base, leftover = divmod(total_cents, n_people)
    return [base + 1 if i < leftover else base for i in range(n_people)]

For example, split_evenly(1000, 3) computes divmod(1000, 3) = (333, 1), so the first person gets 334 and the others get 333: [334, 333, 333]. Keep this function in mind; we will test it in many different ways and then ask how good those tests are.

23.4 Levels and Types of Testing🔗

23.4.1 Testing scale: unit, integration, system🔗

Tests differ by how much of the system one test exercises:

  • Unit testing checks individual functions or classes in isolation. Example: split_evenly(1000, 3) == [334, 333, 333].

  • Integration testing checks that several components work together. Example: posting a transaction to the web API stores it and updates /balances—which exercises routing, validation, the parser, the ledger, and the arithmetic together.

  • System (end-to-end) testing checks the entire application from the user’s point of view. Example: in a browser, open the TerpLedger web page, add a transaction, and see the updated balances on screen.

  • Acceptance testing checks whether the system meets the user’s needs. Example: the club treasurer tries TerpLedger for a month and confirms it matches the bank statement. This is validation, not just verification.

23.4.2 The V model🔗

The V model pairs each phase of development with the level of testing that checks it. Going down the left side, we refine customer needs into code; going up the right side, we test each level against the phase that produced it:

Requirements analysis ·························► Acceptance testing

   (customer needs)                            (user's needs met?)

        │                                               ▲

        ▼                                               │

  Architectural design ···················► System testing

   (components, connections)              (assembled system meets spec?)

            │                                       ▲

            ▼                                       │

      Subsystem design ·············► Integration testing

       (structure, behavior)          (do components work together?)

                │                               ▲

                ▼                               │

          Implementation ·········► Unit testing

             (code!)                (individual functions, classes)

There are many variants of this model, but the core idea is useful everywhere: every decision you make while building a system should have a test that checks it.

23.4.3 Regression testing🔗

Definition. Regression testing re-runs existing tests after a change to check that previously working behavior still works.

Regression testing is not a separate level; it is a way of using tests of every level. Every test you write becomes a regression test the moment it passes. Suppose a teammate (or an LLM) “simplifies” split_evenly to:

return [total_cents // n_people] * n_people

The old unit tests immediately fail:

FAILED tests/test_split.py::test_split_gives_leftover_cents_to_first_people

FAILED tests/test_split.py::test_split_examples[1-2-expected0] - assert [0, 0...

FAILED tests/test_split.py::test_split_examples[5-4-expected1] - assert [1, 1...

FAILED tests/test_split.py::test_split_examples[200-6-expected3] - assert [33...

4 failed, 9 passed in 0.02s

The “simplification” loses the leftover cents. Without the regression suite, money would silently disappear from the ledger.

23.4.4 The test pyramid🔗

How many tests of each level should you write?

     ╱ E2E ╲           few, slow, realistic

    ╱────────╲

   ╱integration╲       some, slower

  ╱──────────────╲

 ╱   unit tests   ╲    many, fast, precise

╱──────────────────╲

This is the test pyramid. Unit tests are cheap to run and point directly at the broken function, so you can afford thousands. End-to-end tests are slow, sometimes flaky, and vague about where a failure is, so you want only a few. The shape is an economic argument, not a law: spend most of your effort where tests are cheap and precise.

23.4.5 Testing by purpose🔗

So far we classified tests by scale. We can also classify them by what quality they check:

Type

Question it answers

TerpLedger example

Functional

Does it compute the right result?

balances sum to zero

Performance

Is it fast enough under load?

/balances answers in under 100 ms with 100,000 transactions

Security

Can an attacker misuse it?

a malformed import line cannot crash the server

Usability

Can real users use it easily?

a new club member can record a purchase without help

Availability

Is it up when needed?

the deployed service answers /health

Most of this lecture is about functional testing, because that is where developers spend most of their testing effort.

23.4.6 When are tests written?🔗
  • Test first (test-driven development, TDD). Write a failing test before the code, then write just enough code to pass it. We will walk through an example below.

  • Test after. Write the code, then write tests to check whether it works. This is common, but it has a trap: tests written after the code tend to check what the code does rather than what it should do.

  • Iteration. Either way, development is a loop: change the code, re-run the tests (retesting), and clean up the design (refactoring) with the tests as a safety net.

23.5 Designing Good Test Cases🔗

Writing a test is easy. Choosing which tests to write is the real skill. Random hand-picked examples tend to test the same cases the author already thought about. This section gives systematic techniques.

23.5.1 A warm-up: withdraw🔗

Consider this function:

def withdraw(balance, amount):
    if amount <= balance:
        return balance - amount
    return balance

What test cases should we write?

Balance

Amount

Expected result

Why this case?

100

50

50

typical withdrawal

100

100

0

boundary: withdraw everything

100

101

100

boundary: one cent too much

100

0

100

boundary: zero amount

100

-10

?

invalid input

The last case is the interesting one. The code returns 110: a negative “withdrawal” becomes a deposit. Is that a bug? The code does not say, and neither does any specification. To write the expected result, we had to decide what the function should do—and discovered that nobody had decided. Testing exposes missing requirements, not only bugs.

23.5.2 Equivalence partitioning🔗

Definition. Equivalence partitioning divides the input domain into classes (partitions) such that the program is expected to behave the same way for every input in a class. You then test at least one representative from each class.

In plain English: if the program treats 37 and 52 the same way, testing both tells you little more than testing one. Spend your tests on inputs that are treated differently.

Example. The parser’s amount field (the third field of a record) must be a positive integer number of cents. Its partitions:

Partition

Representative

Expected

positive integer

2350

accepted

zero

0

ParseError

negative integer

-5

ParseError

decimal number

12.50

ParseError

not a number

abc

ParseError

empty

(empty string)

ParseError

Partitions also apply to the structure of the input. For the whole record: exactly five fields (valid), fewer than five, more than five; a valid date, an unparseable date (yesterday), an impossible date (2026-02-30); one participant, many participants, an empty name in the list (bob;;carol).

23.5.3 Boundary value analysis🔗

Definition. Boundary value analysis tests inputs at, just below, and just above the edges of each partition.

Intuition. Bugs cluster at boundaries because that is where programmers write comparisons, and comparisons are easy to get slightly wrong: < instead of <=, an off-by-one loop bound, a forgotten empty case.

Example. The boundary between “rejected” and “accepted” amounts is between 0 and 1. Boundary value analysis says to test both sides: 0 must be rejected and 1 must be accepted. Keep this example in mind: later, mutation testing will show that TerpLedger’s real test suite tests 0 but forgets to test 1.

For split_evenly, the boundaries are:

  • n_people: 0 (error), 1 (one person gets everything), 2.

  • total_cents: -1 (error), 0 (all zeros), 1 (one cent to share).

  • The leftover: a total divisible by n_people (no leftover), and one that leaves n_people - 1 leftover cents (the most leftover possible).

23.5.4 Edge cases, positive and negative tests🔗

Edge cases are unusual but legal inputs: empty lists, a single element, very large numbers, duplicate names, Unicode names, leap days. They are not necessarily at a numeric boundary, but they are cases the author was probably not picturing.

Positive tests check that valid input produces the right result. Negative tests check that invalid input is handled correctly—usually by raising a specific, documented error. Beginners write mostly positive tests. Bugs, and security holes, live mostly in the negative cases.

23.5.5 Test oracles🔗

Every test needs an expected result, and something must decide what it is. Common oracles:

  • The specification. The docstring says shares sum to the total; we compute the expected list by hand.

  • A reference implementation. A slow but obviously correct version of the same function. (This is differential testing, covered later.)

  • A property. We may not know the exact output, but we know facts that must hold, such as “the balances sum to zero.” (This is property-based testing.)

  • An implicit oracle. “The program does not crash.” Weak, but it needs no knowledge of the right answer at all. (This is what fuzzing uses.)

Common misconception. “The expected value is whatever the code returns.” If you run the code, copy its output into the test, and call it the expected value, the code is its own oracle. The test will pass forever, including when the code is wrong. This mistake will come back when we discuss AI-generated tests.

23.6 Testing Frameworks and Automation🔗

Testing by hand—running the program and looking at the output—does not scale and is not repeatable. A testing framework discovers tests, runs them, reports failures clearly, and provides tools for setup, teardown, and replacing dependencies. Every language has one:

Language

Frameworks

Python

pytest, unittest (standard library)

Java

JUnit

JavaScript / TypeScript

Jest, Vitest

OCaml

OUnit, Alcotest

The ideas are the same in all of them. We use pytest, the de facto standard in Python.

23.6.1 Assertions and the Arrange–Act–Assert pattern🔗

pytest’s rules are minimal: files named test_*.py, functions named test_*, and plain assert statements. Here are real tests from TerpLedger’s tests/test_split.py:

from app.split import split_evenly


def test_split_divides_evenly_when_possible():
    assert split_evenly(900, 3) == [300, 300, 300]


def test_split_gives_leftover_cents_to_first_people():
    # 1000 / 3 = 333.33...; the extra cent goes to the first person.
    assert split_evenly(1000, 3) == [334, 333, 333]


def test_split_single_person_gets_everything():
    assert split_evenly(2350, 1) == [2350]


def test_split_zero_total_is_all_zeros():
    assert split_evenly(0, 4) == [0, 0, 0, 0]

Each test follows the Arrange–Act–Assert pattern: arrange the inputs and state, act by calling the code under test, assert on the result. Here all three steps fit on one line; in larger tests they appear as three visible paragraphs.

Notice which cases were chosen: a typical case (divides evenly), the interesting case (leftover cents), and two boundaries (one person, zero total). This is the test design from the previous section, put into code.

23.6.2 Reading a failure🔗

Run pytest against the broken “simplification” [total_cents // n_people] * n_people:

    def test_split_gives_leftover_cents_to_first_people():

>       assert split_evenly(1000, 3) == [334, 333, 333]

E       assert [333, 333, 333] == [334, 333, 333]

E

E         At index 0 diff: 333 != 334

 

tests/test_split.py:21: AssertionError

pytest rewrites assert so a failure shows the actual values on both sides. Read failures top to bottom: which test, which line, expected versus actual. The test name also helps: it tells you which promise was broken before you read any code. Name tests after behaviors (test_split_gives_leftover_cents_to_first_people), not after functions (test_split_1).

23.6.3 Parameterized tests🔗

When the same check applies to many input/output pairs, do not copy-paste test functions. Give one test a table of cases:

import pytest


@pytest.mark.parametrize(
    "total,n,expected",
    [
        (1, 2, [1, 0]),
        (5, 4, [2, 1, 1, 1]),
        (7, 7, [1, 1, 1, 1, 1, 1, 1]),
        (200, 6, [34, 34, 33, 33, 33, 33]),
    ],
)
def test_split_examples(total, n, expected):
    assert split_evenly(total, n) == expected

pytest runs this as four separate tests and reports each one. Parametrization is the natural way to write down an equivalence partitioning table. Here is the parser’s negative-test table, taken almost directly from the partitions above:

@pytest.mark.parametrize(
    "bad_line",
    [
        "",                                         # no fields at all
        "2026-09-12,Pizza,2350,alice",              # only 4 fields
        "2026-09-12,Pizza,2350,alice,bob,extra",    # 6 fields
        "yesterday,Pizza,2350,alice,bob",           # bad date
        "2026-02-30,Pizza,2350,alice,bob",          # impossible date
        "2026-09-12,Pizza,12.50,alice,bob",         # amount not integer cents
        "2026-09-12,Pizza,-5,alice,bob",            # negative amount
        "2026-09-12,Pizza,0,alice,bob",             # zero amount
        "2026-09-12,,2350,alice,bob",               # empty description
        "2026-09-12,Pizza,2350,,bob",               # empty payer
        "2026-09-12,Pizza,2350,alice,",             # empty participants
        "2026-09-12,Pizza,2350,alice,bob;;carol",   # empty name in list
    ],
)
def test_parse_rejects_malformed_input(bad_line):
    with pytest.raises(ParseError):
        parse_transaction(bad_line)

pytest.raises fails the test if the exception is not raised. Note that it names the specific exception. pytest.raises(Exception) would also pass if the function crashed with an unrelated IndexError—a weak assertion we will see again in AI-generated tests.

23.6.4 Fixtures: setup and teardown🔗

A fixture is reusable setup that a test requests by naming it as a parameter. pytest calls the fixture and passes the result in. If the fixture uses yield, the code after yield is teardown: it runs after the test, whether the test passed or failed.

@pytest.fixture
def client():
    # A fresh app per test: no state leaks from one test to the next.
    return TestClient(create_app(llm=KeywordLLM()))


def test_health(client):    # pytest sees the parameter name and injects the fixture
    assert client.get("/health").json() == {"status": "ok"}

Every test that names client gets a brand-new application with an empty ledger. If all tests shared one app, a transaction added by one test would change the balances seen by the next, and the suite would pass or fail depending on the order the tests ran.

A fixture with teardown looks like this:

@pytest.fixture
def db():
    ledger = SqliteLedger(":memory:")   # setup: a private throwaway database
    yield ledger                         # the test runs here
    ledger.close()                       # teardown: runs even if the test failed

pytest also ships built-in fixtures, such as tmp_path (a fresh temporary directory) and monkeypatch (temporarily replace an attribute or environment variable; undone automatically after the test).

23.6.5 Test suites and test organization🔗

A test suite is a collection of tests run together. In pytest, the suite is simply every test_* function pytest finds under the configured directory. TerpLedger configures this in pyproject.toml:

[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
markers = [
    "live: talks to a real external service (an LLM); skipped unless enabled",
]

You can run subsets: pytest tests/test_split.py (one file), pytest -k leftover (tests whose names match), or pytest -m live (tests with a marker). Markers let slow or external tests be skipped by default and run on purpose.

23.6.6 Mocking and stubbing: test doubles🔗

Some code depends on things you cannot use in a fast, repeatable test: a payment service, the current time, a database server, an LLM. You replace such a dependency with a test double: an object with the same interface but controlled behavior. The word “mock” is often used loosely for all of them, but the precise vocabulary is useful:

Double

What it does

Example

stub

returns canned answers; no logic

a fake LLM that always answers "food"

fake

a simplified but working implementation

TerpLedger’s in-memory Ledger instead of a database; KeywordLLM, which picks a category by keyword

spy

a stub or fake that also records how it was called

ScriptedLLM, which records every prompt it receives

mock

pre-programmed with expectations; fails the test if they are violated

an LLM double that raises AssertionError if it is ever called

Each double answers a different question. A stub lets you test what your code does with an answer. A spy lets you test what your code asked. A mock lets you test what your code must never do. A fake lets a whole subsystem run without its real dependency.

TerpLedger’s categorizer asks an LLM for an expense category. A real LLM is slow, costs money, and gives different answers on different runs, so the unit tests use a scripted spy instead:

class ScriptedLLM:
    """Returns pre-scripted replies in order and records every prompt."""

    def __init__(self, *replies: str) -> None:
        self._replies = list(replies)
        self.prompts: list[str] = []

    def complete(self, prompt: str) -> str:
        self.prompts.append(prompt)
        if not self._replies:
            raise AssertionError("ScriptedLLM ran out of replies -- unexpected extra call")
        return self._replies.pop(0)

With it, a test can check how our code handles a messy reply such as "Food." or "groceries", and can check that the prompt mentioned the transaction’s description—deterministically, in milliseconds, with no model running.

How does the double get into the code? Through a seam: a place where a dependency can be swapped. The cleanest seam is dependency injection: make the dependency a parameter. TerpLedger builds its web app with create_app(ledger=..., llm=...) precisely so tests can pass in fakes.

Common misconception. “More mocking means better tests.” Tests that mock everything end up checking that the code calls its collaborators in a particular order. They test the implementation, not the behavior, and they keep passing when the real collaborator changes. Prefer real objects when they are fast and deterministic, prefer fakes to mocks, and mock only at boundaries you do not own.

23.6.7 What makes a test good?🔗
  • Fast. Milliseconds. The whole TerpLedger suite (74 tests, including API tests) runs in under a second. A slow suite stops being run.

  • Isolated. No test depends on another test having run first, on the network, or on shared state.

  • Deterministic. Same code, same verdict, every time. A test that sometimes fails for no reason is flaky; flaky tests teach people to ignore failures.

  • One behavior per test. When it fails, you know exactly which fact stopped being true.

  • Tests behavior, not implementation. Assert on what the function returns, not on how it computes it.

  • Seen to fail. A test you have never seen fail is a test you do not know works. Break the code on purpose once and watch the test go red.

23.7 Test-Driven Development🔗

Definition. Test-driven development (TDD) is a development process in which you write a failing test before writing the code that makes it pass, in short cycles of red–green–refactor.

     ┌──────────────────────────────────────────┐

     ▼                                          │

┌─────────┐      ┌─────────┐      ┌───────────┐ │

│   RED   │ ───► │  GREEN  │ ───► │ REFACTOR  │─┘

│ write a │      │ write   │      │ clean up; │

│ failing │      │ just    │      │ tests     │

│ test    │      │ enough  │      │ stay      │

└─────────┘      │ code    │      │ green     │

                 └─────────┘      └───────────┘

Intuition. Writing the test first forces you to decide what the code should do—its inputs, outputs, and edge cases—before you get attached to an implementation. It also guarantees every test has been seen to fail at least once.

Worked example. The club wants to know how much each person has paid out of pocket. Red: we write the tests first, in tests/test_split.py:

from app.split import total_paid_by


def _tx(amount, payer):
    return Transaction(datetime.date(2026, 9, 12), "x", amount, payer, ("alice", "bob"))


def test_total_paid_by_sums_only_that_payer():
    txs = [_tx(2350, "alice"), _tx(500, "bob"), _tx(150, "alice")]
    assert total_paid_by(txs, "alice") == 2500


def test_total_paid_by_unknown_person_is_zero():
    assert total_paid_by([_tx(2350, "alice")], "zoe") == 0

Running pytest fails, as it should—the function does not exist yet:

E   ImportError: cannot import name 'total_paid_by' from 'app.split'

Notice that writing the second test forced a design decision: a person who never paid gets 0, not an error. Green: write just enough code to pass:

def total_paid_by(transactions: Iterable[Transaction], person: str) -> int:
    """Total cents `person` has paid out of pocket."""
    return sum(t.amount_cents for t in transactions if t.payer == person)

2 passed in 0.05s

Refactor: if the code had duplication or poor names, we would clean it up now, re-running the tests after every change. Here it is already simple, so we move on to the next test.

Common misconception. “TDD means writing all the tests up front.” No: TDD is one small test at a time, each followed immediately by the code that passes it.

23.8 Property-Based Testing🔗

Every unit test above hard-codes both the input and the expected output. That has a built-in blind spot: you only test the inputs you thought of, and you are likely to think of the same inputs the implementer thought of. The bug lives in the inputs nobody thought of.

Definition. A property-based test states a property—a boolean condition that must hold for every input in some domain—together with a generator that produces inputs from that domain. The framework repeatedly generates random inputs, checks the property on each, and reports any input that makes it false.

                  ┌──────────── true ───────────┐

                  ▼                             │

───────►  ┌────────────────┐           ┌────────┴────────┐   false

          │ Generate input │ ────────► │ Property(input)?│ ────────►  BUG!

          └────────────────┘           └─────────────────┘   (shrink and report)

In plain English: instead of “split_evenly(1000, 3) returns [334, 333, 333],” you say “for any total and any group size, the shares add up to the total”—and the computer goes hunting for a counterexample.

Intuition. A unit test is a point check; a property is a law. You stop being the person who picks the test inputs (you are biased) and become the person who states what must always be true.

This idea started with QuickCheck for Haskell (Claessen and Hughes, 2000). There are now versions for every language: QCheck for OCaml, junit-quickcheck and jqwik for Java, fast-check for TypeScript, and Hypothesis for Python, which we use here.

23.8.1 A first property: reversing a list🔗

Here is the classic first example. Reversing a list twice gives back the original list:

from hypothesis import given
from hypothesis import strategies as st


def reverse(xs):
    return xs[::-1]


@given(xs=st.lists(st.integers()))
def test_reverse_twice_is_identity(xs):
    assert reverse(reverse(xs)) == xs

st.lists(st.integers()) is a strategy: Hypothesis’s name for a generator. The @given decorator turns the function into a test that runs on (by default) 100 generated lists. The same test in Java with junit-quickcheck looks like this:

@RunWith(JUnitQuickcheck.class)
public class PBT {
    @Property(trials = 1000)                    // test 1000 times
    public void testList(List<String> l1) {     // a random string list is generated
        List<String> l2 = new ArrayList<>(l1);
        Collections.reverse(l2);
        Collections.reverse(l2);
        assertEquals(l1, l2);                    // ...and the property is checked
    }
}

23.8.2 A weak property can miss a bug🔗

Now consider a buggy reverse that returns the list unchanged:

def reverse_buggy(xs):
    return list(xs)      # BUG: does not reverse at all

The property reverse_buggy(reverse_buggy(xs)) == xs still holds for every list! Doing nothing twice is the same as doing nothing. The property passes all 100 runs and the bug survives. Meanwhile, a single ordinary unit test catches it:

def test_reverse_example():
    assert reverse_buggy([1, 2, 3]) == [3, 2, 1]

E       assert [1, 2, 3] == [3, 2, 1]

Two lessons. First, a property is only as good as what it says. “Reverse twice is identity” is true of reverse, but also of the identity function and many others. Second, property tests complement unit tests; they do not replace them.

A stronger property pins reverse down more tightly: reversing a concatenation reverses the order of the pieces.

@given(l1=st.lists(st.integers()), x=st.integers(), l2=st.lists(st.integers()))
def test_reverse_distributes(l1, x, l2):
    assert reverse(l1 + [x] + l2) == reverse(l2) + [x] + reverse(l1)

For example, reverse([1, 2] + [3] + [4, 5]) must equal reverse([4, 5]) + [3] + reverse([1, 2]), that is, [5, 4, 3, 2, 1]. Against the buggy version, Hypothesis finds a counterexample immediately:

E       assert [0, 1] == [1, 0]

E       Failing test case: test_reverse_distributes_buggy(

E           l1=[],

E           x=0,

E           l2=[1],

E       )

23.8.3 Hypothesis hunts a TerpLedger bug🔗

Back to our running example. Here is a property from TerpLedger’s tests/test_properties.py:

totals = st.integers(min_value=0, max_value=10_000_000)
group_sizes = st.integers(min_value=1, max_value=200)


@given(total=totals, n=group_sizes)
def test_shares_sum_to_total(total, n):
    """Conservation of money: not a cent appears or disappears."""
    assert sum(split_evenly(total, n)) == total

Against the broken [total // n] * n version:

E       assert 0 == 1

E        +  where 0 = sum([0, 0])

E        +    where [0, 0] = split_evenly(1, 2)

E       Failing test case: test_shares_sum_to_total(

E           total=1,

E           n=2,

E       )

Hypothesis found the bug without anyone choosing inputs, and it reported the simplest possible failing case: splitting one cent between two people loses the cent. Debugging that is easy; debugging “splitting 8,392,107 cents among 173 people is short by 134” is not. That simplification is called shrinking.

Common misconception. “I can’t write a property test because I can’t compute the expected output for a random input.” You do not need to. The property above never says what split_evenly(7, 3) equals. Properties constrain the output without specifying it.

23.8.4 Shrinking🔗

Definition. Shrinking is the process of automatically simplifying a failing input to the smallest or simplest input that still triggers the failure.

Suppose a function that totals a list of amounts forgets to reject negative refunds, and the property is “the total is never negative.” The first failing input Hypothesis stumbles on may be large and ugly. Shrinking then repeatedly tries simpler inputs, keeping each one that still fails:

  [123, -999999999999, 5, 0, 5]     fails   (first random failure)

→ [123, -1, 5, 0, 5]                fails   (shrink a number toward 0)

→ [123, -1]                         fails   (remove elements)

→ [-1]                              fails   (remove more)

→ []                                passes  -- too far, so stop at [-1]

Running this example with Hypothesis reports exactly that:

E       assert -1 >= 0

E        +  where -1 = total_spent([-1])

E       Failing test case: test_total_never_negative(

E           amounts=[-1],

E       )

Shrinking strategies depend on the data type:

  • Numbers: try smaller magnitudes, or 0.

  • Lists: try removing elements or shortening the list.

  • Strings: try shorter substrings or simpler characters.

  • Custom objects: shrink their fields recursively. For a tree: shrink the value at each node, remove subtrees, replace a node with one of its subtrees, and shrink recursively within subtrees.

23.8.5 Generators🔗

A generator creates input data for the system under test. Good generators do not produce uniformly random values. Hypothesis deliberately over-samples boundaries—0, 1, the minimum and maximum, empty lists, strange Unicode—because that is where bugs live.

Generators compose like the data they describe. TerpLedger builds a generator of whole transactions from generators of dates, names, and amounts:

names = st.text(alphabet="abcdefghijklmnopqrstuvwxyz", min_size=1, max_size=8)

transactions = st.builds(
    Transaction,
    date=st.dates(min_value=datetime.date(2000, 1, 1), max_value=datetime.date(2099, 12, 31)),
    description=names,
    amount_cents=st.integers(min_value=1, max_value=1_000_000),
    payer=names,
    participants=st.lists(names, min_size=1, max_size=10).map(tuple),
)


@given(txs=st.lists(transactions, max_size=20))
def test_balances_sum_to_zero(txs):
    """Every credit is someone else's debit, so the books balance."""
    assert sum(balances(txs).values()) == 0

23.8.6 A catalog of properties🔗

The hard part of property-based testing is thinking of properties. These patterns cover most cases:

Pattern

Meaning

TerpLedger example

Invariant

some quantity is preserved or bounded

sum(shares) == total; max(shares) - min(shares) <= 1; len(shares) == n

Round-trip

encoding then decoding gives back the input

parse_transaction(format_transaction(t)) == t

Conservation

totals balance across a whole system

balances sum to zero

Oracle

agrees with a trusted reference implementation

agrees with a one-cent-at-a-time split (next section)

Idempotence

doing it twice equals doing it once

normalize(normalize(s)) == normalize(s) for LLM category text

Robustness

never fails in an undocumented way

the parser raises only ParseError on any string

The round-trip property deserves a second look. This one test checks every field of the format in combination:

@given(t=transactions)
def test_parse_inverts_format(t):
    """Round-trip: formatting then parsing gives back the same value."""
    assert parse_transaction(format_transaction(t)) == t

Anything with a pair of inverse operations—save/load, encode/decode, to-JSON/from-JSON—supports a round-trip property.

The robustness property is the weakest useful one: it says nothing about the value, only that the code never explodes unexpectedly. Hold on to it; it is the bridge to fuzzing.

@given(line=st.text(max_size=200))
def test_parser_never_raises_anything_but_parse_error(line):
    try:
        parse_transaction(line)
    except Exception as e:  # noqa: BLE001 -- the broad catch is the point
        assert type(e).__name__ == "ParseError"

23.8.7 Seeds and reproducibility🔗

A random test that fails on Tuesday and passes on Wednesday is flaky. Property-based testing frameworks control randomness with a seed, the source of randomness that determines which inputs are generated. Hypothesis also keeps a database of past failures (the .hypothesis/ directory) and replays them first on the next run, so a found bug stays found. For CI, TerpLedger uses a derandomized profile so the same commit always gets the same verdict:

settings.register_profile("dev", max_examples=100)
settings.register_profile("ci", max_examples=300, derandomize=True, deadline=None)
settings.register_profile("nightly", max_examples=20_000, deadline=None)
settings.load_profile(os.environ.get("HYPOTHESIS_PROFILE", "dev"))

When Hypothesis finds a bug, copy the shrunk counterexample into an @example(total=1, n=2) decorator above @given. It then runs every time, forever: a permanent regression test.

23.9 Differential Testing🔗

Definition. Differential testing detects bugs by running the same input through multiple implementations of the same functionality and comparing their outputs. Any disagreement means at least one implementation is wrong.

                ┌─────────────┐     ┌────────┐

           ┌──► │ Application │ ──► │ output │ ──┐

           │    └─────────────┘     └────────┘   │    ┌────────────┐

┌───────┐  │                                     ├──► │ Comparator │

│ Input │──┤                                     │    └────────────┘

└───────┘  │    ┌─────────────┐     ┌────────┐   │

           └──► │   Oracle    │ ──► │ output │ ──┘

                └─────────────┘     └────────┘

Intuition. Differential testing solves the oracle problem by letting a second implementation be the oracle. You do not need to know the right answer; you only need two programs that should agree.

Worked example 1: a slow reference implementation. The obviously correct way to split money is to hand out cents one at a time, round-robin. It is far too slow for real use, but it is easy to trust—which makes it a perfect oracle for the fast divmod version:

def split_one_cent_at_a_time(total_cents, n_people):
    shares = [0] * n_people
    for i in range(total_cents):
        shares[i % n_people] += 1      # hand out one cent to the next person
    return shares


@given(total=st.integers(0, 5_000), n=st.integers(1, 50))
def test_split_agrees_with_oracle(total, n):
    assert split_evenly(total, n) == split_one_cent_at_a_time(total, n)

Trace (1000, 3) through the oracle: cent 0 goes to person 0, cent 1 to person 1, cent 2 to person 2, cent 3 to person 0, and so on. Cent 999 goes to person 999 % 3 = 0, so person 0 ends with 334 and the others with 333: [334, 333, 333], matching split_evenly. Unlike the sum property, this test also checks who gets the leftover cents. This pattern is especially valuable when optimizing: keep the old, slow version as the oracle for the new, fast one.

Worked example 2: two storage backends. TerpLedger has an in-memory Ledger (used in fast tests) and a SqliteLedger (used in production). The in-memory one is only a legitimate stand-in if the two behave the same. A parametrized fixture runs one suite against both:

@pytest.fixture(params=["memory", "sqlite"])
def store(request):
    if request.param == "memory":
        yield Ledger()
    else:
        db = SqliteLedger(":memory:")
        yield db
        db.close()


def test_add_many_is_atomic(store):
    bad = Transaction(PIZZA.date, "Refund", -100, "alice", ("bob",))
    with pytest.raises((ValueError, sqlite3.IntegrityError)):
        store.add_many([PIZZA, bad])
    assert store.all() == []   # all or nothing: the pizza must NOT be stored

This kind of suite is often called a contract test. A true story from building TerpLedger: the first in-memory add_many simply looped over add. SQLite rejected the negative refund inside a database transaction and stored nothing; the in-memory version stored the pizza and then accepted the refund. The fake was more permissive than the real thing, so tests that used it would have passed on code that production rejects. Running the same test against both implementations exposed the difference.

Worked example 3: Csmith. Csmith is a famous differential testing tool for C compilers. It generates random, valid C programs, compiles each one with several compilers, runs them, and compares the results:

gcc   test.c -o a.out
clang test.c -o b.out
tcc   test.c -o c.out
# run all three; if the outputs differ, at least one compiler has a bug

Csmith found hundreds of bugs in GCC and Clang—mature, heavily used compilers.

23.10 Code Coverage🔗

We now have many tests. Are they enough? The first, cheapest way to grade a test suite is to ask which parts of the program it never even runs.

Definition. Code coverage measures how much of a program is executed by a test suite, according to some criterion:

  • Function coverage: has each function been called?

  • Statement (line) coverage: has each statement been executed?

  • Branch coverage: has each branch of each control structure (both sides of every if, every loop entered and skipped) been executed?

  • Condition (predicate) coverage: has each boolean sub-expression evaluated to both true and false?

There are many more criteria, but these four are the most common.

Intuition. Coverage is a flashlight. It cannot tell you that the lit parts of the room are clean, but it shows you exactly which corners are dark—and a bug in a dark corner is guaranteed to go unnoticed.

23.10.1 Measuring coverage🔗

In Python, coverage.py records which lines and branches run, and the pytest-cov plugin connects it to pytest. TerpLedger’s configuration:

[tool.coverage.run]
source = ["app"]
branch = true
omit = ["app/parser_v1.py"]   # the deliberately buggy draft is not shipped code

[tool.coverage.report]
show_missing = true
fail_under = 90

A real run:

$ pytest --cov
Name                Stmts   Miss Branch BrPart  Cover   Missing
---------------------------------------------------------------
app/__init__.py         0      0      0      0   100%
app/categorize.py      36      0      4      0   100%
app/ledger.py          53      1      6      0    98%   59
app/main.py            59      0      4      0   100%
app/models.py          10      0      0      0   100%
app/parser.py          38      3     20      3    90%   75, 77, 79
app/split.py           19      0      8      0   100%
---------------------------------------------------------------
TOTAL                 215      4     42      3    97%
Required test coverage of 90.0% reached. Total coverage: 97.28%

Stmts/Miss are statements and missed statements; Branch/BrPart are branches and partially covered decisions (one side taken, never the other); Missing lists the dark lines. Each one is worth investigating:

  • ledger.py line 59 is Ledger.add_line. No test calls it—and neither does any production code. Coverage found dead code. The right fix is to delete it, not to write a test for it.

  • parser.py lines 75, 77, 79 are the three raise ValueError lines in format_transaction. The round-trip property only generates valid transactions, so the error paths never run. Coverage found a genuinely untested part of the contract.

23.10.2 Statement, branch, and condition coverage compared🔗

Statement coverage can be fooled by any one-line conditional. Consider the last line of split_evenly:

return [base + 1 if i < leftover else base for i in range(n_people)]

The single test split_evenly(900, 3) executes this line, giving 100% statement coverage. But leftover is 0, so the base + 1 side never runs. The leftover-cent logic—the whole reason the function is interesting—is “covered” and untested. Branch coverage reports the untaken side. This is why TerpLedger sets branch = true.

Condition coverage looks inside a decision. The formatter checks:

if not text or any(c in text for c in ",;\n\r"):
    raise ValueError(...)

Branch coverage is satisfied by two tests: one where the whole condition is true and one where it is false. For example, an empty description (the left side is true) and a normal description (both sides false). Condition coverage additionally requires the right side to be true at least once—a description containing a comma—so that each sub-expression is seen both ways.

Test input

not text

contains delimiter

whole condition

""

true

(not evaluated)

true

"Pizza"

false

false

false

"Pizza, large"

false

true

true

The first two rows give full branch coverage. All three are needed for condition coverage.

23.10.3 What coverage cannot tell you🔗

Common misconception. “100% coverage means the code is fully tested.” Coverage measures what the tests ran, not what they checked. This test achieves 100% statement coverage of split_evenly’s happy path and catches nothing:

def test_split_runs():
    split_evenly(1000, 3)   # executes every line; asserts nothing

Practical implications.
  • Use coverage negatively: a line at 0% is definitely untested; go look at it. Do not use it as proof of quality.

  • A coverage floor in CI (fail_under = 90) stops coverage from silently eroding. Do not make 100% a target: “when a measure becomes a target, it ceases to be a good measure” (Goodhart’s law), and people respond to coverage targets with assertion-free tests.

  • Coverage is not only a report; it can also drive test generation, as we will see in coverage-guided fuzzing.

23.11 Mutation Testing🔗

Coverage asks “did the tests run this code?” Mutation testing asks the question we actually care about: “if this code were wrong, would the tests notice?”

Definition. Mutation testing evaluates a test suite by modifying the program in small ways. Each modified copy, called a mutant, contains one small deliberate fault produced by a mutation operator. The suite is run against each mutant. A mutant is killed if at least one test fails and survives if all tests pass. The mutation score is

mutation score = killed mutants / (total mutants - equivalent mutants)

where an equivalent mutant changes the code but not its behavior, so no test could ever kill it.

In plain English: plant a bug, run the tests, and see if they go red. Repeat for every bug you can plant mechanically. Every surviving mutant is a bug your suite would let into production.

Intuition. Mutation testing tests the tests. It is a fire drill: set small, controlled fires and check whether the alarm rings. Coverage only tells you the alarm has power.

23.11.1 Mutants and mutation operators🔗

Here is a mutant produced by the condition mutation operator, which replaces && with ||:

original                    mutant

if (a && b)                 if (a || b)

  { c = 1; }                  { c = 1; }

else                        else

  { c = 0; }                  { c = 0; }

A test suite that kills this mutant must include an input where a and b differ (say a true, b false): the original sets c = 0, the mutant sets c = 1. If no test has such an input, the mutant survives, and we have learned exactly which test is missing.

Researchers have explored many mutation operators. Common ones for imperative languages:

  • Statement deletion.

  • Statement duplication or insertion (Apple’s famous “goto fail;” security bug was a duplicated line).

  • Replacing boolean sub-expressions with true or false.

  • Replacing arithmetic operators: + with *, - with /, and so on.

  • Replacing relational operators: > with >=, == with <=, and so on.

  • Replacing constants: 0 with 1.

  • Removing a method body.

Each operator models a mistake real programmers make: an off-by-one comparison, a sign error, a wrong constant. The competent programmer hypothesis says real code is usually close to correct, and the coupling effect says tests that catch these simple faults tend to catch complex ones too. That is why mechanically planted one-character bugs are a good stand-in for real ones.

23.11.2 A tiny mutation tester🔗

Industrial tools exist (mutmut and cosmic-ray for Python, PIT for Java, Stryker for JavaScript), but the idea is simple enough that TerpLedger ships its own, mutation/mutate.py, in about 120 lines. It works on the program’s abstract syntax tree, the parsed tree form of the source code. Its mutation operators:

BINOP_SWAPS = {ast.Add: ast.Sub, ast.Sub: ast.Add,
               ast.Mult: ast.FloorDiv, ast.FloorDiv: ast.Mult}
CMPOP_SWAPS = {ast.Lt: ast.LtE, ast.LtE: ast.Lt,
               ast.Gt: ast.GtE, ast.GtE: ast.Gt,
               ast.Eq: ast.NotEq, ast.NotEq: ast.Eq}
# ...plus: every integer constant c becomes c + 1

The loop is: find every mutation site in the target module; for each one, write the mutated module into a scratch copy of the project and run pytest; a non-zero exit code means killed. The tester first runs the suite on the unmutated code and refuses to continue if it fails (otherwise every mutant would look “killed”).

23.11.3 Worked example: a strong suite🔗

Run against app/split.py with the real unit tests:

$ python -m mutation.mutate --tests tests/test_split.py
[  0] killed   line  25: Lt -> LtE
[  1] killed   line  27: Lt -> LtE
[  2] killed   line  44: Add -> Sub
[  3] killed   line  25: constant 1 -> 2
[  4] killed   line  27: constant 0 -> 1
[  5] killed   line  46: Sub -> Add
[  6] killed   line  30: Lt -> LtE
[  7] killed   line  30: Add -> Sub
[  8] killed   line  30: constant 1 -> 2

mutation score: 9/9 killed (100%)

Trace two of them. Mutant 4 changes line 27, if total_cents < 0, to if total_cents < 1: now splitting a zero total raises ValueError. It is killed by test_split_zero_total_is_all_zeros—a boundary test earning its keep. Mutant 6 changes i < leftover to i <= leftover, handing an extra cent to one person too many. It is killed by test_split_gives_leftover_cents_to_first_people.

23.11.4 Worked example: finding a real gap🔗

Now the parser, with its unit tests:

$ python -m mutation.mutate --target app/parser.py --tests tests/test_parser.py
[  0] killed   line  35: NotEq -> Eq
[  1] killed   line  51: LtE -> Lt
[  2] SURVIVED line  76: LtE -> Lt
[  3] killed   line  35: constant 5 -> 6
[  4] SURVIVED line  51: constant 0 -> 1
[  5] SURVIVED line  76: constant 0 -> 1

mutation score: 3/6 killed (50%)

Line 51 is if amount_cents <= 0: raise ParseError(...). Mutant 1 (<= 0 becomes < 0) is killed, because the negative-test table includes an amount of 0. But mutant 4 (<= 0 becomes <= 1) survives: the parser now rejects a 1-cent transaction, and no test notices, because no test ever parses an amount of exactly 1.

This is exactly the boundary we discussed in the test-design section: the suite tests the rejecting side of the boundary (0) but not the accepting side (1). Statement coverage of line 51 is 100%; only mutation testing saw the gap. The fix is a one-line test that "2026-09-12,Gum,1,alice,bob" parses.

Mutants 2 and 5 are on line 76, in format_transaction’s error path—the same lines coverage flagged. Coverage and mutation testing agree there.

23.11.5 Equivalent mutants and cost🔗

Not every survivor is a missing test. An equivalent mutant changes the code without changing behavior—for example, mutating if n < 1 to if n <= 0 for an integer n. No test can kill it, and recognizing one requires human judgment.

Mutation testing is also expensive: one full test run per mutant, and real modules have hundreds of mutation sites. Standard mitigations: mutate only changed files, run only the tests that cover each mutant, and run mutation testing nightly rather than on every change.

Common misconception. “Aim for a 100% mutation score.” As with coverage, the number is a guide. The value is in the list of survivors, each of which asks a concrete question: “would you really not notice if this were wrong?”

23.12 Fuzz Testing🔗

Definition. Fuzz testing (fuzzing) is a quality assurance technique for discovering coding errors and security loopholes. It feeds a program massive amounts of automatically generated random or malformed input, called fuzz, and watches for crashes, hangs, or failed assertions rather than for wrong answers. The tool that does this is called a fuzzer.

In plain English: throw garbage at the program, millions of variations of it, and watch for anything that breaks in a way no input should ever be able to break it.

Intuition. Every parser is a wall between your program and the outside world. Fuzzing is hiring a tireless vandal to hurl things at the wall all night. The vandal has no idea what the wall protects, and does not need to. That ignorance is the strength: the fuzzer is immune to the author’s assumptions about what inputs “would never happen.”

Fuzzing began in 1988, when Barton Miller’s class at the University of Wisconsin piped random bytes into standard Unix utilities and crashed roughly a third of them. Modern fuzzers, such as AFL++, libFuzzer, and Google’s OSS-Fuzz service, have found tens of thousands of real bugs, including serious security vulnerabilities.

Common misconception. “Crashes on garbage input don’t matter—nobody types that.” Attackers type exactly that. So do buggy clients, truncated uploads, corrupted files, and LLMs producing malformed tool arguments. In a web service, an unexpected exception is a 500 error; in C, it may be an exploitable memory corruption.

23.12.1 The harness🔗

A fuzzer needs a harness: a small function that takes one arbitrary input, feeds it to the code under test, and decides “fine” or “crash.” For TerpLedger’s parser the contract is: for any string whatsoever, either return a Transaction or raise ParseError—nothing else.

try:
    parse(line)
except ParseError:
    pass                      # clean rejection: allowed by the contract
except Exception as e:
    report_crash(line, e)     # anything else is a bug in the parser

Here is a first draft of the parser—the kind of draft an LLM (or any of us) produces on the first try. It handles the happy path perfectly:

def parse_transaction(line: str) -> Transaction:
    fields = line.split(",")
    date = datetime.date.fromisoformat(fields[0])   # ValueError on a bad date
    amount_cents = int(fields[2])                    # IndexError, ValueError
    participants = tuple(fields[4].split(";"))       # IndexError
    return Transaction(date, fields[1], amount_cents, fields[3], participants)

TerpLedger keeps this draft as app/parser_v1.py. Its fuzzer, fuzz/fuzz_parser.py, is about sixty lines of plain Python:

  1. Start from a seed corpus of valid inputs (three realistic transaction lines).

  2. Mutate a seed: delete a character, insert an “interesting” one (a comma, a digit, a NUL byte, an emoji), duplicate a chunk, or truncate.

  3. Run the harness.

  4. Deduplicate crashes by exception type, and minimize each crashing input by deleting characters while it still crashes—the same idea as shrinking.

$ python -m fuzz.fuzz_parser --target buggy --iterations 50000
[      1] CRASH IndexError: list index out of range
          input:     '2026-12-01,Lab s'
          minimized: '2026-12-01'
[      2] CRASH ValueError: Invalid isoformat string: '2026-1--31'
          input:     '2026-1--31,Lab snacks,105,caro0,carol'
          minimized: ''

50000 iterations, 2 distinct crash(es)

Both bugs surfaced within the first few iterations, and both minimized inputs are easy to understand: a date with nothing after it crashes the field access, and the empty string crashes the date parse. The fixed parser in app/parser.py checks the field count and wraps every failure in ParseError; against it, the same run reports 0 crashes.

Note the word mutation. Fuzzers mutate inputs to find bugs in the program; mutation testing mutates the program to find gaps in the tests. Same verb, opposite targets.

23.12.2 Generators: random bytes versus structured input🔗

Completely random bytes are usually rejected immediately by the first line of a parser, so they never reach the interesting code. A generator-based fuzzer instead produces structured, meaningful inputs, closer to what the program actually expects:

  • It defines the structure of valid inputs (a grammar, a JSON schema, an XML format, a binary protocol).

  • It produces variations that test edge cases: tiny and huge values, missing fields, weird nesting.

  • It mostly maintains validity, so the target program does not reject everything outright.

Mutating valid seeds, as TerpLedger’s fuzzer does, achieves a similar effect cheaply: the inputs stay “almost well-formed” and reach deep code paths. Csmith, from the differential testing section, is a generator-based fuzzer for C compilers.

23.12.3 Coverage-based randomized testing🔗

Blind fuzzing does not learn. Coverage-based randomized testing (coverage-guided fuzzing) combines random input generation with code coverage feedback. Instead of generating inputs blindly, it observes which parts of the program each input executes and uses that to guide future inputs toward new execution paths.

  1. Generate a random input (a file, a data record, a request).

  2. Run the program under instrumentation.

  3. Measure coverage: which functions, lines, or branches ran.

  4. Keep interesting inputs—those that reached new code.

  5. Mutate those interesting inputs to explore further.

  6. Repeat!

┌──────────────┐  selection  ┌──────┐  mutation  ┌───────┐

│  Seed pool   │ ──────────► │ seed │ ─────────► │ input │

│ ▢ ▢ ▢ ...    │             └──────┘            └───┬───┘

└──────▲───────┘                                     ▼

       │                                     ┌────────────────┐

       │  new coverage? keep the input       │ Target program │ ───►  BUG!

       └──────────── Coverage ◄───────────── └────────────────┘

On TerpLedger’s parser, the seed pool evolves like this:

seed: "2026-09-12,Pizza night,2350,alice,alice;bob;carol"

   ├─ mutate → "2026-09-12,Pizza night"        new path: field-count check  → KEEP

   ├─ mutate → "2026-09-1x,Pizza,2350,a,b"     new path: bad-date branch    → KEEP

   ├─ mutate → "2026-09-12,Pizza,23x0,a,b"     new path: bad-amount branch  → KEEP

   └─ mutate → "2026-09-12,Pizzq night,..."    nothing new                  → discard

Coverage, which earlier was only an after-the-fact report, becomes a live search signal. This is how AFL++, libFuzzer, and Atheris (for Python) work.

23.12.4 Fuzzing versus property-based testing🔗

The Hypothesis robustness property and our fuzzer check the same contract. The techniques are siblings, with different strengths:

Property-based testing

Fuzzing

Property

rich, domain-specific (“shares sum to the total”)

generic (“does not crash”)

Duration

about 100 examples, inside the test suite

millions of inputs, for hours or days

Guidance

clever sampling of a declared generator

often coverage-guided

Rule of thumb: code with a specification gets property tests; code that reads input you do not control (file uploads, network requests, LLM-generated tool arguments) gets fuzzed too.

23.13 Static Analysis🔗

Everything so far was dynamic: run the code on particular inputs. Static analysis examines code without running it and says something about all executions at once. It cannot prove a program correct (that question is undecidable in general), but it catches certain mistakes instantly, on every line.

A type checker turns a policy into a mechanical check. Suppose someone “improves” the split to use division:

def split_evenly(total_cents: int, n_people: int) -> list[int]:
    share = total_cents / n_people
    return [share] * n_people

/ produces a float, so money is now inexact. mypy catches it without running anything:

$ mypy --strict app
app/split.py:3: error: List item 0 has incompatible type "float";
                       expected "int"  [list-item]

A linter flags patterns where bugs tend to hide. TerpLedger’s first draft of balances paired participants with their shares using zip(t.participants, shares), and ruff warned:

$ ruff check .
B905 `zip()` without an explicit `strict=` parameter

zip silently stops at the shorter list, so if the share list ever became shorter than the participant list, someone would silently be charged nothing. With strict=True, that future bug becomes an immediate error. Ruff also flags pytest.raises(Exception) (rule B017), a weak-test smell.

Common misconception. “It type-checks, so it works.” Types prove the shapes fit, not that the values are right: total - share and total + share have the same type. Static analysis is the cheapest first filter, not a replacement for tests.

23.14 Testing in Modern Software Development🔗

23.14.1 Continuous integration🔗

Tests only help if they are run. Human discipline does not scale: someone will forget, or skip the tests “just this once.”

Definition. Continuous integration (CI) automatically builds and tests every change when it is pushed or proposed as a pull (merge) request, and blocks the merge if any check fails.

The mechanics of CI pipelines were introduced in the Git, GitHub, and CI/CD notes. TerpLedger ships a GitHub Actions workflow (the course’s own projects use GitLab CI; the ideas are identical). An abbreviated .github/workflows/ci.yml:

name: CI
on:
  push:
    branches: [main]
  pull_request:

jobs:
  lint:                        # cheapest checks first: fail in seconds
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install ruff && ruff check .

  typecheck:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt && mypy app

  test:
    needs: [lint, typecheck]   # don't spend test minutes on code that doesn't lint
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.11", "3.12", "3.13"]
    env:
      HYPOTHESIS_PROFILE: ci   # derandomized: no flaky failures
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "${{ matrix.python-version }}" }
      - run: pip install -r requirements.txt
      - run: pytest --cov      # unit + property + integration, coverage floor 90%

  fuzz-smoke:
    needs: [test]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: python -m fuzz.fuzz_parser --target fixed --iterations 50000 --seed 0

The design choices summarize this lecture:

  • Cheapest first. Lint and type checks run in seconds; tests wait for them. Fast feedback keeps developers from working around the pipeline.

  • Automated regression testing. Every test ever written runs on every change, on three Python versions.

  • Deterministic. Hypothesis is derandomized and the fuzzer uses a fixed seed. A CI check that fails randomly blocks everyone’s work.

  • A coverage floor, not a target. pytest --cov fails the job if coverage drops below 90%.

Slow or random checks—long fuzzing campaigns, thousands of Hypothesis examples with fresh seeds, mutation testing—run in a separate nightly pipeline (.github/workflows/nightly.yml), where a failure means “a new bug was found” rather than “your pull request is broken.”

23.14.2 Code review, test requirements, and deployment🔗
  • Tests are part of the change. Many teams require that a pull request include tests for new behavior, and that a bug fix include a test that fails without the fix. Reviewers read the tests first: they say what the change is supposed to do.

  • CI green is required to merge. Branch protection rules make the red X physically block the merge button.

  • Testing before and after deployment. The built artifact (for TerpLedger, a Docker image) is tested, not just the source: CI starts the container and runs a few smoke tests against it. After each deployment, the same smoke tests (“does /health answer?”) decide whether the release proceeds. See the Cloud Deployment: From GitLab CI/CD to Google Cloud notes.

Common misconception. “CI green means the code is correct.” CI is a gate, not an oracle: it certifies exactly that the checks you wrote pass. Thin checks give a thin guarantee.

23.15 Testing and Generative AI🔗

LLMs are good at writing tests: “write pytest tests for app/split.py” produces a plausible suite in seconds. This is one of the most valuable and most dangerous uses of an assistant. Valuable, because writing tests is tedious and models are tireless at listing cases. Dangerous, because a test suite is a judgment about what the code should do, and a model that reads the code tends to conclude that the code should do whatever it currently does.

23.15.1 Using LLMs to generate unit tests: the oracle problem🔗

When a model generates a test from the implementation, the implementation becomes the oracle. Ask a model to test the buggy [total // n] * n version of split_evenly, and a typical generated test is:

def test_split_evenly_uneven():
    assert split_evenly(1000, 3) == [333, 333, 333]   # "expected" = whatever the code returned

The test passes, the bug is now enshrined, and a later correct fix will be “caught” as a regression. A test that asserts the code does what the code does will pass forever and protect nothing. The defense: give the model the specification (the docstring, the requirements), not just the code, and check every expected value against the specification yourself.

23.15.2 Asking AI for missing edge cases and properties🔗

Models are excellent at brainstorming. Prompts such as these work well:

  • “Here is the docstring of parse_transaction. List 15 edge cases and invalid inputs I should test.” (Expect: leap days, Unicode names, a trailing newline, a very long amount, duplicate participants, a comma in the description.)

  • “What properties should split_evenly satisfy for all inputs? Write them as Hypothesis tests.” (Expect: conservation, fairness, length, non-negativity—and sometimes a wrong one, such as “all shares are equal,” which you must reject.)

You stay in charge of deciding which cases match the specification. The model enumerates; you judge.

23.15.3 Explaining failures and generating fixtures🔗

Paste a failing pytest report and ask “explain this failure and suggest the smallest fix.” Models are good at reading tracebacks and assertion diffs. Treat the suggested fix as a hypothesis: the test must still pass for the right reason, and the fix must not weaken the test.

Models are also good at the boilerplate of testing: fixtures that build sample transactions, fake LLM clients, parametrized tables of inputs. These are low risk because their correctness is easy to check by reading.

23.15.4 Testing AI-generated code🔗

When an LLM writes the code, every technique in this lecture becomes your review process. You cannot audit everything a model writes at the speed it writes it, but you can:

  • run the linter and type checker (instant; they catch invented APIs and type confusions);

  • write the properties yourself, from the specification, and let Hypothesis referee;

  • fuzz anything that parses input (the draft parser above is exactly what models produce on the first try);

  • check the requirements models tend to forget, such as all-or-nothing imports, error paths, and boundaries.

23.15.5 Using mutation testing to evaluate AI-generated tests🔗

TerpLedger keeps, as an exhibit, a suite of exactly the kind a model produces from a bare “write unit tests for app/split.py” prompt (weak_tests/test_split_ai_generated.py). An excerpt:

def test_split_evenly_returns_list():
    result = split_evenly(1000, 3)
    assert isinstance(result, list)
    assert len(result) == 3


def test_split_evenly_invalid_people():
    with pytest.raises(Exception):  # noqa: B017 -- any exception "passes"
        split_evenly(100, 0)


def test_balances_returns_dict():
    t = Transaction(datetime.date(2026, 9, 12), "Pizza", 1000, "alice", ("alice", "bob"))
    result = balances([t])
    assert isinstance(result, dict)
    assert "alice" in result
    assert "bob" in result

It looks professional: descriptive names, a test per function, error cases included. Now grade it. Coverage first:

$ pytest weak_tests --cov=app.split --cov-branch --cov-report=term-missing
Name           Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------
app/split.py      19      0      8      0   100%

Perfect statement and branch coverage. Now mutation testing:

$ python -m mutation.mutate --tests weak_tests/test_split_ai_generated.py
[  0] killed   line  25: Lt -> LtE
[  1] SURVIVED line  27: Lt -> LtE
[  2] SURVIVED line  44: Add -> Sub
[  3] killed   line  25: constant 1 -> 2
[  4] SURVIVED line  27: constant 0 -> 1
[  5] SURVIVED line  46: Sub -> Add
[  6] SURVIVED line  30: Lt -> LtE
[  7] SURVIVED line  30: Add -> Sub
[  8] SURVIVED line  30: constant 1 -> 2

mutation score: 2/9 killed (22%)

Seven of nine planted bugs survive. Mutant 2 makes the payer owe the money they paid (+= becomes -=); the suite only checks that alice and bob appear in the result. Mutants 6–8 break the leftover-cent rule three different ways; the suite only checks the list’s type and length. The real suite scores 9/9 on the same mutants. Same coverage, very different protection—and mutation testing tells them apart objectively, in seconds.

23.15.6 A workflow for AI-assisted testing🔗
  1. Prompt with the specification, and ask for boundaries, specific exception types, and properties.

  2. Ask for ideas before code: edge cases and properties, which you filter.

  3. Review every assertion: did this expected value come from the requirements, or from running the code?

  4. Watch for test smells: pytest.raises(Exception); assertions only on type, length, or key presence; tests that re-implement the function to compute the expected value; heavy mocking.

  5. Grade mechanically with coverage (what was skipped) and mutation testing (what was not checked). Feed survivors back: “this mutant survives: changing += to -= on line 44; write a test that kills it.”

  6. Make every new test fail once by breaking the code by hand.

A classroom experiment. Take a buggy implementation (for example, app/parser_v1.py), ask an LLM to write tests for it, and check whether those tests detect the bugs. Then give a fresh session only the specification and compare. The difference between the two suites is the oracle problem made visible.

23.16 Which Technique Catches Which Bug?🔗

The techniques are a portfolio, not a ranking. Here are the real mistakes from this lecture, and what catches each:

The mistake

Caught by

[total // n] * n loses the leftover cents

unit test (if you picked a non-divisible example); property test (always)

a negative withdrawal becomes a deposit

test design: writing the negative test exposed a missing requirement

a buggy reverse that does nothing

a unit test or a stronger property; not “reverse twice is identity”

leftover cents go to the wrong person

differential test against the one-cent-at-a-time oracle

the in-memory fake accepts what the database rejects

differential (contract) test over both implementations

format_transaction’s error paths never run

coverage report

no test checks that a 1-cent amount is accepted

mutation testing only—coverage of that line is 100%

the parser crashes with IndexError on "2026-12-01"

fuzzing; also the robustness property

total / n makes money a float

type checker, instantly

an AI-written suite asserts only types and lengths

mutation testing (22% at 100% coverage)

any of the above, re-introduced next month

CI—the only thing that re-checks everything, every time

Read the right column again: almost no overlap. Each technique sees bugs the others are blind to. This is defense in depth: independent layers with independent blind spots.

23.17 The TerpLedger Project🔗

The complete project from these notes is in the course repository at notes/testing/code/terpledger, ready to push to a GitHub repository of your own (the CI workflows activate automatically):

terpledger/

├── app/

│   ├── models.py        # Transaction dataclass

│   ├── split.py         # money-splitting logic

│   ├── parser.py        # one-line record parser (fuzz target, fixed)

│   ├── parser_v1.py     # first buggy draft, kept for the fuzzing demo

│   ├── ledger.py        # in-memory and SQLite stores

│   ├── categorize.py    # LLM categorizer + Ollama client

│   └── main.py          # FastAPI web API

├── tests/               # the real suite -- what `pytest` runs

│   ├── conftest.py              # Hypothesis profiles: dev / ci / nightly

│   ├── fakes.py                 # ScriptedLLM, KeywordLLM, BrokenLLM

│   ├── test_split.py            # unit tests

│   ├── test_parser.py           # unit tests (partitions, boundaries)

│   ├── test_properties.py       # property-based tests (Hypothesis)

│   ├── test_ledger_contract.py  # one suite, two storage backends

│   ├── test_categorize.py       # LLM code tested with test doubles

│   └── test_api.py              # integration tests (FastAPI TestClient)

├── weak_tests/          # an "AI-generated" suite: 100% coverage, weak assertions

├── smoke/               # smoke tests against a running deployment

├── fuzz/fuzz_parser.py  # tiny mutation-based fuzzer

├── mutation/mutate.py   # tiny mutation tester

├── Dockerfile

├── .github/workflows/ci.yml       # lint, typecheck -> test + coverage -> fuzz, docker smoke

├── .github/workflows/nightly.yml  # long Hypothesis, long fuzzing, mutation testing

├── pyproject.toml       # pytest, coverage, ruff, and mypy configuration

└── requirements.txt

Setup and a tour:

cd notes/testing/code/terpledger
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

pytest                                      # 73 passed, 1 skipped (the live LLM test)
pytest --cov                                # branch coverage; fails below 90%
HYPOTHESIS_PROFILE=ci pytest                # derandomized, exactly as in CI

python -m mutation.mutate                   # real suite vs split.py: 9/9 killed
python -m mutation.mutate --target app/parser.py --tests tests/test_parser.py   # 3/6
python -m mutation.mutate --tests weak_tests/test_split_ai_generated.py         # 2/9

python -m fuzz.fuzz_parser --target buggy   # watch it find the two crashes
python -m fuzz.fuzz_parser --target fixed   # the fixed parser survives

ruff check .                                # linter
mypy app                                    # type checker

uvicorn app.main:app --reload               # then open http://127.0.0.1:8000/docs

23.18 Practice Exercises🔗

23.18.1 Basic🔗
  1. Give one example each of a verification question and a validation question about TerpLedger. Then explain the difference between testing and debugging using the leftover-cent bug.

  2. List the equivalence partitions and boundary values for the n_people argument of split_evenly. Which of them does tests/test_split.py already cover?

  3. The test suite never checks split_evenly(0, 1). Predict the output, add the unit test, and check your prediction.

  4. Classify each as unit, integration, system, acceptance, or regression testing, and as functional, performance, security, usability, or availability testing: (a) “GET /balances returns the right JSON after a POST”; (b) “the club treasurer confirms the monthly totals match the bank”; (c) “/balances answers in under 100 ms with 100,000 transactions”; (d) “1 million random strings never crash the parser.”

  5. For the condition if amount <= balance in withdraw, give a minimal set of tests that achieves (a) statement coverage and (b) branch coverage.

23.18.2 Intermediate🔗
  1. Kill the surviving parser mutant on line 51 (<= 0 became <= 1) with the smallest possible new test, and rerun the mutation tester to confirm.

  2. The coverage report lists app/parser.py lines 75, 77, 79 as missing. Write tests that cover them, rerun pytest --cov, and confirm the lines disappear. Which surviving parser mutants did your tests also kill?

  3. Write a property test asserting that in split_evenly, the shares are non-increasing (the people listed first absorb the leftover cents). Then break the implementation in a way your new property catches but the four existing split_evenly properties do not.

  4. Using TDD, add a function owes(transactions, a, b) that returns how much a owes b. Write each test before the code that passes it, and record each red and green step.

  5. Extend mutation/mutate.py with one new mutation operator from the list in these notes (for example, replacing and with or, or deleting a raise statement). Report the new survivors it finds in app/parser.py.

23.18.3 Advanced🔗
  1. The round-trip property tests parse(format(t)) == t. The other direction, format(parse(line)) == line for every line the parser accepts, is false for this format. Find a counterexample, explain the root cause, and propose either a format restriction or a weaker property that does hold.

  2. Write a coverage-guided version of fuzz/fuzz_parser.py: use Python’s sys.settrace (or coverage.py’s API) to record which lines of parser_v1.py each input executes, and keep inputs that reach new lines as new seeds. Compare how many iterations each fuzzer needs to find both crashes.

  3. AI-generated tests, measured. Give an LLM only the source of app/parser.py and ask for a test suite. Measure it with coverage and mutation/mutate.py. Then give a fresh session only the module docstring and measure again. Finally, feed the surviving mutants back to the model and measure a third time. Report the three scores, and classify each assertion in the first suite as “from the specification” or “from the implementation.”

23.19 Summary🔗

23.19.1 Key takeaways🔗
  • Testing shows the presence of bugs, never their absence. Each technique widens the net in a different direction, and the techniques overlap remarkably little.

  • Bugs get more expensive the later they are found, so test early, automatically, and on every change.

  • Unit, integration, system, and acceptance tests check different levels of the V model; regression testing re-runs all of them after every change. Most tests should be small, fast unit tests.

  • Choose test inputs systematically: one per equivalence partition, both sides of every boundary, positive and negative cases, and a clear oracle. Writing tests often exposes missing requirements.

  • Frameworks like pytest provide assertions, parametrization, fixtures with setup and teardown, and markers. Test doubles replace dependencies you cannot use in a fast test; prefer fakes to mocks. TDD writes each test before the code that passes it.

  • Property-based tests state laws over whole input domains; generators produce inputs and shrinking reduces failures to minimal counterexamples. A weak property (“reverse twice is identity”) can miss an obvious bug.

  • Differential testing uses a second implementation as the oracle.

  • Coverage shows what the tests never ran; branch and condition coverage are stricter than statement coverage. 100% coverage does not mean the code is tested.

  • Mutation testing shows what the tests never checked, and is the objective way to grade a suite—including one an AI wrote.

  • Fuzzing feeds massive amounts of generated input to code that faces untrusted data; coverage guidance steers it toward new code paths.

  • CI runs every check on every change; slow checks run nightly. LLMs are useful test assistants, but every expected value they write must be checked against the specification.

23.19.2 Terminology🔗

Term

Meaning

test case

input, action, and expected result

test oracle

whatever decides the expected result

verification / validation

building the product right / building the right product

regression

previously working behavior broken by a change

V model

development phases paired with the test levels that check them

test pyramid

many unit tests, fewer integration tests, fewest end-to-end tests

equivalence partitioning

dividing inputs into classes treated the same way

boundary value analysis

testing at and around the edges of each class

Arrange–Act–Assert

the three-phase shape of a test: set up, call, check

fixture

reusable setup (and teardown) a test requests by name

parameterized test

one test body run over a table of cases

flaky test

a test whose verdict varies without a code change

test double: stub / fake / spy / mock

canned answers / simplified working implementation / records calls / fails on violated expectations

TDD

red–green–refactor: write a failing test, pass it, clean up

property

a boolean condition that must hold for every input in a domain

generator (strategy)

a description of how to produce random inputs

shrinking

reducing a failing input to a minimal one that still fails

seed

the source of randomness that makes a random run reproducible

differential testing

comparing outputs of several implementations

function / statement / branch / condition coverage

fraction of functions called / statements run / branch outcomes taken / sub-conditions seen true and false

mutant, mutation operator

program copy with one planted fault / the rule that plants it

killed / survived

some test failed on the mutant / all tests passed

mutation score

killed mutants divided by non-equivalent mutants

equivalent mutant

a mutant that behaves identically to the original

fuzzing, harness, seed corpus

testing with massive generated input / adapter that feeds it to the code / valid starting inputs

coverage-guided fuzzing

fuzzing that keeps inputs reaching new code paths

CI

automatically building and testing every change

23.19.3 Common mistakes🔗
  • Testing only the happy path—no boundaries, no negative tests, no malformed input.

  • Testing one side of a boundary but not the other (0 rejected, but 1 never checked).

  • Copying the code’s output into the test as the expected value.

  • pytest.raises(Exception) instead of the specific exception type.

  • Chasing 100% coverage with assertion-free or type-only tests.

  • Writing a property so weak that wrong code also satisfies it.

  • Tests that share state and pass only in one order.

  • A fake that is more permissive than the real thing, with no differential test to catch the difference.

  • Mocking everything, so tests check call sequences instead of behavior.

  • Unseeded random tests that fail and cannot be replayed.

  • Tolerating flaky tests and a red CI “that’s always like that.”

  • Trusting AI-generated tests because they pass.

23.19.4 Looking ahead🔗

The ideas in this lecture return throughout the rest of the course. Every API you build gets integration tests; every pipeline you write runs these checks as quality gates (Cloud Deployment: From GitLab CI/CD to Google Cloud); every security rule deserves both an “allowed” and a “denied” test. When we evaluate LLM applications, the same vocabulary returns in a new form: regression suites of prompts, properties over model outputs, an LLM as a statistical oracle, and red-teaming as fuzzing of a system whose “parser” is a neural network. The problems get harder, but the toolbox is the same.