Building software on top of Large Language Models for CMSC398z

For presentation in UMD’s CMSC 398z Oct 3rd, 2025, which is offered by Bill Pugh and Derek Willis.

These slides have been adapted from a workshop presented by Simon Willison at PyCon 2025.

See Simon’s blog for his accompanying annotated slides and the original github repository.

As a resource, you might want to also look at Full documentation of llm.

Among other changes, we

• Updated the model used from gpt-4.1-mini to gpt-5-mini,

• Use uv for managing python dependencies rather than pip,

• So far, only cover about half of the material in Simon’s workshop,

• Have more discussion of correct and consistent data extraction

• Describe projects for week 5 of CMSC 398z

• Replaced all references to pelicans with terrapins.

We plan to cover much of the rest of the material in Simon’s workshop on Oct 10th.

• Setup
  • Setting up your environment
  • Configuring the OpenAI API key
  • Testing that it worked
• Prompting with LLM
  • Setting a default model
  • Running a prompt
  • What did that do for us?
  • Continuing the conversation
  • Seeing it in the logs
  • Browsing the logs with Datasette
  • Using different models
    • Logs are shared across llm installations
  • Piping in content
  • Using system prompts
  • Prompting with an image
  • Using fragments
  • Fragment plugins
• Structured data generation and extraction
  • Using simple schemas
  • Something a bit more impressive
  • Structured data from images
  • Structured data extraction in Python with Pydantic
• Prompting from Python
  • Streaming
  • Using attachments
  • Using system prompts
  • Specifying schemas
  • Formatting json
  • Logging when calling from python
    • Why the difference?
    • How to log from Python
• Correct and consistent extraction
• Week 5 project
  • Json object returned by data extraction
  • First steps
  • Writing up what you found
  • Optional: Automate your analysis
  • Optional: Evaluate model and prompt choices
  • Details on data sources
    • Unparsable addresses
    • FEMA emergency declarations
    • Sanctions of Maryland attorneys
    • Census data from 1930
• Building a text to SQL tool
  • The UMD course database
  • Prototyping against the UMD course database
  • A more detailed schema
  • Turning that into a Python function
  • Upgrading that to a CLI tool
  • Ways to make this better
• Semantic search and RAG
  • Semantic search
  • Generating embeddings with LLM
  • Answering questions against those course descriptions
  • RAG does not have to use semantic search!
  • RAG is dead?
• Tool usage
  • Passing Python functions to an LLM
  • Multiple tools can run in a row
  • Tools in Python
  • Running calculations with simpleeval
  • Let’s build some tools!
  • Chat with your database, implemented using tools
• Security and prompt injection