CMSC 838Z Spring 2005

Language-Based Security


Simply stated, the goal of computer security is to prevent "bad" things from happening, or at the least, catch the bad guys who made them happen.  Computer security grew out of work that started in the late 1960's on multi-access (time-shared) computers.  Much of the hardware and operating systems we have today are based on long-standing notions of protection (prevent one user's process from trashing another's memory, or the memory of the kernel), and access control (prevent processes from reading, writing, or running files for which they are not authorized).  The hardware and the operating system cooperate to enforce these notions.

While these notions are still useful, many things have changed since the 1970's.  Back then, operating systems were small, and therefore more trustworthy.  Now they are huge (50 million lines of code in Windows XP).  Back then, computers were largely isolated, and accessed by a limited number of people.  Now, computers are constantly open to a possibly-malicious, networked environment in which someone in Ukraine is as close as the guy down the hall.  Software comes from a variety of sources, and the user has no hope of understanding what it does, much less trusting that it should work.  Indeed, the success of many of today's viruses stems from executable content transferred in e-mail or made available on the web.  This content is executed by default, and circumvents OS-level protection mechanisms to cause serious damage.  Some ad hoc solutions exist, but they are not perfect (e.g., virus scanning).  Shutting off all executable content is a non-starter for most users.  New solutions are needed.


In this course, we consider a variety techniques by which security can be ensured at the programming language level:

  • Reference Monitors.  The most common security enforcement mechanism is the operating system, acting as a reference monitor.  A reference monitor "watches" security-related actions, and prevents those actions that would lead to a violation.  We will look at a variety of techniques for implementing reference monitors through program transformation and checking.  Examples include Java stack inspection and Software-based Fault Isolation.
  • Type checking.  Traditional views of types are as descriptors of data format, e.g. as integers, strings, and pointers.  Ensuring that a program adheres to the sepcification of its types, termed type-safety, is sufficient to rule out many common attacks, including buffer overflows.  We will consider two ways in which types have been applied to machine code to certify that an executable is type-safe.
  • Information Flow.  One problem with traditional access-control systems is that they only govern the release of data, not its propagation, and thus do not satisfy security end-to-end.  Information flow security has a recent history of application by language-based techniques.
  • Static program analysis.  Type checking is one kind of static analysis.  We will examine other static analysis techniques aimed at finding security bugs, including model checking, bug pattern detection, and others.
  • Protocol Checking.  Language-based techniques can also be used to model distributed systems and the protocols used to communicate within them.  Some interesting results are able to make authenticity and confidentiality in protocols correspond to type-checking in the modeling language.

The first half of the course will be spent reading and discussing research papers, many of which will have a theoretical element to them.  There will be some homeworks, both written and programming, to ensure that students understand the techniques being covered.  Students will implement a larger project in the second half of the course.


The course requires students to have taken CMSC 631 or CMSC 838Y(Z), or else acquire instructor approval.

Class Structure and Grading

The majority of the course will consist of reading and discussing web-available technical papers on the above topics.  In general, the course (and grade) will have three elements (subject to change up until class starts):

  • Class Participation. (35%) This consists of (1) a written review for each paper; (2) discussion during class; and (3) a presentation the student will give about a particular topic. Presentation guidelines are given below.  There may be some small homework assignments as well; any scores are included in this element.
  • Project. (40%) The student will propose a project idea roughly 1/3 of the way through the semester, to be finalized 1/2 of the way through, and then completed by the end of the semester. Students may also propose to work in groups of two. A main goal of the first half of the course is to chart research ideas that will be turned into projects.
  • Final Exam. (25%) The final exam will be comprehensive, and along with the project will count for comp credit.


For most papers that we will read in the class, students must submit a review of each paper.  These will be used to drive discussion of the paper during class.  Reviews are submitted to by 12 noon the day the paper is to be discussed.


The project will take place in two stages:

  1. Project proposal. The proposal can either be written (roughly 1 page) that indicates your thoughts on projects, or oral (i.e. meet with me in person). You should indicate what areas you are interested in exploring, and how you hope to explore them. The latter part should express your general feeling as to methodology (e.g. whether you want to do something theoretical or practical, whether you want to implement something, do a survey, etc.). You should then justify both the relevance of the project to the course (why will you learn something by doing it?), and also the magnitude of the effort (can you get it done this semester, or at least get some reasonable piece done that fits into a more ambitious effort?).

  2. Do the project. Projects should at the least have a written component, reporting what you accomplished. My belief is that the best projects will have both an implementation and analytic component.  Chances are, you will want to build on the area in which you presented. Project ideas will be posted, but start thinking about the project from the start.


Each student will give one lecture about a particular tool or technique, starting at roughly halfway through the semester. Presentation quality is important: read your papers early and be sure you fully understand them. I will be available for help, if you need it. Be sure that you have prepared a good presentation (consider practicing it beforehand). 

Presentations will be judged based on the following criteria (horked from Chris Hawblitzel's course syllabus):

  • understanding: does the presenter understand the material?
  • thoughtfulness: does the presenter have insights and opinions beyond what was in the paper?
  • background/perspective: did the presenter read background papers?
  • clarity: can the audience understand the presentation? is the "big picture" clear? are there useful examples?
  • materials: do the slides or use of blackboard illustrate and support the talk? are there diagrams to help convey the technicalities? (when your talk gets into deep territory, a diagram is worth 10K words)
  • delivery: has the the presenter practiced?
  • non-regurgitation: did the presenter do something beyond simply typing sections of the paper as bullet points? did the presenter motivate the ideas in their own words or just state ideas from the paper verbatim?
  • answering questions: can the presenter handle questions from the audience?
Remember that you will likely be able to explain more detail than you can hope to cover in a single lecture. This is one reason that it's hard work to prepare a good presentation: not only do you need to understand the paper, but you need to filter out the irrelevant details and amplify the key arguments. You'll probably have omit entire sections of the paper from your talk -- don't worry about it. Simply mimicking the structure of the paper ("regurgitating it") tends to produce a disconnected sequence of boring facts. A good talk should tell a story; every idea should be motivated, and all facts should fit together in a coherent picture. Telling such a story in a short time often requires creating your own explanations, motivation, and examples. 

Final Exam

The final exam will likely be a take-home exam.

Academic Dishonesty

The college policy on academic dishonesty is strictly followed.  All graded materials (whether exams, summaries, presentations, or projects) must be strictly individual efforts.  In the case of a group project, only collaboration amongst the group is permitted.