Projects of the Experimental Software Engineering Group at the University of Maryland

Experimental Designs (2)


This project has three phases:
  1. A literature search was performed in order to identify those techniques that have been used in previous software development projects. We have identified 12 experimental design, broken down into three broad categories: historical methods, observational methods, and controlled methods. Roughly speaking, the historical methods are those most associated with fields such as archaeology where completed projects are studied, the observational methods are like astronomy where you have little control over your environment, and the controlled methods are used heavily in fields such as psychology where desired effects can be replicated on numerous subjects.
  2. A literature search of over 600 published software engineering papers was performed in order to identify how well these 12 methods are used in practice.
  3. We are currently studying how well these techniques apply to industrial environments by evaluating several organizations in order to understand how well they undertake technology transition tasks.

Validation Strategy

A literature search in fields other than computer science has been undertaken to see how well the computer science models relate to other disciplines. Evaluation studies in other organization is being performed in order to understand the merits of the various experimental models being proposed.

Project Status

Active - Industrial evaluation underway


An initial classification of 12 experimental designs has been developed. The initial study of 612 published papers in 1985, 1990, and 1995 shows that:
  1. About 30% of software engineering papers have no experimental component and another 30% have a weak form of experimental validation.
  2. If theoretical papers are deleted from the list of no experimentation papers, then the percent of papers with no experimentation drops to under 20% -- not far removed from the numbers in physics.
  3. However, the 30% of papers that were subjectively classified as assertions (i.e., weak experimental designs) seems high compared to other fields.

Top Level Page

<- ESEG Home Page

Last updated: February 12, 1997 by Marvin Zelkowitz