Projects of the Experimental Software Engineering Group at the University of Maryland

Reading Techniques for

Fault Detection

Approach

We have been working on an approach to generating families of reading techniques. Each family (and thus each technique) is associated with the particular document (e.g., requirements, design) and notation (e.g., English text, a formal notation) in which the document is written. Each technique within the family is (1) tailorable, based upon the project and environment characteristics; (2) detailed, in that it provides the reader a well-defined set of steps to follow; (3) specific, in that the reader has a particular purpose or goal for reading the document and the procedures that support the goal; (4) focused, in that it provides a particular coverage of the document, and a combination of techniques in the family provides coverage of the entire document; (5) studied empirically to determine if and when it is most effective.
Generating a family of reading techniques consists of building operational scenarios which require the reader to first create a model of the product, and then answer questions based on analyzing the model with a particular emphasis. Each reading technique in the family can be based upon a different abstraction and question set. The choice of abstraction and the types of questions may depend on the document being read, the problem history of the organization, or the goals of the organization.

Validation Strategy

This experiment was part of a first series of experiments, aimed at discovering if scenario-based reading is more effective than current practices. (A second series will be used to discover under which circumstances each of the various scenario-based reading techniques, or families of techniques is most effective.)
One of our goals is to replicate the experiments in different environments in order to better understand how to tailor the techniques to different domains, organizations and cultures. So far, replications have been undertaken:
  • at the University of Kaiserslautern, Germany (contact person: Oliver Laitenberger)
  • at the University of Trondheim, Norway (contact person: Sivert Sørumgård)

  • Our experiences from all of these experiments led us to make improvements to PBR. The new version of the technique was studied empirically in an experiment in Fall 1997. The analysis is not yet completed, but we have created an initial lab package describing the new techniques and the experiment.

    Project Status

    Active

    The original experiment has been completed, along with some replications in other contexts (see Validation Strategy).
    We have made improvements to PBR based on these experiments and have run a new study to evaluate the effectiveness of the changes. The analysis of this study is still ongoing.

    Results

    Modeling

    So far, two different families of reading techniques have been defined for requirements documents: defect-based reading and perspective-based reading.
    Defect-based reading focuses on a model of the data and functions of the requirements in a form of state machine notation. The different model views are based upon focusing on some specific defects classes: data type inconsistency, incorrect functions, and ambiguity or missing information. The analysis questions are generated by combining and abstracting a set of questions that are used in checklists for evaluating the correctness and reliability of requirements documents.
    Perspective-based reading focuses on different product customer perspective, e.g., reading from the perspective of the software designer, the tester, the end-user, the maintainer, the hardware engineer. The analysis questions are generated by focusing predominantly in various requirements type errors, e.g., incorrect fact, omission, ambiguity, and inconsistency.

    Validation

    Techniques from both families have been studied experimentally. and a Lab Package for the Empirical Investigation of Perspective-Based Reading was developed. In the Lab package we present our experiences from previous runs of this experiment, as well as our experimental design and materials, in order that we may serve as a source of information for other researchers to replicate the experiment in different environments.
    In the defect-based reading study, we evaluated and compared defect-based reading, ad hoc reading and checklist-based reading, with respect to their effect on fault detection effectiveness in the context of an inspection team. The study, a blocked subject-project, was replicated twice using graduate students at the University of Maryland. Major results were that (1) the defect-based readers performed better than ad hoc and checklist readers with an improvement of 35%; (2) the defect-based reading procedures helped reviewers focus on specific fault classes but were no less effective at detecting other faults; and (3) checklist reading was no more effective than ad hoc reading.
    In the perspective-based reading study, we evaluated and compared perspective-based reading and NASA's current reading technique with respect to their effect on fault detection effectiveness in the context of an inspection team. Three types of perspective-based reading techniques were defined and studied: tester-based, designer-based, and user-based. The study, again a blocked subject-project, was replicated twice in the SEL environment with NASA professionals. Major results were that (1) perspective-based reading was effective for generic documents both at the individual level and team level; and (2) perspective-based reading was effective for the NASA documents at the team level. It was felt that the techniques could be better tailored to the NASA style document to improve individual scores.

    Top Level Page


    <-Back to ESEG Home Page
    Last updated: March 31, 1998 by Forrest Shull