CMSC838M: Advanced Topics in Software Testing

Atif M. Memon
Fall 2002

Course Description

This course will examine advanced software testing techniques. In particular, the important phases of testing will be reviewed, emphasizing on the significance of each phase when testing different types of software. Students will learn the state of the art in testing technology for object-oriented, component-based, concurrent, distributed, graphical-user interface, and web software. In addition, closely related concepts such as model checking and program analysis will also be studied. Emerging concepts such as test-case prioritization and their impact on testing will be examined. Students will gain hands-on testing/analysis experience by proposing new solutions to open research problems in the field of software testing and experimentally demonstrating the strengths/weaknesses of their solutions.

By the end of this course, students should be familiar with the state-of-the-art in software testing. Students should also be aware of the major open problems in testing.

Is the course valid for PhD qualifying coursework?

Yes (Software Engineering/Programming Languages)

Is the course valid for MS qualifying coursework?

Yes (Software Engineering/Programming Languages)

Is the course valid for MS comps?

Yes. (Both Midterm and Final exams count towards the MS comps.)

Meeting Times

Tue.Thu - 12:30PM - 1:45PM (CSI 2107)

Office Hours

Tue.Thu - 1:45PM - 4:00PM (4115 A. V. Williams Building)

Prerequisites

CMSC 435 or equivalent.

Schedule and required readings

(Subject to change; additional readings may be assigned)
 

Week

Dates

Topic

Readings (copies available in the instructor’s office)

1

Sep. 3-5

Course Overview

 

(Slides)

Testing: a roadmap

Mary Jean Harrold

Proceedings of the conference on The future of Software engineering May 2000

Introduction to special section on software testing

R. Hamlet

Communications of the ACM June 1988
Volume 31 Issue 6

Testing: principles and practice

Stephen R. Schach

ACM Computing Surveys (CSUR) March 1996
Volume 28 Issue 1

2

Sep. 10-12

Introduction to Testing (preliminary concepts – testability, whitebox, blackbox, unit, integration,

mutation)

 

(Slides)

Software safety: why, what, and how

Nancy G. Leveson

ACM Computing Surveys (CSUR) June 1986
Volume 18 Issue 2

Validation, Verification, and Testing of Computer Software

W. Richards Adrion , Martha A. Branstad , John C. Cherniavsky

ACM Computing Surveys (CSUR) June 1982
Volume 14 Issue 2

3

Sep. 17-19

Taxonomies of techniques,

 

Test-case Generation I

 

(Slides) (Slides) (Slides)

Rethinking the taxonomy of fault detection techniques

Michael Young, Richard N. Taylor

Proceedings of the 11th international conference on Software engineering May 1989

A test generation strategy for pairwise testing
Kuo-Chung Tai; Yu Lei
Software Engineering, IEEE Transactions on , Volume: 28 Issue: 1 , Jan. 2002
Page(s): 109 -111

The category-partition method for specifying and generating fuctional tests

T. J. Ostrand , M. J. Balcer

Communications of the ACM June 1988
Volume 31 Issue 6

4

Sep. 24-26

Test-case Generation II

 

(Slides) (Slides)

Hierarchical GUI test case generation using automated planning
Memon, A.M.; Pollack, M.E.; Soffa, M.L.
Software Engineering, IEEE Transactions on, Volume: 27 Issue: 2, Feb. 2001
Page(s): 144 –155

Using a goal-driven approach to generate test cases for GUIs

Atif M. Memon, Martha E. Pollack, Mary Lou Soffa

Proceedings of the 21st international conference on Software engineering May 1999

Predicate-based test generation for computer programs
Kuo-Chung Tai
Software Engineering, 1993. Proceedings., 15th International Conference on , 1993
Page(s): 267 –276

5

Oct. 1-3

Test-case Generation III

 

(Slides)

 

Test Coverage I

 

(Slides) (Slides) (Slides)

A heuristic approach for test case generation

Kai-Hsiung Chang, W. Homer Carlisle, James H. Cross, David B. Brown

Proceedings of the 19th annual conference on Computer Science April 1999

Software unit test coverage and adequacy

Hong Zhu, Patrick A. V. Hall, John H. R. May

ACM Computing Surveys (CSUR) December 1997
Volume 29 Issue 4

The evaluation of program-based software test data adequacy criteria

E. J. Weyuker

Communications of the ACM June 1988 Volume 31 Issue 6

Coverage criteria for GUI testing

Atif M. Memon, Mary Lou Soffa, Martha E. Pollack

ACM SIGSOFT Software Engineering Notes, Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT symposium on Foundations of software engineering September 2001
Volume 26 Issue 5

6

Oct. 8-10

Data-flow Analysis,

Data-flow Testing

 

(Slides)

Interprocedual data flow testing

M. Harrold, M. Soffa

ACM SIGSOFT Software Engineering Notes, Proceedings of the ACM SIGSOFT '89 third symposium on Software testing, analysis, and verification November 1989
Volume 14 Issue 8

An applicable family of data flow testing criteria
Frankl, P.G.; Weyuker, E.J.
Software Engineering, IEEE Transactions on, Volume: 14 Issue: 10, Oct. 1988
Page(s): 1483 –1498

7

Oct. 15-17

Test Oracles

 

(Slides)

Automated test oracles for GUIs

Atif M. Memon, Martha E. Pollack, Mary Lou Soffa

ACM SIGSOFT Software Engineering Notes, Proceedings of the eighth international symposium on Foundations of software engineering for twenty-first century applications November 2000
Volume 25 Issue 6

Specification-based test oracles for reactive systems

Debra J. Richardson, Stephanie Leif Aha, T. Owen O'Malley

Proceedings of the 14th international conference on Software engineering June 1992

8

Oct. 22-24

Test Coverage II

 

(Slides)

 

&

Regression Testing

(Slides)

 

&

Student Project Proposals

{Scott McMaster, Avik Sinha, Arunchandar Vasan, Harsh Nanda, Duy Huynh, Nada Hashmi, Adithya Nagarajan, Aleks Aris}

Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M.; Foster, H.; Goradia, T.; Ostrand, T.
Software Engineering, 1994. Proceedings. ICSE-16., 16th International Conference on, 1994
Page(s): 191 –200

An empirical study of regression test selection techniques

Todd L. Graves, Mary Jean Harrold, Jung-Min Kim, Adam Porter, Gregg Rothermel

ACM Transactions on Software Engineering and Methodology (TOSEM) April 2001
Volume 10 Issue 2

9

Oct. 29-31

Student Project Proposals

{Cyntrica N Eaton, Susmita Ghose, Damayanti Gupta, Hyma S Murthy, Chuk Yang Seng,

Anil Sharma, Dave Alan Tahmoush, Dongfeng Zhu}

&

Mid-term exam (Oct. 31)

 

10

Nov. 5-7

Student Presentations

Topic: Testing Spreadsheets

Presenter 1: Alex Aris (Slides - ppt, pdf)

Paper(s): WYSIWYT testing in the spreadsheet paradigm: an empirical evaluation

Karen J. Rothermel, Curtis R. Cook, Margaret M. Burnett, Justin Schonfeld, T. R. G. Green, Gregg Rothermel

Proceedings of the 22nd international conference on Software engineering June 2000

 

Topic: Network Security Testing

Presenter 1: Anil Sharma (Slides - ppt, pdf)

Paper(s): Using model checking to analyze network vulnerabilities
Ritchey, R.W.; Ammann, P.
Security and Privacy, 2000. S&P; 2000. Proceedings. 2000 IEEE Symposium on , 2000, Page(s): 156 -165

 

Topic: Web Testing

Presenter 1: Adithya Nagarajan (Slides - ppt, pdf)

Paper(s): Analysis and testing of web applications
Ricca, F.; Tonella, P.
Software Engineering, 2001. ICSE 2001. Proceedings of the 23rd International Conference on, 2001
Page(s): 25 –34

Going Faster: Testing The Web Application (NOTE: Requires IEEE Computer society password)

Edward Hieatt, Robert Mee

IEEE Software, March/April 2002 (Vol. 19, No. 2), pp.  60-65 

 

Presenter 2: Arunchandar Vasan (Slides - ppt, pdf)

Paper(s): An object-oriented web test model for testing Web applications
Kung, D.C.; Chien-Hung Liu; Pei Hsia
Quality Software, 2000. Proceedings. First Asia-Pacific Conference on, 2000
Page(s): 111 –120

Using the WWW Test Pattern to check HTML client compliance
Berghel, H.
Computer, Volume: 28 Issue: 9, Sept. 1995
Page(s): 63 –65

11

Nov. 12-14

Student Presentations

Topic: Combining and Comparing Techniques

Presenter 1: Nada Hashmi (Slides - ppt, pdf)

Paper(s): On comparisons of random, partition, and proportional partition testing
Ntafos, S.C.
Software Engineering, IEEE Transactions on, Volume: 27 Issue: 10, Oct. 2001
Page(s): 949 -960

 

Presenter 2: Damayanti Gupta (Slides - ppt, pdf)

Paper(s): Further empirical studies of test effectiveness

Phyllis G. Frankl , Oleg Iakounenko

ACM SIGSOFT Software Engineering Notes, Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering November 1998
Volume 23 Issue 6

 

Topic: Defect and Failure Estimation and Analysis

Presenter 1: Scott McMaster (Slides - ppt, pdf)

Paper(s): Predicting fault incidence using software change history
Graves, T.L.; Karr, A.F.; Marron, J.S.; Siy, H.
Software Engineering, IEEE Transactions on, Volume: 26 Issue: 7, July 2000
Page(s): 653 –661

 

Presenter 2: Dongfeng Zhu (Slides - ppt, pdf)

Paper(s): A critique of software defect prediction models
Fenton, N.E.; Neil, M.
Software Engineering, IEEE Transactions on, Volume: 25 Issue: 5, Sept.-Oct. 1999
Page(s): 675 –689

12

Nov. 19-21

Student Presentations (Nov. 19)

&

Testing Tools (Nov. 21)

  1. Ishan Banerjee: GUI Testing tools (pdf)
  2. Nada Hashmi: Tutorial on JUnit & JFCUnit (pdf)

Topic: Model Checking

Presenter 1: Susmita Ghose (Slides - ppt, pdf)

Paper(s): Using model checking to generate tests from requirements specifications

Angelo Gargantini, Constance Heitmeyer

ACM SIGSOFT Software Engineering Notes, Proceedings of the 7th European Engineering Conference held jointly with the 7th ACM SIGSOFT symposium on Foundations of software engineering October 1999
Volume 24 Issue 6

 

Presenter 2: Avik Sinha (Slides - ppt, pdf)

Paper(s): SMC: a symmetry-based model checker for verification of safety and liveness properties

A. Prasad Sistla, Viktor Gyuris, E. Allen Emerson

ACM Transactions on Software Engineering and Methodology (TOSEM) April 2000
Volume 9 Issue 2

13

Nov. 26

Student Presentations (Nov. 26)

Topic: Test Case Prioritization

Presenter 1: Cyntrica Eaton (Slides - ppt, pdf)

Paper(s): Effectively prioritizing tests in development environment

Amitabh Srivastava, Jay Thiagarajan

ACM SIGSOFT Software Engineering Notes, Proceedings of the international symposium on Software testing and analysis May 2002, Volume 27 Issue 3

 

Presenter 2: Harsh Nanda (Slides - ppt, pdf)

Paper(s): Test case prioritization: a family of empirical studies
Elbaum, S.; Malishevsky, A.G.; Rothermel, G.
Software Engineering, IEEE Transactions on, Volume: 28 Issue: 2, Feb. 2002
Page(s): 159 –182

14

Dec. 3-5

Student Presentations

Topic: Testing Concurrent Programs

Presenter 1: Chuk Yang Seng (Slides - ppt, pdf)

Paper(s): Incremental integration testing of concurrent programs
Koppol, P.V.; Carver, R.H.; Kuo-Chung Tai
Software Engineering, IEEE Transactions on, Volume: 28 Issue: 6, June 2002
Page(s): 607 -623

 

Presenter 2: Hyma Murthy (Slides - ppt, pdf)

Paper(s): All-du-path coverage for parallel programs

Cheer-Sun D. Yang, Amie L. Souter , Lori L. Pollock,

ACM SIGSOFT Software Engineering Notes, Proceedings of ACM SIGSOFT international symposium on Software testing and analysis March 1998
Volume 23 Issue 2

 

Topic: Software Testing Maturity Model (SW-TMM)

Presenter: Duy Huynh (Slides - ppt, pdf)

Paper(s): Using SW-TMM to Improve the Testing Process,

Thomas C. Staab,
CROSSTALK, The Journal of Defense Software Engineering, Nov. 2002.

 

Topic: Testing Object oriented Software

Presenter: Dave Tahmoush (Slides - ppt, pdf)

Paper(s): Incremental Testing Of Object-Oriented Class Structures
Harrold, M.J.; McGregor, J.D.; Fitzpatrick, K.J.
Software Engineering, 1992. International Conference on, 1992
Page(s): 68 –80

15

Dec. 10-12

Student Project Presentations

Dec. 10: {Aleks, Harsh, Scott, Adithya, Dave, Arunchandar, Dongfeng}

Dec. 12: {Cyntrica, Susmita, Duy, Hyma, Chuk, Anil, Avik}

 

Student Projects

Students should choose a project with the consent of the instructor. The student(s) may develop a new testing technique for one of the following software:

        Object-oriented

        Component-based

        Concurrent

       Distributed

       Graphical-user Interface

       Web

       OR suggest your own project.

A one-page project proposal, (due on Oct. 29), summarizing your project and your new technique will be presented (8-10 minutes) in class.

The student must experimentally demonstrate that the new technique can be successfully used to test the software. A project report must be submitted at the end of the semester. The report should contain the following sections:

1.     Abstract

2.     Introduction (motivation, importance of testing the particular type of software, one paragraph on why existing techniques are insufficient, your technique and its key contributions)

3.     Detailed description of your idea (several sections, if necessary)

4.     Algorithms (pseudo-code) and Experiments (graphs/tables)

5.     Related Work (weaknesses of each related approach)

6.     Conclusions (weaknesses of your technique and possible future directions)

Remember, suggestions for new projects are welcome.

Student Presentations

 

Students must present a paper (perhaps from the list of suggested papers) on one of the following topics:

o      Combining and Comparing Techniques

o      Defect and Failure Estimation and Analysis

o      Model Checking

o      Test Case Prioritization

o      Testing Concurrent Programs

o      Testing Object oriented Software

o      Testing Spreadsheets

o      Web Testing

 

Each presentation will be 30 minutes long, including 5 minutes for questions/discussion. The presentation will be divided into the following parts:

       Problem definition/motivation

       What are the challenges?

       Background literature surveyed by the authors

       Specific technique developed in this paper

       Weaknesses of the technique

 

Since two students will present two different papers from each topic on the same day, they may work together to avoid repetitions. Students will need to develop a power-point presentation, which will be made available on the course web page at least 2 days before the presentation. The final exam will be based on the contents of these presentations.

Assessment

       25% Mid-term Exam.

       25% Final Exam.

       20% Topic Presentation (40 minutes).

       5% Project Presentation (10 minutes).

       25% Term Project (chosen by the student and approved by the instructor. May be a team (2 students) project, depending on its scope).

It is expected that all students understand University policies on academic honesty. Cheating on assignments or exams is very serious and will not be tolerated in this class.   It is permissible to talk to other students about assignments and to discuss particular solutions. However, you are not to share code with anyone except your partner.

Important Dates – Fall 2002

o      Sep. 2, 2002: Campus Closed - Labor Day Holiday

o      Sep. 3, 2002: First Class - Fall 2002 semester

o      Oct. 31, 2002: Midterm Exam

o      Nov. 28-29, 2002: Thanksgiving Recess

o      Dec. 13, 2002:  Last Class, Fall 2002

o      Dec. xx, 2002: Final Exam