CMSC838M: Advanced Topics in Software Testing
This course will examine advanced software testing techniques. In particular, the important phases of testing will be reviewed, emphasizing on the significance of each phase when testing different types of software. Students will learn the state of the art in testing technology for object-oriented, component-based, concurrent, distributed, graphical-user interface, and web software. In addition, closely related concepts such as model checking and program analysis will also be studied. Emerging concepts such as test-case prioritization and their impact on testing will be examined. Students will gain hands-on testing/analysis experience by proposing new solutions to open research problems in the field of software testing and experimentally demonstrating the strengths/weaknesses of their solutions.
By the end of this course, students should be familiar with the state-of-the-art in software testing. Students should also be aware of the major open problems in testing.
Is the course valid for PhD qualifying coursework? |
Yes (Software Engineering/Programming Languages) |
Is the course valid for MS qualifying coursework? |
Yes (Software Engineering/Programming Languages) |
Is the course valid for MS comps? |
Yes. (Both Midterm and Final exams count towards the MS comps.) |
Tue.Thu - 12:30PM - 1:45PM (CSI 2107)
Tue.Thu - 1:45PM - 4:00PM (4115 A. V.
Williams Building)
CMSC 435 or equivalent.
(Subject to change; additional readings may be assigned)
Week
|
Dates
|
Topic |
Readings
(copies available in the instructor’s
office) |
1 |
Sep. 3-5 |
Course Overview (Slides) |
Mary
Jean Harrold Proceedings
of the conference on The future of Software engineering May 2000 Introduction to special section on software testing R.
Hamlet Communications
of the ACM
June 1988 Testing: principles and practice Stephen
R. Schach ACM
Computing Surveys (CSUR) March 1996 |
2 |
Sep. 10-12 |
Introduction to Testing
(preliminary concepts – testability, whitebox, blackbox, unit, integration, mutation) (Slides) |
Software safety: why, what, and how Nancy G.
Leveson ACM
Computing Surveys (CSUR) June 1986 Validation, Verification, and Testing of Computer Software
W.
Richards Adrion , Martha A. Branstad , John C. Cherniavsky ACM
Computing Surveys (CSUR) June 1982 |
3 |
Sep. 17-19 |
Taxonomies of techniques, Test-case Generation I |
Rethinking the taxonomy of fault detection techniques Michael
Young, Richard N. Taylor Proceedings
of the 11th international conference on Software engineering May 1989 A test generation strategy for pairwise testing The category-partition method for specifying and generating
fuctional tests T. J.
Ostrand , M. J. Balcer Communications
of the ACM
June 1988 |
4 |
Sep. 24-26 |
Test-case Generation II |
Hierarchical GUI test case generation using
automated planning Using a goal-driven approach to generate test cases for GUIs
Atif M.
Memon, Martha E. Pollack, Mary Lou Soffa Proceedings
of the 21st international conference on Software engineering May 1999 Predicate-based test generation for computer
programs |
5 |
Oct. 1-3 |
Test-case Generation III (Slides) Test Coverage I |
A heuristic approach for test case generation Kai-Hsiung
Chang, W. Homer Carlisle, James H. Cross, David B. Brown Proceedings
of the 19th annual conference on Computer Science April 1999 Software unit test coverage and adequacy Hong
Zhu, Patrick A. V. Hall, John H. R. May ACM
Computing Surveys (CSUR) December 1997 The evaluation of program-based software test data adequacy
criteria E. J.
Weyuker Communications
of the ACM
June 1988 Volume 31 Issue 6 Coverage criteria for GUI testing Atif M.
Memon, Mary Lou Soffa, Martha E. Pollack ACM
SIGSOFT Software Engineering Notes, Proceedings of the 8th European software
engineering conference held jointly with 9th ACM SIGSOFT symposium on
Foundations of software engineering September 2001 |
6 |
Oct. 8-10 |
Data-flow Analysis, Data-flow Testing (Slides) |
Interprocedual data flow testing M.
Harrold, M. Soffa ACM
SIGSOFT Software Engineering Notes, Proceedings of the ACM SIGSOFT '89 third
symposium on Software testing, analysis, and verification November 1989 An applicable family of data flow testing criteria |
7 |
Oct. 15-17 |
Test Oracles (Slides) |
Automated test oracles for GUIs Atif M.
Memon, Martha E. Pollack, Mary Lou Soffa ACM
SIGSOFT Software Engineering Notes, Proceedings of the eighth international
symposium on Foundations of software engineering for twenty-first century
applications
November 2000 Specification-based test oracles for reactive systems Debra J.
Richardson, Stephanie Leif Aha, T. Owen O'Malley Proceedings
of the 14th international conference on Software engineering June 1992 |
8 |
Oct. 22-24 |
Test Coverage II (Slides) & Regression Testing (Slides) & Student Project Proposals {Scott McMaster, Avik
Sinha, Arunchandar Vasan, Harsh Nanda, Duy Huynh, Nada Hashmi, Adithya
Nagarajan, Aleks Aris} |
Experiments on the effectiveness of dataflow- and
control-flow-based test adequacy criteria An empirical study of regression test selection techniques
Todd L.
Graves, Mary Jean Harrold, Jung-Min Kim, Adam Porter, Gregg Rothermel ACM
Transactions on Software Engineering and Methodology (TOSEM) April 2001 |
9 |
Oct. 29-31 |
Student Project Proposals {Cyntrica N Eaton,
Susmita Ghose, Damayanti Gupta, Hyma S Murthy, Chuk Yang Seng, Anil Sharma, Dave Alan
Tahmoush, Dongfeng Zhu} & Mid-term exam (Oct. 31) |
|
10 |
Nov. 5-7 |
Student Presentations |
Topic: Testing SpreadsheetsPresenter 1: Alex Aris (Slides - ppt,
pdf) Paper(s): WYSIWYT testing in the spreadsheet paradigm: an empirical
evaluation Karen J.
Rothermel, Curtis R. Cook, Margaret M. Burnett, Justin Schonfeld, T. R. G.
Green, Gregg Rothermel Proceedings
of the 22nd international conference on Software engineering June 2000 Topic: Network Security Testing
Presenter 1: Anil Sharma (Slides -
ppt,
pdf) Paper(s): Using model checking to analyze network
vulnerabilities Topic: Web TestingPresenter 1: Adithya Nagarajan
(Slides - ppt,
pdf) Paper(s): Analysis and testing of web applications Going Faster:
Testing The Web Application (NOTE: Requires IEEE Computer society password) Edward Hieatt, Robert Mee IEEE Software, March/April
2002 (Vol. 19, No. 2), pp.
60-65 Presenter 2: Arunchandar Vasan
(Slides - ppt,
pdf) Paper(s): An object-oriented web test model for testing Web
applications Using the WWW Test Pattern to check HTML client
compliance |
11 |
Nov. 12-14 |
Student Presentations |
Topic: Combining and Comparing Techniques Presenter 1: Nada Hashmi (Slides - ppt, pdf) Paper(s): On comparisons of random, partition, and
proportional partition testing Presenter 2: Damayanti Gupta
(Slides - ppt,
pdf) Paper(s): Further empirical studies of test effectiveness Phyllis
G. Frankl , Oleg Iakounenko ACM
SIGSOFT Software Engineering Notes, Proceedings of the 6th ACM SIGSOFT
international symposium on Foundations of software engineering November 1998 Topic: Defect and Failure Estimation and Analysis Presenter 1: Scott McMaster
(Slides - ppt,
pdf) Paper(s): Predicting fault incidence using software change
history Presenter 2: Dongfeng Zhu (Slides
- ppt,
pdf) Paper(s): A critique of software defect prediction models |
12 |
Nov. 19-21 |
Student Presentations (Nov.
19) & Testing Tools (Nov. 21) |
Topic: Model CheckingPresenter 1: Susmita Ghose (Slides
- ppt,
pdf) Paper(s): Using model checking to generate tests from requirements
specifications Angelo
Gargantini, Constance Heitmeyer ACM
SIGSOFT Software Engineering Notes, Proceedings of the 7th European
Engineering Conference held jointly with the 7th ACM SIGSOFT symposium on
Foundations of software engineering October 1999 Presenter 2: Avik Sinha (Slides - ppt,
pdf) Paper(s): SMC: a symmetry-based model checker for verification of
safety and liveness properties A.
Prasad Sistla, Viktor Gyuris, E. Allen Emerson ACM
Transactions on Software Engineering and Methodology (TOSEM) April 2000 |
13 |
Nov. 26 |
Student Presentations (Nov.
26) |
Topic: Test Case PrioritizationPresenter 1: Cyntrica Eaton
(Slides - ppt,
pdf) Paper(s): Effectively prioritizing tests in development environment
Amitabh
Srivastava, Jay Thiagarajan ACM
SIGSOFT Software Engineering Notes, Proceedings of the international
symposium on Software testing and analysis May 2002, Volume 27 Issue 3 Presenter 2: Harsh Nanda (Slides -
ppt,
pdf) Paper(s): Test case prioritization: a family of empirical
studies |
14 |
Dec. 3-5 |
Student Presentations |
Topic: Testing Concurrent ProgramsPresenter 1: Chuk Yang Seng
(Slides - ppt,
pdf) Paper(s): Incremental integration testing of concurrent
programs Presenter 2: Hyma Murthy (Slides -
ppt,
pdf) Paper(s): All-du-path coverage for parallel programs Cheer-Sun
D. Yang, Amie L. Souter , Lori L. Pollock, ACM
SIGSOFT Software Engineering Notes, Proceedings of ACM SIGSOFT international
symposium on Software testing and analysis March 1998 Topic: Software Testing Maturity Model (SW-TMM)Presenter: Duy Huynh (Slides - ppt,
pdf) Paper(s): Using SW-TMM to Improve the Testing Process, Thomas C. Staab, Topic: Testing Object oriented SoftwarePresenter: Dave Tahmoush (Slides
- ppt,
pdf) Paper(s): Incremental Testing Of Object-Oriented Class
Structures |
15 |
Dec. 10-12 |
Student Project Presentations Dec. 10: {Aleks, Harsh,
Scott, Adithya, Dave, Arunchandar, Dongfeng} Dec. 12: {Cyntrica,
Susmita, Duy, Hyma, Chuk, Anil, Avik} |
|
Students should choose a project with the consent of the instructor. The student(s) may develop a new testing technique for one of the following software:
· Object-oriented
· Component-based
· Concurrent
· Distributed
· Graphical-user Interface
· Web
·
OR suggest your own project.
A one-page project proposal, (due on Oct. 29), summarizing your project and your new technique will be presented (8-10 minutes) in class.
The student must experimentally demonstrate that the new technique can be successfully used to test the software. A project report must be submitted at the end of the semester. The report should contain the following sections:
1. Abstract
2. Introduction (motivation, importance of testing the particular type of software, one paragraph on why existing techniques are insufficient, your technique and its key contributions)
3. Detailed description of your idea (several sections, if necessary)
4. Algorithms (pseudo-code) and Experiments (graphs/tables)
5. Related Work (weaknesses of each related approach)
6. Conclusions (weaknesses of your technique and possible future directions)
Remember, suggestions for new projects are welcome.
Students must present a paper (perhaps from the list of suggested papers) on one of the following topics:
o Combining and Comparing Techniques
o Defect and Failure Estimation and Analysis
o Model Checking
o Test Case Prioritization
o Testing Concurrent Programs
o Testing Object oriented Software
o Testing Spreadsheets
o Web Testing
Each presentation will be 30 minutes long, including 5 minutes for questions/discussion. The presentation will be divided into the following parts:
· Problem definition/motivation
· What are the challenges?
· Background literature surveyed by the authors
· Specific technique developed in this paper
· Weaknesses of the technique
Since two students will present two different papers from each topic on the same day, they may work together to avoid repetitions. Students will need to develop a power-point presentation, which will be made available on the course web page at least 2 days before the presentation. The final exam will be based on the contents of these presentations.
· 25% Mid-term Exam.
· 25% Final Exam.
· 20% Topic Presentation (40 minutes).
· 5% Project Presentation (10 minutes).
·
25% Term Project (chosen by the student and approved by
the instructor. May be a team (2 students) project, depending on its scope).
It is expected that all students understand University policies on academic honesty. Cheating on assignments or exams is very serious and will not be tolerated in this class. It is permissible to talk to other students about assignments and to discuss particular solutions. However, you are not to share code with anyone except your partner.
o Sep. 2, 2002: Campus Closed - Labor Day Holiday
o Sep. 3, 2002: First Class - Fall 2002 semester
o Oct. 31, 2002: Midterm Exam
o Nov. 28-29, 2002: Thanksgiving Recess
o Dec. 13, 2002:
Last Class, Fall 2002
o Dec. xx, 2002: Final Exam