Workshop on the Evaluation of Software Defect Detection Tools

Sunday, June 12th, 2005
Co-located with PLDI 2005

Workshop co-chairs: William Pugh (University of Maryland) and Jim Larus (Microsoft Research)

Program chair: Dawson Engler (Stanford University)

Program Committee: Andy Chou, Manuvir Das, Michael Ernst, Cormac Flanagan, Dan Grossman, Jonathan Pincus, Andreas Zeller

8:30 am

Discussion on Soundness

  • The Soundness of Bugs is What Matters, Patrice Godefroid, Bell Laboratories, Lucent Technologies
  • Soundness and its Role in Bug Detection Systems, Yichen Xie, Mayur Naik, Brian Hackett, Alex Aiken, Stanford University
9:15 am


9:30 am

Research presentations

  • Locating Matching Method Calls by Mining Revision History Data, Benjamin Livshits, Thomas Zimmermann, Stanford University
  • Evaluating a Lightweight Defect Localization Tool, Valentin Dallmeier, Christian Lindig, Andreas Zeller, Saarland University
10:30 am


10:45 am

Invited talk on Deployment and Adoption, Manuvir Das, Microsoft

11:15 am

Discussion of Deployment and Adoption

  • The Open Source Proving Grounds, Ben Liblit, University of Wisconsin-Madison
  • Issues in deploying SW defect detection tools, David Cok, Eastman Kodak R&D
  • False Positives Over Time: A Problem in Deploying Static Analysis Tools, Andy Chou, Coverity
12 noon


1:00 pm

Research presentations

  • Model Checking x86 Executables with CodeSurfer/x86 and WPDS++, Gogul Balakrishnan, Thomas Reps, Nick Kidd, Akash Lal, Junghee Lim, David Melski, Radu Gruian, Suan Yong, Chi-Hua Chen, Tim Teitelbaum, Univ. of Wisconsin
  • Empowering Software Debugging Through Architectural Support for Program Rollback, Radu Teodorescu, Josep Torrellas, UIUC Computer Science
  • EXPLODE: A Lightweight, General Approach to Finding Serious Errors in Storage Systems, Junfeng Yang, Paul Twohey, Ben Pfaff, Can Sar, Dawson Engler, Stanford University
2:30 pm


2:45 pm

Research presentations

  • Experience from Developing the Dialyzer: A Static Analysis Tool Detecting Defects in Erlang Applications, Kostis Sagonas, Uppsala University
  • Soundness by Static Analysis and False-alarm Removal by Statistical Analysis: Our Airac Experience, Yungbum Jung, Jaehwang Kim, Jaeho Sin, Kwangkeun Yi, Seoul National University
3:45 pm


4:00 pm

Discussion of Benchmarking

  • Dynamic Buffer Overflow Detection, Michael Zhivich, Tim Leek, Richard Lippmann, MIT Lincoln Laboratory
  • Using a Diagnostic Corpus of C Programs to Evaluate Buffer Overflow Detection by Static Analysis Tools, Kendra Kratkiewicz, Richard Lippmann, MIT Lincoln Laboratory
  • BugBench: A Benchmark for Evaluating Bug Detection Tools, Shan Lu, Zhenmin Li, Feng Qin, Lin Tan, Pin Zhou, Yuanyuan Zhou, UIUC
  • Benchmarking Bug Detection Tools. Roger Thornton, Fortify Software
  • A Call for a Public Bug and Tool Registry, Jeffrey Foster, Univ. of Maryland
  • Bug Specimens are Important, Jaime Spacco, David Hovemeyer, William Pugh, University Maryland
  • NIST Software Assurance Metrics and Tool Evaluation (SAMATE) Project, Michael Kass, NIST
5:00 pm

Discussion of New Ideas

  • Deploying Architectural Support for Software Defect Detection in Future Processors, Yuanyuan Zhou, Josep Torrellas, UIUC
  • Using Historical Information to Improve Bug Finding Techniques, Chadd Williams, Jeffrey Hollingsworth, Univ. of Maryland
  • Locating defects is uncertain, Andreas Zeller, Saarland University
  • Is a Bug Avoidable and How Could It Be Found?, Dan Grossman, Univ. of Washington
5:45 pm

wrap up and discussion of future workshops

6:00 pm