The Observe-Model-Exercise*  Benchmark

Experimentation Tools

List of tools use by the paper. The sample application is used to demonstrate how the tools work.
Refer to the README file in each tool for more detail usage instructions.

  1. Model builders and test case generators: Tools for initially riping applications, enhancing the model with runtime observations and generating test cases from the obtained models.
  2. Test case executor: A tool for executing test cases and collecting runtime information (i.e., GUI states, logs, etc).
  3. Random GUI walker: A tool for performing automated random-walk on the GUI.
  4. Customized Instrumentor: A customized version of Cobertura to collect additional method and class coverage.
  5. Sample application: An application used to demonstrate how the tools are used (see the README files).
All tools in one package

Experimentation Artifacts

List of 8 popular open-source subject applications. Click on each individual links bellow for experimentation artifacts of each application.

  1. ArgoUML
  2. Buddi
  3. CrosswordSage
  4. DrJava
  5. JabRef
  6. OmegaT
  7. Pdf Split and Merge
  8. Rachota

ArgoUML

Summary

Application Under Test

Experimentation Artifacts 

Model Visualization & Code Coverage Obtained

Clicking on the images to zoom in and the below links for detail code coverages






Baseline
coverage
Iteration 1
coverage
Iteration 2
coverage
Iteration 3
coverage
Iteration 4
coverage

Fault report

ID
Summary
Report
AU1
Incorrect log information when delete object with a Baselineank document
http://argouml.tigris.org/issues/show_bug.cgi?id=6202
AU2
Exception with invalid input file name (/crash/crash/) for Export Graphic
http://argouml.tigris.org/issues/show_bug.cgi?id=6205
AU3
Exception with invalid input file name (/crash/crash/) for Export All Graphic
http://argouml.tigris.org/issues/show_bug.cgi?id=6205
AU4
ProfileException with unexisting input file name Properties/Load Profile from file
http://argouml.tigris.org/issues/show_bug.cgi?id=6206
AU5
NullPointerException with Layout in ActivityDiagram
http://argouml.tigris.org/issues/show_bug.cgi?id=6207
AU6
NullPointerException when trying Revert to Saved an unsaved document
http://argouml.tigris.org/issues/show_bug.cgi?id=6237
AU7
InvalidObjectException when interact between Open and Save Project
http://argouml.tigris.org/issues/show_bug.cgi?id=6394

Back to top

Buddi

Summary


Application Under Test

Experimentation Artifacts 

Model Visualization & Code Coverage Obtained

Clicking on the images to zoom in and the below links for detail code coverages






Baseline
coverage
Iteration 1
coverage
Iteration 2
coverage
Iteration 3
coverage
Iteration 4
coverage
Iteration 5
coverage

Fault report

ID
Summary
Report
BD1
java.lang.IllegalArgumentException with an out-of-range Proxy Port
https://sourceforge.net/tracker/?func=detail&aid=3184045&group_id=167026&atid=841302
BD2
UnknownHostException when updating with a wrong proxy address
https://sourceforge.net/tracker/?func=detail&aid=3181764&group_id=167026&atid=841302
BD3
InvalidValueException in  Edit Scheduled Transactions window with an empty transaction name
http://sourceforge.net/tracker/?func=detail&aid=3181739&group_id=167026&atid=841302
BD4
FileNotFoundException  when saving with a non-encrypted file with an invalid name
https://sourceforge.net/tracker/index.php?func=detail&aid=3413056&group_id=167026&atid=841302#
BD5
FileNotFoundException  when saving with an encrypted file with an invalid name
https://sourceforge.net/tracker/index.php?func=detail&aid=3413056&group_id=167026&atid=841302#
BD6
ZipException with an invalid plugin name 
https://sourceforge.net/tracker/?func=detail&aid=3413069&group_id=167026&atid=841302
BD7
FileNotFoundException with an non-existing language file
https://sourceforge.net/tracker/index.php?func=detail&aid=3413079&group_id=167026&atid=841302

Back to top

CrosswordSage

Summary

Application Under Test

Experimentation Artifacts 

Model Visualization & Code Coverage Obtained

Clicking on the images to zoom in and the below links for detail code coverages





Baseline
coverage
Iteration 1
coverage



Fault report

ID
Summary
Report
CS1
NullPointerException when loading an empty CrossWord
http://sourceforge.net/tracker/?func=detail&aid=1532566&group_id=148348&atid=771298
CS2
Null Pointer when printing with an invalid printer
http://sourceforge.net/tracker/?func=detail&aid=1532557&group_id=148348&atid=771298
CS3
NullPointer with Cancel button in Split Word
http://sourceforge.net/tracker/?func=detail&aid=1532566&group_id=148348&atid=771298
CS4
NullPointer when PuBaselineishing an empty crossword
http://sourceforge.net/tracker/?func=detail&aid=1532560&group_id=148348&atid=771298
CS5
FileNotFoundException when providing an invalid crossword list file
TO REPORT
CS6
FileNotFoundException when providing an invalid file name to save
TO REPORT
CS7
NullPointerException with Write Clue
TO REPORT
CS8
NullPointerException with Suggest Word
TO REPORT

Back to top


DrJava

Summary

Application Under Test

Experimentation Artifacts 

Model Visualization & Code Coverage Obtained

Clicking on the images to zoom in and the below links for detail code coverages






Baseline
coverage
Iteration 1
coverage
Iteration 2
coverage
Iteration 3
coverage


Fault report

ID
Summary
Report
DJ1
IOException when entering invalid filename in Executing History window
https://sourceforge.net/tracker/?func=detail&aid=3192492&group_id=44253&atid=438935
DJ2
IOException when saving file with invalid file names
https://sourceforge.net/tracker/?func=detail&aid=3192492&group_id=44253&atid=438935
DJ3
Crash when uncommenting in an empty file
https://sourceforge.net/tracker/?func=detail&atid=438935&aid=2959994&group_id=44253
DJ4
GUI error when a modal window is still selectaBaselinee
https://sourceforge.net/tracker/?func=detail&aid=2962878&group_id=44253&atid=438935

Back to top


JabRef

Summary

Application Under Test

Experimentation Artifacts 

Model Visualization & Code Coverage Obtained

Clicking on the images to zoom in and the below links for detail code coverages





Baseline
coverage
Iteration 1
coverage
Iteration 2
coverage
Iteration 3
coverage
Iteration 4
coverage

Fault report

ID
Summary
Report
JR1
FileNotFoundException in Journal abbreviations window
https://sourceforge.net/tracker/?func=detail&aid=3196238&group_id=92314&atid=600306
JR2
StringIndexOutOfBoundsException with invalid Preview text in Preferences
https://sourceforge.net/tracker/?func=detail&aid=3196223&group_id=92314&atid=600306
JR3
ArrayIndexOutOfBoundsException  when generating keyword with field name disaBaselineed
https://sourceforge.net/tracker/?func=detail&aid=3193488&group_id=92314&atid=600306
JR4
ArrayIndexOutOfBoundsException when generating keyword with a closed database
https://sourceforge.net/tracker/?func=detail&aid=3193488&group_id=92314&atid=600306
JR5
java.util.regex.Pattern.error when searching with an illegal string
https://sourceforge.net/tracker/?func=detail&aid=3200762&group_id=92314&atid=600306
JR6
java.io.IOException when setting default owner name  containing a "{" character (under Preferences)
https://sourceforge.net/tracker/?func=detail&aid=3191737&group_id=92314&atid=600306
JR7
ClassNotFoundException with invalid Look and Feel https://sourceforge.net/tracker/?func=detail&aid=3184025&group_id=92314&atid=600306
JR8
ServerSocketException with an invalid Proxy port
https://sourceforge.net/tracker/?func=detail&aid=3184019&group_id=92314&atid=600306
JR9
ZipException with invalid zip file name
https://sourceforge.net/tracker/?func=detail&aid=3179823&group_id=92314&atid=600306
JR10
NullPointerException with invalid import file name and Cancel
https://sourceforge.net/tracker/?func=detail&aid=3179824&group_id=92314&atid=600306
JR11
NullPointerException with invalid import folder name
https://sourceforge.net/tracker/?func=detail&aid=3179824&group_id=92314&atid=600306
JR12
ArrayIndexOutOfBoundsException when changing the properties of a closed database
https://sourceforge.net/tracker/?func=detail&aid=3413917&group_id=92314&atid=600306
JR13
StringIndexOutOfBoundsException with non-existing ImportFormat Subclass
https://sourceforge.net/tracker/?func=detail&aid=3179824&group_id=92314&atid=600306

Back to top


OmegaT

Summary

Application Under Test

Experimentation Artifacts 

Model Visualization & Code Coverage Obtained

Clicking on the images to zoom in and the below links for detail code coverages






Baseline
coverage
Iteration 1
coverage
Iteration 2
coverage
Iteration 3
coverage

Fault report

ID
Summary
Report
OT1
NullPointerException  with invalid dictionary file folder in Spell Checker window
http://sourceforge.net/tracker/?func=detail&aid=2993845&group_id=68187&atid=520347
OT2
NullPointerException with Spell Checker window when the plugin name is Baselineank
https://sourceforge.net/tracker/?func=detail&aid=2978134&group_id=68187&atid=520347
OT3
MalformedURLException with Import From MediaWiki... https://sourceforge.net/tracker/?func=detail&aid=2978138&group_id=68187&atid=520347
OT4
NullPointerException with various menu items on a Baselineank project
https://sourceforge.net/tracker/?func=detail&atid=520347&aid=2970745&group_id=68187

Back to top


Pdf Split and Merge

Summary

Application Under Test

Experimentation Artifacts 

Model Visualization & Code Coverage Obtained

Clicking on the images to zoom in and the below links for detail code coverages






Baseline
coverage
Iteration 1
coverage




Fault report

ID
Summary
Report
PS1
ParseException on Alternative Mix plugin with non-numeric value of Number of pages
https://sourceforge.net/tracker/?func=detail&aid=3197420&group_id=160044&atid=814265
PS2
ParseException on Splite plugin with non-numeric value of split after these pages
https://sourceforge.net/tracker/?func=detail&aid=3197420&group_id=160044&atid=814265
PS3
ParseException on Splite plugin with non-numeric value of split every "n" pages
https://sourceforge.net/tracker/?func=detail&aid=3197420&group_id=160044&atid=814265
PS4
ParseException in Splite plugin with non-numeric value for split at this size
https://sourceforge.net/tracker/?func=detail&aid=3197420&group_id=160044&atid=814265
PS5
NullPointerException with split by bookmarks level
http://sourceforge.net/tracker/?func=detail&aid=3283869&group_id=160044&atid=814265
PS6
FileNotFoundException with non-existing output directory for All Plugins
http://sourceforge.net/tracker/?func=detail&aid=3283869&group_id=160044&atid=814265
PS7
FileNotFoundException  when saving with an invalid environment file name
https://sourceforge.net/tracker/?func=detail&aid=3413481&group_id=160044&atid=814265

Back to top


Rachota

Summary

Application Under Test

Experimentation Artifacts 

Model Visualization & Code Coverage Obtained

Clicking on the images to zoom in and the below links for detail code coverages






Baseline
coverage
Iteration 1
coverage
Iteration 2
coverage
Iteration 3
coverage
Iteration 4
coverage

Fault report

ID
Summary
Report
RC1
NullPointerException  when generating report with a non-existing file name
https://sourceforge.net/tracker/?func=detail&aid=3195188&group_id=144949&atid=760391
RC2
NullPointerException  when adjusting time after changing the active day in the main menu by clicking "Previous" day
https://sourceforge.net/tracker/?func=detail&aid=3184271&group_id=144949&atid=760391
RC3
NumberFormatException when entering invalid number to  Inactivity time
https://sourceforge.net/tracker/?func=detail&aid=2960289&group_id=144949&atid=760391
RC4
NumberFormatException when entering invalid number to Hibernation time
https://sourceforge.net/tracker/?func=detail&aid=2960289&group_id=144949&atid=760391
RC5
NumberFormatException when entering invalid number to Working hours
https://sourceforge.net/tracker/?func=detail&aid=2960289&group_id=144949&atid=760391
RC6
Crash with invalid format xml data file
https://sourceforge.net/tracker/?func=detail&aid=2955215&group_id=144949&atid=760391
RC7
NullPointerException  when adjusting starting time and correcting time simultaneously
https://sourceforge.net/tracker/?func=detail&aid=3184271&group_id=144949&atid=760391

Back to top