Notice
This page contains a supplementary data for our TSE paper. While it
is under review, please do not link to, or share this page. We will make this page public upon publication of this work.
Abstract
System testing of a GUI-based application requires that test cases, consisting of sequences of user actions/events,
be executed and the software’s output be verified. To enable automated re-testing, such test cases are increasingly being coded
as low-level test scripts, to be replayed automatically using test harnesses. Whenever the GUI changes—widgets get moved
around, windows get merged—some scripts become unusable because they no longer encode valid input sequences. Moreover,
because the software’s output may have changed, their test oracles—assertions and checkpoints—encoded in the scripts may
no longer correctly check the intended GUI objects. We present ScrIpT repAireR (SITAR), a technique to automatically repair
unusable low-level test scripts. SITAR uses reverse engineering techniques to create an abstract test for each script, maps it
to an annotated event-flow graph (EFG), uses repairing transformations and human input to repair the test, and synthesizes
a new “repaired” test script. During this process, SITAR also repairs the reference to the GUI objects used in the checkpoints
yielding a final test script that can be executed automatically to validate the revised software. SITAR ammortizes the cost of
human intervention across multiple scripts by accumulating the human knowledge as annotations on the EFG. An experiment
using QTP test scripts suggest that SITAR is effective in that 41–89% unusable test scripts were repaired.
Annotations significantly reduced human cost after 20% test
scripts had been repaired.
Framework
A logical perspective of the components of SITAR is
shown in the figure on the left.
SITAR deals with three distinct
spaces (levels of abstraction): (1) execution, (2) script,
and (3) model.
Ripping raises the level from execution
to model.
And mapping first raises the level of
abstraction from script (for version n) to model and then, once the scripts have been repaired, from the
model back to script (version n+1).
Experiment Settings
- Tool Versions
- HP QTP (Quick Test Professional): 11.00
- Java Runtime: Oracle JDK 1.6.0_45
- Cobertura (for code coverage): 1.9.4.1
- Subject Applications
- PDFSam
- PDF1 Version: 1.2.0
- PDF2 Version: 2.0.0
- OmegaT
- OT1 Version: 1.8.1 07
- OT2 Version: 2.0.5 03
Supplemental Data
The QTP scripts used in our experiments are available for download from the following links. The scripts are named as StudentId_FUID_TCID where FUID is the id of the functional unit and TCID is the id of the test case. To view the contents of a script, please go to "Action1/Scripts.mts" under the directory of the test case.
Instructions for Creating Test Scripts
Functional Units of CrosswordSage
- Unit 1: Creating new crosswords
Requirements: Create new crosswords and cover the following operations in your test:
adding word, suggesting word, setting clue number, writing clue, getting clue, deleting selected word,
typing in word, saving crossword, etc. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 4 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Adding word
- Step 1: Click on button “Add Word.
- Step 2: Click inside the grid to add a word starting/ending at your selected position.
- 2. Suggesting word:
- Step 1: Select a word in the grid.
- Step 2: Click on button “Suggest Word”.
- Step 3: Select a suggested word.
- 3. Setting clue numbers:
- Step 1: Select a word in the grid (optional).
- Step 2: Click on button "Set Clue Numbers", then numbers will be set for all words in current puzzle.
- 4. Writing clue:
- Step 1: Select a word in the grid.
- Step 2: Click on button "Write Clue".
- Step 3: Enter clue in the text field in the dialog popped out.
- Step 4: Click on button "OK" to finish setting clue.
- 5. Getting clue:
- Step 1: Select a word in the grid.
- Step 2: Click on button "Get Clue", then the clue will show up in the text area at the bottom of window.
- 6. Deleting selected words:
- Step 1: Select a word in the grid.
- Step 2: Click on button "Delete Selected Word".
- 7. Saving crosswords:
- Step 1: Click on menu "File -- Save Crossword".
- Step 2: Select a destination file.
- Step 3: Click on "Open" to save.
- Unit 2. Loading and solving existing crossword from file
Requirements: Load existing crossword from file and cover the following operations in your test:
solving words, checking correctness of your solution, etc, and close the application in the end.
Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 2 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Loading crosswords:
- Step 1: Click on menuitem "File -- Load Crossword to Solve".
- Step 2: Select a source file.
- Step 3: Click on "Open" to open.
- 2. Checking solution:
- Step 1: Type in your solution. This could be further split into steps including selecting squares in the grid and then typing in characters.
- Step 2: Check the correctness of your solution.
- Step 3: You may go on with more steps depending on whether your solution is correct.
- Unit 3. Loading and editing existing crossword from file
Requirements: Load existing crossword from file and cover the following operations in your test:
adding word, suggesting word, setting clue number, writing clue, getting clue, deleting selected word,
typing in word, saving crossword, etc. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 2 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Loading crosswords:
- Step 1: Click on menuitem "File -- Load Crossword to Edit".
- Step 2: Select a source file.
- Step 3: Click on "Open" to open.
- 2. Adding word
- Step 1: Click on button “Add Word.
- Step 2: Click inside the grid to add a word starting/ending at your selected position.
- 3. Suggesting word:
- Step 1: Select a word in the grid.
- Step 2: Click on button “Suggest Word”.
- Step 3: Select a suggested word.
- 4. Setting clue numbers:
- Step 1: Select a word in the grid (optional).
- Step 2: Click on button "Set Clue Numbers", then numbers will be set for all words in current puzzle.
- 5. Writing clue:
- Step 1: Select a word in the grid.
- Step 2: Click on button "Write Clue".
- Step 3: Enter clue in the text field in the dialog popped out.
- Step 4: Click on button "OK" to finish setting clue.
- 6. Getting clue:
- Step 1: Select a word in the grid.
- Step 2: Click on button "Get Clue", then the clue will show up in the text area at the bottom of window.
- 7. Deleting selected words:
- Step 1: Select a word in the grid.
- Step 2: Click on button "Delete Selected Word".
- 8. Saving crosswords:
- Step 1: Click on menu "File -- Save Crossword".
- Step 2: Select a destination file.
- Step 3: Click on "Open" to save.
- Unit 4. Printing current crossword
Requirements: Print current crossword and cover operations like entering title in your test. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 1 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Printing crosswords:
- Step 1: Select the crossword to print.
- Step 2: Click on print.
- Step 3: Input title.
- Unit 5. Utilities for solving new word
Requirements: Enter the view for solving new word and cover the following operations: finding matches by pattern, and finding matches by anagram. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 2 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Finding matches by pattern:
- Step 1: Select menuitem "Tools -- Solve New Word" to enter the view.
- Step 2: Input pattern.
- Step 3: Click on button "Find Possible matches".
- 2. Finding matches by anagram:
- Step 1: Select menuitem "Tools -- Solve New Word" to enter the view.
- Step 2: Input anagram.
- Step 3: Click on button "Find Possible matches".
Functional Units of PDFSam
- Unit 1. Splitting PDF files
Requirements: Split PDF files and cover the following operations: adding files to split, removing file to split, viewing document properties, setting split options, setting destination foler, setting output options, etc. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 2 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Adding a file to split:
- Step 1: Click on "Add".
- Step 2: Select file in the file chooser.
- Step 3: Click on "Open" to add file.
- 2. Removing the file to split:
- Step 1: Select a file previously added.
- Step 2: Click on "Remove" to remove file.
- 3. Viewing document properties:
- Step 1: Right click on a file.
- Step 2: Click on "Document properties" to view.
- Step 3: Close property viewer.
- 4. Setting split options:
- 4.1: Burst (split into single pages).
- 4.2: Split even pages.
- 4.3: Split Odd pages.
- 4.4: Split after a certain pages.
- 4.5: Split every n pages.
- 4.6: Split at a certain size.
- 4.7: Split by bookmark level.
- 5. Setting destination folder:
- 5.1: Same as source.
- 5.2: Choose a foler.
- 5.3: Overwrite if already exists.
- 5.4: Compress output file/files.
- 5.5: Output document PDF version.
- 6. Setting output options:
- 6.1: Set output file names prefix.
- Unit 2. Rotating PDF files
Requirements: Rotate PDF files and cover the following operations: adding files to rotate, removing files to rotate, clearing files to rotate, viewing file properties, setting rotation options, setting destination foler, setting output options, etc. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 2 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Adding a file to split:
- Step 1: Click on "Add".
- Step 2: Select file in the file chooser.
- Step 3: Click on "Open" to add file.
- 2. Removing files to rotate:
- Step 1: Select a file previously added.
- Step 2: Click on "Remove" to remove file.
- 2. Clearing files to rotate:
- Step 2: Click on "Clear" to clear files.
- 4. Viewing document properties:
- Step 1: Right click on a file.
- Step 2: Click on "Document properties" to view.
- Step 3: Close property viewer.
- 5. Setting rotation options:
- 5.1: Clockwise rotation degrees.
- 5.2: Pages to rotate.
- 6. Setting destination folder:
- 6.1: Same as source.
- 6.2: Choose a foler.
- 6.3: Overwrite if already exists.
- 6.4: Compress output file/files.
- 6.5: Output document PDF version.
- 7. Setting output options:
- 7.1: Set output file names prefix.
- Unit 3. Mixing PDF files
Requirements: Mix PDF files and cover the following operations: adding files to mix, removing files to mix, clearing files to rotate, viewing file properties, setting mix options, setting destination foler, etc. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 2 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Adding a file to mix:
- Step 1: Click on "Add".
- Step 2: Select file in the file chooser.
- Step 3: Click on "Open" to add file.
- 2. Removing files to mix:
- Step 1: Select a file previously added.
- Step 2: Click on "Remove" to remove file.
- 2. Clearing files to mix:
- Step 2: Click on "Clear" to clear files.
- 4. Viewing document properties:
- Step 1: Right click on a file.
- Step 2: Click on "Document properties" to view.
- Step 3: Close property viewer.
- 5. Setting mix options:
- 5.1: Reverse first document.
- 5.2: Reverse second document.
- 5.3: Number of pages to switch document.
- 6. Setting destination folder:
- 6.1: Same as source.
- 6.2: Choose a foler.
- 6.3: Overwrite if already exists.
- 6.4: Compress output file/files.
- 6.5: Output document PDF version.
- Unit 4. Merging PDF files
Requirements: Merge PDF files and cover the following operations: adding files to merge, removing files to merge, clearing files to merge, reordering files to merge, viewing file properties, setting merge options, setting destination foler, etc. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 2 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Adding a file to merge:
- Step 1: Click on "Add".
- Step 2: Select file in the file chooser.
- Step 3: Click on "Open" to add file.
- 2. Removing files to merge:
- Step 1: Select a file previously added.
- Step 2: Click on "Remove" to remove file.
- 2. Clearing files to merge:
- Step 2: Click on "Clear" to clear files.
- 4. Reordering files to merge:
- Step 1: Select a file previously added.
- Step 2: Click on "Move Up" or "Move Down" to adjust position of file in the list.
- 5. Viewing document properties:
- Step 1: Right click on a file.
- Step 2: Click on "Document properties" to view.
- Step 3: Close property viewer.
- 6. Setting merge options:
- 6.1: PDF documents contain forms.
- 7. Setting destination folder:
- 7.1: Same as source.
- 7.2: Choose a foler.
- 7.3: Overwrite if already exists.
- 7.4: Compress output file/files.
- 7.5: Output document PDF version.
- Unit 5. Saving and Loading environment
Requirements: Test features related with working environment and cover the following operations: saving environment, loading environment, etc. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 1 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Saving a working environment:
- Step 1: Select menuitem "File - Save Environment".
- Step 2: Select "My Document".
- Step 3: Input file name and click "Save".
- 2. Loading working environment from file:
- Step 1: Select menuitem "File - Load Environment".
- Step 2: Select "My Document".
- Step 3: Select file name and click "Open".
- Unit 6. Logging
Requirements: Test features related with logging and cover the following operations: saving logs, clearing logs, etc. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 1 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Saving logs:
- Step 1: Click tool bar icon "Save log".
- Step 2: Select "My Document".
- Step 3: Input file name and click "Save".
- 2. Clearing logs:
- Step 1: Click tool bar icon "Clear log".
- Unit 7. System settings
Requirements: Test features related with system settings. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 2 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Language.
- 2. Look and feel.
- 3. Theme.
- 4. Log level.
- 5. Check for updates.
- 6. Alert sounds.
- 7. Load default environment at startup.
- 8. Default working directory.
Functional Units of OmegaT
- Unit 1. Creating new projects
Requirements: Test features related with creating new projects. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 2 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Source file languages.
- 2. Translated files language.
- 3. Sentence-level segmenting.
- 4. Segmentation setup.
- 5. Creating a directory for source files.
- 6. Creating a directory for translation memory.
- 7. Creating a directory for glossary.
- 8. Creating a directory for translated files.
- Unit 2. Importing/reloading files/directories
Requirements: Import source files from a file/directory and cover the following operations: viewing file list in the project, selecting a certain file to translate, etc. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 2 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Importing files:
- Step 1: Click on "Import Source Files" from menu "Project" or window "Project Files".
- Step 2: Select a file to import.
- Step 3: Click on "Choose" to finish importing.
- 2. Importing directories:
- Step 1: Click on "Import Source Files" from menu "Project" or window "Project Files".
- Step 2: Select a directory to import.
- Step 3: Click on "Choose" to finish importing.
- 3. Reloading sources.
- Unit 3. Translating files
Requirements: Open or import source files to translate segment by segment and cover the following operations: marking translated/untranslated segments, navigating between segments, fuzzy matching operations, etc. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 6 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Openning/importing files:
- Step 1: Click on menuitem "Project -- Open" to open a project.
- Step 2: Select a project file in the chooser.
- Step 3: Click on "Choose" to open project.
- Step 4: Click on a file in the source list of the project to proceed with its translation.
- 2. Highlighting the text to be translated:
- Step 1: Click on a source file to translate.
- Step 2: Check if the source text are highlighted in green in the editing field.
- 3. Marking translated/untranslated segments:
- Step 1: Check/uncheck option "View -- Mark Translated Segment" to mark/unmark translated segments.
- Step 2: Check/uncheck option "View -- Mark Untranslated Segment" to mark/unmark untranslated segments.
- Step 3: Check/uncheck option "View -- Display source segments" to show/hide source segments.
- 4. Translate a segment:
- 4.1: Input your translation between "segment 0001" and "end segment".
- 4.2: Or select a matching result as translation.
- 5. Navigating between segments:
- 5.1: Go to next untranslated segment.
- 5.2: Go to next segment.
- 5.3: Go to previous segment.
- 5.4: Go to segment number.
- 5.3: Forward in history.
- 5.4: Backward in history.
- 6. Setting editing behavior options:
- 6.1: Click on "Options - Editting Behavior" to set default editing behavior for untranslated segments.
- 6.2: Select among the options for untranslated text: same as source text, leave the segment empty, insert the best fuzzy match, etc.
- 7. Translating source files of different format:
- 7.1: Text files.
- 7.2: PO files.
- 7.3: Java resource bundles.
- 7.4: INI (key=value) property files.
- 7.5: HTML and XHTML file.
- 7.6: OpenDocument/OpenOffice.org files.
- 7.7: MS open XML files.
- 8. If OmegaT found a fuzzy match with precision no less than 30%, it will be shown in the match viewer. You can select translations in the match viewer or using menu items.
- 9. Automatically insert the best fuzzy matches:
- Step 1: Click on "Options - Editting Behavior".
- Step 2: Check option "Insert the best fuzzy match".
- Step 3: Set minimal similarity and/or prefix.
- Unit 4. Setting options
Requirements: Test features related with setting options. Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 2 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. Use TAB to advance.
- 2. Always confirm quit, ensure this option is unchecked at the end of the test case.
- 3. Font.
- 4. File filters.
- 5. Segmentation.
- 6. Spell checking.
- 7. Editing behavior.
- 8. Restore main window.
- Unit 5. Help menu
Requirements: Test features related with menu "Help". Each test script should contain a meaningful flow of events to fulfill a certain task.
Make at least 1 different test scripts to cover the whole unit.
You may refer to the following guidelines for performing each operation:
- 1. OmegaT's users' manual.
- 2. OmegaT's about information.
This research is sponsored in part by National
Basic Research Program of China (973 Program
2014CB340702), NSFC Program (61170067 and
61373013). Any opinions, findings, and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views of
the NSFC.