Merging Errors and Faults as a Team
In this session subjects were asked to meet as a team and produce a
final list of faults and errors. Partly this was to see how difficult
of a job it would be to merge error lists.
Teams were assigned in the following way: we used a questionnaire to
assess subjects' experience in areas related to the PBR2 perspectives
they would be using in the rest of the experiment. We then
assigned reviewers to PBR2 perspectives in such a way as to provide
equal numbers of reviewers at each level of experience. Teams were
then created according to the following criteria:
- One member came from each of the PBR2 perspectives.
- All members of a team had the same level of experience.
Although the PBR2 perspectives had no bearing on this session, we
wanted to use the same teams in session 4C as in this session and so
composed PBR2 teams ahead of time. Naturally, other schemes for team
assignment can be used depending on the experimental goals.
On average, teams in our study took 2.3 hours to complete their meeting.
Description and Training
- We provided an overview of the session to our subjects. Again,
there are many details specific to our environment that will need to
be changed if this document is reused.
- We did not provide any training as to how the team should
accomplish its task of producing representative fault and error lists.
This was due to the fact that we were unsure what would be the best
way to approach merging lists at the level of abstraction of errors.
We therefore allowed the teams to do whatever they felt was useful and
asked them to explain the process they used on questionnaire 1.3 (see
Data Collection and Questionnaires, below).
Handouts
None.
Experimental Artifacts
Subjects used the same requirements document that was assigned in
session 1A.
Data Collection and Questionnaires
- Form 1.3 was provided as a template for submitting the fault list
for the team. There was also a questionnaire which we asked to be
submitted for each team member.
Web Accessibility