The Problem: Combining into Teams

According to the current design of the experiment, we end up with four groups of people for each document; design, test-, and use-based, and the group applying the usual NASA technique. Let us denote the number of subjects in each group as , , and respectively. Now, since these subjects did the tests on an individual basis, they need to be combined into ``artificial'' teams in front of the analysis.

When combining individual results into team results, the union of the defect coverage of the various team members is considered the team defect coverage, and the defect detection rate is computed accordingly. The problem is the assignment of subjects to teams. Initially, we considered two approaches:

• Combine the individuals in all possible ways forming one team where all members applied the ususal NASA techniqe, and another team where all members applied PBR with one member from each of the three perspectives. Both teams had three members. This will give us:

  * *

  PBR teams, and

  

  NASA teams. The problem with this solution is that each of these teams are not statistically independent because each individual data point will occur in various teams. Thus we can not do any reasonable statistical analysis of this data.

• The other solution is to make sure we have statistical independence between the teams we form. Thus, each individual data point can occur only once, and this will give us only

  ( + + )/3

  PBR teams, and

  ()/3

  NASA teams. Thus, unless we have a vast number of subjects in the experiment, this approach will give us too few data points for any further analysis to make sense. We also have a problem in how to combine the individuals into these teams since there are a large number of possibilities (as indicated in the first approach).

In order to find a solution to this problem, we contacted Chip Denman and Paul Smith from the Statistics Lab at the University of Maryland. They suggested that we applied a permutation test as explained below.

Web Accessibility