Since it would be prohibitively expensive to actually run the experiment with enough 3-person review teams to get statistically significant results, participants will review the documents on an individual basis only. The expected performance of teams using PBR will be studied during the analysis of the experiment by using statistical techniques to explore the defect coverage that could result from such teams.
We blocked the design on technique, perspective, document, and reviewer group. Since we could not ask participants to apply their usual technique after they had learned PBR (we assume that the learning of a systematic technique may bias future application of a non-systematic one) we chose to do all usual technique tests on the first day, and have the training in and application of PBR take place on the second day. Additionally, in order to minimize any learning effects (we assume that participants may not only need some practice to get used to applying PBR, but may also need practice in recording defects on our forms) we placed a small training session before each test. The whole experimental design is illustrated below, and shows the sequence of events for both of the groups of participants (each composed in turn of subjects applying each of the PBR perspectives):
We provide a page from which our experimental materials can be downloaded. This page also provides a list of how the documents will need to be arranged into packets for the tasks on both days.
The choice of experimental design is explained in more detail in our summary paper.