Pilot Results (Due 11/17/04)
The goal of the third part of your project assignment is
for your group to tune your experimental design through pilot studies.
You will be asked to write 3-4 pages (in 2 columns format) describing your
final experimental protocol, your preliminary results as well as a
discussion about how you improved your experimental design in view of
your preliminary results.
What to do
Build and test your final apparatus
If you have not done so, you should complete the construction of your apparatus. As with any apparatus, it is important to test your apparatus thoroughly before going forward. The following aspects are of particular importance:
-
Behavior. Does the apparatus behave as expected? For example is the implementation of the technique under test accurate?
-
Protocol. Does your apparatus accurately support the protocol you wish to follow? For example, will the stimuli be presented in the right order?
-
Reliability. Participant are often confused by new techniques and might not follow the instruction exactly. Accordingly, it is important that the apparatus be robust to unexpected participant interactions (within reason of course). In particular the system should be stable enough so that the chance of a failure during the participant session is very low.
-
Accuracy. Does the apparatus accurately measure the dependent variable? For example does the system provide enough resolution for the experiment? Is the discriminant validity adequate?
Run pilot experiments
Running pilot participants is important as it will help
you find out problems in your apparatus as well as in your experimental
design. Accordingly while running pilots it is important:
-
To observe and take notes of unexpected events that
might occur,
-
To check after each pilot that your apparatus behaved as
expected and that all key pieces of information were properly logged,
-
To check after each pilot that the expected behavior was
observed. For example, if one is testing a new menu selection technique,
it is often useful to check visually the cursor paths; or if one is
comparing two techniques, it is often useful to compare the relative
speed between the different techniques.
If a discrepancy between the expected and measured result is
observed, then one of the following might apply:
-
The apparatus is at fault and does not measure the
dependent variables accurately. In that case the apparatus should be
fixed before running the next pilot.
-
The apparatus is working properly, but the experimental
design is introducing a confounding variables. For example the protocol
does not provide adequate training or does not counter-balanced
conditions correctly. In that case a new experimental design must be
designed before running the next pilot.
-
The discrepancy might be simply due to participant to
participant variability. Comparing the behavior of different
participants as well as the general statistical trend will help
identifying this case. Even in this case, it is very important to
identify the reasons for such a discrepancy, as they might be useful to
improve the protocol and or the technique in future implementation.
-
The experimental hypothesis is wrong. In that case, a
new hypothesis should be formulated and a new experiment designed
before running new pilots. In general several pilots should be run
before reaching that conclusion.
It is often the case that a substantial among of analysis
will be needed to put apart the different cases above. Accordingly it is
very important to reserve large amount of time between pilots. This is
especially true early on during piloting, as one has to get used to the data
analysis routine and changes from one pilots to the next might be important.
Finally it is very important to test protocol from beginning to end each
time it is modified.
For this part, you are asked to run a minimum of 6
pilots, with at least 3 pilots being tested in near experimental condition
(i.e. there was not change in the experimental protocol/apparatus for the
last 3 participants, and the observed results are similar to the expected
results.
Deliverables
Your deliverable will have three parts:
-
A description of your final experimental protocol,
-
A description of your results (without statistical
analysis),
-
A discussion about problems you observed during you pilot
study and how you addressed them.
Presentation
Two to three group will be asked to present their work in
class on the due date. Presentation should be 15 minutes long and will be
followed by 10 minutes of questions/answer. It should include:
-
A brief introduction of the topic,
-
A presentation of how your project will contribute to
the field (please includes key references),
-
A description of your design and your rational for it,
-
A presentation of the preliminary finding,
-
A conclusion.
While the presentation should focus on the substance rather
than the form, a reasonable presentation style is expected.
|