CMSC 838G - Fall 2005
Prof Guimbretière

Advanced Introduction to HCI

Method and design (Due 10/20/05)

The goal of the second part of your project assignment is for your group to describe your proposed experimental design. You will be asked to write 2-3 pages (in 2 columns format) describing your design in detail as well as the results you are expecting. Please explain all your claims.

What to do

Establish the goal of your experiment

The first step will be to refine the goal of your experiment. For that part, you will need to describe which question(s) your experiment is designed to answer. There could be several questions but in general one or two is a good starting point. Of course some questions might also have sub-questions.

Once one has established the question(s) to be answered, the next step is to establish the different variables you will consider in your experiment. For this assignment, you are asked to establish the list of the following type of variables you think you need to consider to conduct your experiment:

• Independent variables,

• Controlled variables,

• Possible randomized variables,

• Possible confounding variables,

• Dependent variables.

For each, you will need to describe this variable (textually) and why you think it is an important part of your experiment (in particular with respect to the question(s) your experiment is trying to answer). Furthermore, you will need to provide an operational definition for independent, controlled and dependent variables. This definition should include a clear description of the steps necessary to measure each variable. The level of the description should be such, that a skilled graduate student should be able to perform the measurement.

Design

For this section you are asked to present your experimental design including:

• How you will select your participant (and why),

• Which general style of experimental design you will follow. For this class a wide variety of design can be considered including qualitative (surveys, interviews...), quantitative (factorial, converging series), and non-experimental design (quasi-experiment, baseline), with a clear explanation of why you think this design is optimal,

• A precise description of the experimental design you picked. As the details will depend on each design, we are using a quantitative factorial design as an example: in that case the description should include the description of the factorial design, including which factors will be within and which factor will be between subjects. It should also include for each factor, levels that will be considered in the experiment and the order in which the different levels will be presented. Explain and justify all your answers. 

Protocol

The last step before running an experiment is to consider how you will run each participant. This is important so that you could understand the practicality of the design, as well as gather all the information necessary to submit an application to the IRB. For this project you are required to design an experiment which will take less than one hour for each participant including training, pauses and debriefing. Your description you include:

• You application to the IRB,

• A description of what each participant will have to do,

• The expected participation time.

Expected results

As a last check of the validity of your design, it is often useful to consider the most probable outcome of the experiment, and check if the design could provide the data necessary. For this reason, you are asked to provide a sketch of your final results section describing the expected outcome of the experiment, now you will present your results (graphs, tables...), and what the results will be in the best case senario.

Important remarks

As it is clear from the description above, designing and experiment requires making a large number of decisions which one will need to justified in the experimental report. These justifications might have several distinctive sources such as: previous literature, small pilot studies, users observations and so on. As a result, it is important for you to document clearly how you came to a given decision as you proceed with your design.

Deliverables

Your deliverable will have two parts:

• Your complete IRB application (see IRB application)

• A 2-3 pages (two columns conference format) description of your design and method. You might wish to append this part to your existing project report.

Presentation

Two to three group will be asked to present their work in class on the due date. Presentation should be 10 minutes long and will be followed by 10 minutes of questions/answer. It should include:

• A brief introduction of the topic,

• A presentation of how your project will contribute to the field (please includes key references),

• A description of your design and your rational for it,

• A conclusion.

While the presentation should focus on the substance rather than the form, a reasonable presentation style is expected.