Project phase #4
(due 12/04/03)
Evaluation
Overview
Now that you have a running prototype, it is time
to have target users evaluate it for real. During phase #4, you
will design and conduct a qualitative evaluation of the system
you built during phase #3.
What to do
To conduct your qualitative evaluation you should:
-
Starting from your initial list of goals,
identify at least 3 major tasks you wish to evaluate,
-
Design a protocol for your qualitative
evaluation. Your protocol should describe how you will
evaluate the 3 major tasks you picked including the testing
technique used, where the test will take place, the testing
configuration, the initial state of your system and the final
state to be reached by participants. Your protocol must at least
include a direct observation evaluation, a post-observation
survey and debriefing,
-
Run the protocol for at least 6 participants.
For best results, you must be careful on how to pick your
participants: as before, it is important to pick participants
who are representative of one or more personas and to have at
least 4 participants per persona. Don't forget to have each
participant sign the consent form before
starting the experiment.
To avoid wasting your participants time (and yours) on fixing the
protocol during the experiment, it is very important to debug
your protocol on members of your group before running it with
participants. Finally, for your results to be useful you must
not influence participants during your evaluation but only take notes.
In order preserve the
privacy of the interviewees you should not gather private
information such as name, ID and the like.
-
Analyze the information gathered during the
qualitative evaluation, and identify which part of your design
should be changed.
Deliverables
Your group will deliver a portfolio written to
the imaginary VP of product development in your company. The
portfolio will include the following sections.
Section 1 (40 points): Protocol
-
A detailed explanation of the design rational
behind your protocol (~ 2.5 pages), including:
-
The overall goals of your study,
-
Which testing technique you picked and
why,
-
How you selected the tasks to be
performed and how they are related to the study goals,
-
How you selected your users and why,
-
Your consent form,
-
Your experimental setting including where
the test took place, how it was performed, the initial
state of your interface and the expected final state,
-
The questionnaire you used for the
debriefing and how each question relates to the goals of
your study.
Section 2 (40 points): Observations
-
For each participant in your experiment
present a summary of the testing session (1 page per
participant), including:
-
A summary of your group observations,
-
The participant completed consent form
and survey,
-
A summary of the debriefing sessions.
-
The key findings of your evaluation, and for
each of them, how you reached this conclusion from the data
you collected (1 page).
Section 3 (20 points): New prototype
-
Pick two major problems and propose a
solution to address these problems. Your solution should be
compatible with the rest of the design. It might be useful
to use a Before/After structure so that changes you made are
easy to understand. For each change, do not forget to
explain why your changes will address the problem at hand
(~2 pages excluding figures).
Section 4 (20
points, 828S students only): Experimental design
For this project you are asked to run the
experiment you proposed during phase #3 and produce a report of
your results:
-
Provide a copy of your consent form,
-
Present possible discrepancies between your
proposed design and the design actually used. In particular,
state how many participants you ran (the same participants can
be used for both studies),
-
Present your results. Explain how these
results agree (or not) with your predictions. Explain possible
discrepancies.
-
Present and explain other insights you gained
during the experiment.
For this section it might be useful to look at
the book called "Doing Psychology Experiments" by D. Martin,
This book is placed on reserve at the CS library.
Notes (for 434 and 828S)
The portfolio is intended to document the
progression of your design, which includes your final project.
Your portfolio must be neat, well-organized, and visually
appealing. Portfolios should be constructed out of a 1" or smaller
3-ring binder (we will not appreciate having to carry around
larger binders). Your portfolio should also use titled section
separators to separate the major sections. The cover of the portfolio
should include the names of the group members, the group number,
and the title of the project. The first page should be a table of
contents, which will grow over time.
Each group should be prepared to give a 10
minutes presentation of their portfolio during the class when the
project is due. The presentation could be made either using
PowerPoint slides or by using material provided in the portfolio.
Grading will be based upon the
sophistication, logic and maturity of the presentation, and the
completeness of the work.
|