Project 3: A Usability Study and Heuristic Evaluation

Due: In class on Monday, May 2nd.
Note: I would strongly suggest you target your team to finish this by Friday April 29th... You should plan to have your tasks and user study planned out by April 20th, and run your subjects by April 25th if possible. Project 4 will be posted on April 25th.

Overview
This assignment is a hands-on exercise on qualitative evaluation. Its immediate purpose is to give you experience conducting both a usability study on a prototype as well as performing a heuristic evaluation on that prototype. The secondary purpose is to provide feedback to another group about their project 2. The methods used in the usability study include strict observation, think-aloud, constructive interaction, questionnaires, and interviews. For the heuristic evaluation, the members of your team will undertake the role of HCI experts brought in to review a prototype. Because of the economy of these methods, you are expected to be able to apply them in your actual work practices. Your group will deliver your report to the team whose project 2 you are using for your project 3. Your heuristic evaluation and usability study report will be used to determine your project 3 grade. It will not be used to alter the other team's project 2 grade.

Project 3 Team Assignments

Deliverables
Your group will deliver a substantial technical report written to the designers of the prototype you evaluate. It must include your observations, the findings, the major problems detected, and some design recommendations. It will have two distinct parts: the user-based study results and the heuristic evaluation results. Your report will briefly contrast the user study methods you used, recommending which ones should be adapted in future study.

0. Your Blog
As you are working on this project, you are to (at a minimum) document your plans and progress in the blog that has been set up for your team (each team will have its own blog that I create and invite you to join). You should also explore using the blog as a conversation space while you work on the project so that if you are unable to meet every day, you still have a place to communicate every day. I will have access to the blog and reserve the right to post questions to the blog - these questions should be answered by the team or a member of the team in a timely manner. At the end of the project I reserve the right to post the blog addresses to the class so that everyone in the class can explore the different ways in which the blogs were used and the projects advanced.

1. Background
How can we tell if a computer system is any good for a person to use? Many developers simply create the system, try it out themselves until they are satisfied with it, and then dump it onto the user audience. The result is usually a product that people have problems with.

One of the easiest methods for getting to "know the user" and for evaluating the human computer interface is through usability studies. Although these come in many flavors, they all require an observer to watch a typical user try the system out for a real task. It is surprising how many design flaws can be detected this way!

Usability studies are increasingly popular in industry. Many modern software companies now have usability labs staffed by HCI professionals whose job it is to find usability problems in products as they are being developed. Most labs contain all the equipment permanently in place (e.g., computers), and are instrumented with audio, video, screen-capture software, one-way mirrors, and so on.

Usability studies are extremely practical, and you can do them 'on the cheap' without these special usability labs. The simplest studies just require you: to pull up a chair next to a typical user; to watch them do their work (and perhaps having them explain what they are doing as they are doing it); to jot down any noteworthy events that occurred; and to listen to the user's comments.

Heuristic evaluations are also becoming popular and practical ways to review a prototype system to look for places for improvement.

Your job. Imagine that you work for Usability Inc., a consulting firm that specializes in evaluating interfaces. You and your team have been contracted to do a usability study of the system assigned to your group. You have also been asked to do a heuristic review of the system. Your deliverable will be a report written for the manager in charge of that system's design, use and redevelopment. You should assume that the manager is knowledgeable. Your report will describe how you went about looking for design problems, what problems you saw, and what changes you recommend. Depending upon how convincing you are, the manager will use your report to authorize changes in the upcoming version. You will do this assignment in groups, and will follow the major steps below.

2. Subject selection
You must perform each of the 3 conditions (Silent Observer, Think Aloud, Constructive Interaction) with 2 subjects (for a total of 6 "subjects"). Note: For the third conditions, it takes two individuals to make up a "subject". You may not use other students in the class as subjects. The subjects must be from outside the class, and preferably some should not be students at all if the tasks are not student-specific. Try to find family members, co-workers, or friends outside of schools (or at least outside of CS). It will not be possible to get actual users for various project options (please do not ask department staff to be participants in the user testing of things like the Visitor PDA even though they might be the target of the application). Be sure to inform the participants of the setting in which the program would be used. As always, treat participants with honesty and respect and let them know that they can stop at any time.

3. Things to prepare ahead of time: Selecting a core set of typical tasks
Usability studies requires an observer to watch someone go through the paces with 'typical' tasks. It is your job as experimenter to prepare a set of example tasks ahead of time that the subjects will try to perform. These tasks should be realistic ones that typical users would try to do with the system! But how do you discover what those typical tasks are? When choosing taks, make sure they are tasks that are meant to work in the prototype. It would be wise to confirm with the prototype's design team that you are testing something that is on their list of vertical implementations. Note: If you choose tasks for your study which are based upon parts of the prototype that is not yet implemented, your project 3 grade will suffer since you are not getting useful feedback about the parts that are meant to work. (This is why project 2 required the listing of several tasks that could be accomplished.) Please review the prototype that you obtain soon so that if there are issues with the prototype you have been asked to use in this project, we can resolve them quickly.

The first way is to let subjects use their own real tasks. To do this, you would have to solicit subjects who have a real need, and ask them if you could watch them do their tasks. This is only an option for you if the system being studied is a popular one.
The second way is to ask a random sample of people who are using the system what they typically do with it, and then generalize those as tasks to give to subjects. Again, this is only an option if you can find real users!
The third way is for you to use the system, and contrive a few sample tasks through intuition. Although this will not produce a set of reliable tasks, you may not have any other choice. (By the way, jot down any problems with the system you see as you try it. You can compare these later with the problems you notice in the actual study).
The fourth way is to have the design team tell you what tasks they have compiled. Note: Although this way is not really best (since it means that an oversight on the part of the original design team could be compounded), it might be necessary for your project since not all of the features are going to have been implemented yet.

4. Preparing Pre- and Post- Test Questionnaires
4.1 Pre-test questionnaire
Create a short pre-test questionnaire (~10 questions) testing the subject's experiences and beliefs about the system. It is extremely important that you ask relevant questions that helps you understand a subject's background and beliefs, as related to the task and system. Each subject should at least indicate their prior experience with computers, the windowing system, and the system being tested (why is this important?). They should also indicate their expectations.

Example experience levels include:

never used it,
used it once or twice over the last few years
used it ~3-7 times this year, but not regularly
use it regularly (how often?)

While beliefs may be:

will need personal instruction to get started
will learn it after a bit of playing around
will be able to do simple tasks with no problems
will be able to do complex tasks

4.2 Post-test questionnaire
Similarly, create a post-test questionnaire. Good questions will give you information about how participants judge the system's usability, where they think they had most problems, and so on. You may want to leave space after each question for comments, where you would encourage people to say why they answered a question a certain way. For example, here is such a question that uses a rating scale:

I found the system:
                     easy to use               hard to use
                               1   2   3   4   5 

                               Reason for your rating: _____________________

Among the things to consider might be the question of how many numbers to use on the scale (i.e.: 5, 7, 9, ???).

5. The usability study
See the web page on User Observations for a basic description of the method. You should go through each of the following steps. You will use condition 1 for 2 subjects, condition 2 for 2 subjects, and condition 3 for 2 subjects. You are welcome to use more subjects, but are not required to use more than 6.

Pre-test Questionnaire Take the pre-test questionnaire you had created previously and have subjects fill it out before they do the task.

Condition 1: The Silent Observer
In the first condition, the observer and Subject 1 are not allowed to speak to each other. The first subject should carry out the tasks on the system (remember, tasks were prepared ahead of time!), with the observer taking notes of the subject's behavior and where the system appears to break down (e.g., errors, problems, etc.).
Note: It is sometimes difficult for the observer to figure out what the subject is doing.

Condition 2: Think Aloud Method
This condition is similar to the above, except that Subject 2 is asked to say what they are doing as they are doing it, and they should elaborate on any problems they are having. For example, here is what a subject may say: "I'm going to try to do this task ... OK, this is probably the menu item I should select. Hmmm ... It's not doing anything, what's wrong? Oh, I see, I have to double click it...
As before, the observer must take notes of the subject's behavior and key comments (professional usability people often use a tape recorder or video setup as well). While the observer is allowed to encourage the subject to talk freely (i.e. "What is it you are doing now? Why did you do that?) the observer should not interfere or help the subject in any way, no matter how tempting!
Note: Thinking aloud is sometimes uncomfortable and unnatural for people to do. It may also interfere with the task the person is trying to accomplish.
Caveat: If subjects get stuck. While the experimenter should not help the subject with the task, there are a few exceptions to this rule.

If a subject has problems getting started, record the problems and give them a hint to get going. This is OK, because if they can't get started, they will not be able to do the tasks!

If a subject cannot complete a particular task after a reasonable amount of time, tell them to stop and start them on the next task. Or, give them a hint if they cannot overcome some conceptual problem necessary to trying out other parts of the system. Again, record all problems. Getting stuck is discouraging for subjects. Try to give them an early success experience, and remind them that they can quit at any time for any reason if they wish.

Condition 3: Constructive Interaction
This condition involves Subjects 3 and 4 working together on a new task, with the observer taking notes as before. The difference is that the natural communication between the two subjects will replace the unnatural thinking aloud in condition 2. Also, the differences between subject's knowledge may lead to interesting questions, explorations, and answers between them. The best match of subjects is a semi-knowledgeable person matched with a fairly new user, with the later being in charge of interacting with the system. Thus you hear the new user asking questions, and the knowledgeable one explaining how to do things (sometimes incorrectly!).

Post-test Questionnaire Take the post-test questionnaire you had created previously and have subjects fill it out after they do the task.

Interview
The observer should then interview the subjects about their beliefs on how they performed, where errors were made, where the system helped them, where the system was weak, etc. As before, the observer should be taking detailed notes. Use the things you saw in the previous conditions to guide your interview. You can also use the filled-in questionnaire as a discussion tool (i.e. why did you answer this way?).

6. The write up
Your write-up should be oriented towards a senior person in the company that commisioned the study and who will make the major decisions on the software changes. However, you should also expect the design team will be given your report.

Section 1. Scenario
Give a very brief reminder to the VP on what the system is, and then explain the role of your product evaluation team. Make sure you tell her the point of your work!

Section 2. User Testing Methodology
Explain what you did. Assume that the reader knows what the particular usability methods are (as described in this sheet) and their purpose. Include the number of subjects, the pre-test evaluation, task description, etc. You must provide a list of the tasks that you have developed, and why you included them. You must also provide the pre- and post- test questionnaires, and why you included each question.

Section 3: Observations
Summarize your observations. Where appropriate, use selected raw and collapsed data, paraphrasing, comments, questionnaire and interview results, etc. It is important to present as much information as possible with economy!

Section 4: Interpretation: System strengths and weaknesses
Identify common and important problems and strengths of the system. This should be more than a checklist of all the problems seen. Try to generalize problems when necessary, although you can use examples to highlight them.

Section 5: Suggested improvements
Describe five important changes that you would make to the design of the system based on user feedback and obserations, with explanation. Refer back to your observations and the discussion on design as covered in class.

Section 6: Heuristic Evaluation
Perform your heuristic evaluation as a team. This means doing your own evaluations and then meeting together to create a single report. Make sure you clearly list the problems detected, categorized by heuristics. Include a severity rating of the problems noted.

Section 7: Conclusion
Summarize what you found and the overall recommendations.

Appendix 1: Comparison of different techniques
For future usability studies, you want to tell your product team what worked well and what didn't in this usability study. Briefly summarize your experiences with each method, contrasting them for ease of use, the richness of the information obtained, their advantages, etc. Then recommend the methods you wish your group to use in the future. Which was most useful? Which was least useful? What would you keep? What would you throw away?

Appendix 2: Raw data
All original observations, etc. should be attached here.

Please remember that usability studies and heuristic evaluations can be immensely practical: you can and should use it every time you design (or wish to select!) a user interface.

Good luck, and have fun!

Web Accessibility