This assignment is a hands-on exercise on qualitative evaluation. Its immediate purpose is to give you experience conducting a usability study. Methods used in this study include strict observation, think-aloud, constructive interaction, questionnaires, and interviews.
Please keep in mind that usabliity has nothing to do with the quality of information available on the website. Many websites have a great deal of useful information, but the sites themselves are hard to use. Your focus here is the INTERFACE - the way people interact with the system. Your report should not discuss the quality of content or information. It should be dedicated to whether people can actually complete the required tasks.Your job. You will evaluate a website's interface Your deliverable will be a report , describing how you went about looking for design problems, what problems you saw, and what changes you recommend.
The website for you to analyze: any travel site that is sufficiently complex.
You must evaluate a total of 3 subjects. You can use anyone - roommates, significant others, classmates, parents. Anyone is fine.
Usability studies requires an observer to watch someone go through the paces with 'typical' tasks. It is your job as experimenter to prepare a set of example tasks ahead of time that the subjects will try to perform. These tasks should be realistic ones that typical users would try to do with the system! But how do you discover what those typical tasks are?
The first way is to let subjects use their own real tasks. To do this, you would have to solicit subjects who have a real need, and ask them if you could watch them do their tasks. This is only an option for you if the system being studied is a popular one.
The second way is to ask a random sample of people who are using the system what they typically do with it, and then generalize those as tasks to give to subjects. Again, this is only an option if you can find real users!
The third way is for you to use the system, and contrive a few sample tasks through intuition. Although this will not produce a set of reliable tasks, you may not have any other choice. (By the way, jot down any problems with the system you see as you try it. You can compare these later with the problems you notice in the actual study). This is probably the way most of you will use, since you have to prepare a list of tasks ahead of time. You can't just tell your subjects to use the site - if you want to use that method of understanding tasks, you would need separate steps - one group of subjects to gather tasks from, and a separate group of subjects to run the experiment on. Basically, the quickest way here is to just come up with tasks yourself
You should pick a minimum of 5-7 tasks. To get you started, I show here some sample tasks that you might use if you were evaluating the OSHA website. You will have to come up with tasks specific to the website you evaluate.
Sample Tasks for Occupational Safety and Health Administration (OSHA) - http://www.osha.gov
Where is the OSHA area office nearest you?
What phone number should you call if you have an emergency and need to report a fatality or imminent life-threatening situation?
How much is a subscription to Job Safety and Health Quarterly magazine?
Where is OSHA offering classes in hazardous materials this year?
Who serves on the Maritime Advisory Committee for Occupational Safety and Health?
If you are requesting information related to OSHA inspections or activities which occurred in Washington, DC, where would you submit your Freedom of Information Act (FOIA) request?
If you call your employer's attention to working conditions that you believe are unsafe or unhealthful, and your employer does not correct the hazard or disagrees with you, what can you do?
How many OSHA enforcement inspections were conducted in Pepsi-Cola Company facilities this year?
What does the Standard Industrial Classification (SIC) code 1611 stand for?
As an employer in Washington, DC, who can you contact to obtain a free safety consultation at your workplace or office?
Create a short pre-test questionnaire (~10 questions) testing the subject's experiences and beliefs about the system. It is extremely important that you ask relevant questions that helps you understand a subject's background and beliefs, as related to the task and system. Each subject should at least indicate their prior experience with computers, the windowing system, and the system being tested (why is this important?). They should also indicate their expectations. Example experience levels include:
while beliefs may be:
Similarly, create a post-test questionnaire of about the same length. Good questions will give you information about how participants judge the system's usability, where they think they had most problems, and so on. You may want to leave space after each question for comments, where you would encourage people to say why they answered a question a certain way. For example, here is such a question that uses a rating scale:
I found the system:
easy to use 1 2 3 4 5 hard to
use.
Reason for your rating::_____________________
You should go through each of the following four steps with each of your subjects. You will use condition 1 for one subject, condition 2 for one subject, and condition 3 for one subject. You are welcome to use more subjects, but are not required to use more than 3.
Take the pre-test questionnaire you had created previously and have subjects fill it out after they do the task.
In the first condition, the observer and subjects are not allowed to speak to each other. The subjects should carry out the tasks on the system (remember, tasks were prepared ahead of time!), with the observer taking notes of the subject's behavior and where the system appears to break down (e.g., errors, problems, etc.).
Note: It is sometimes difficult for the observer to figure out what the subject is doing.
This condition is similar to the above, except that the subjects are asked to say what they are doing as they are doing it, and they should elaborate on any problems they are having. For example, here is what a subject may say:
"I'm going to try to do this task ... OK, this is probably the menu item I should select. Hmmm ... It's not doing anything, what's wrong? Oh, I see, I have to double click it...
As before, the observer must take notes of the subject's behavior and key comments (professional usability people often use a tape recorder or video setup as well). While the observer is allowed to encourage the subjects to talk freely (i.e. "What is it you are doing now? Why did you do that?) the observer should not interfere or help the subject in any way, no matter how tempting!
Note: Talking aloud is sometimes uncomfortable and unnatural for people to do. It may also interfere with the task the person is trying to accomplish.
Caveat: If subjects get stuck. While the experimenter should not help the subject with the task, there are a few exceptions to this rule.
… If a subject has problems getting started, record the problems and give them a hint to get going. This is OK, because if they can't get started, they will not be able to do the tasks!
… If a subject cannot complete a particular task after a reasonable amount of time, tell them to stop and start them on the next task. Or, give them a hint if they cannot overcome some conceptual problem necessary to trying out other parts of the system. Again, record all problems.
Getting stuck is discouraging for subjects. Try to give them an early success experience, and remind them that they can quit at any time for any reason if they wish.
This condition involves subjects working together on a new task, with the observer taking notes as before. The difference is that the natural communication between the two subjects will replace the unnatural talking aloud in condition 2. Also, the differences between subject's knowledge may lead to interesting questions, explorations, and answers between them. The best match of subjects is a semi-knowledgeable person matched with a fairly new user, with the later being in charge of interacting with the system. Thus you hear the new user asking questions, and the knowledgeable one explaining how to do things (sometimes incorrectly!).
The best way to do this condition is using two subjects together. If you can get two people together at once, do condition 1 or 2 on the first subject, and then have that subject serve as the helper in this condition. However, if you can't get two people together at once, then you could serve as the constructive interaction helper, pointing things out and offering suggestions when the user needs help. Ask why they did things, what they're looking for, etc. You could get some good insight.Take the post-test questionnaire you had created previously and have subjects fill it out after they do the task.
The observer should then interview the subjects about their beliefs on how they performed, where errors were made, where the system helped them, where the system was weak, etc. As before, the observer should be taking detailed notes. Use the things you saw in the previous conditions to guide your interview. You can also use the filled-in questionnaire as a discussion tool (i.e. why did you answer this way?).
Your write-up should be oriented towards a senior person in the website's company that will make the major decisions on the software changes.
Section 1. Scenario
Give a brief description of what the system is, and then explain the role of your product evaluation team. Make sure you tell me the point of your work!
Section 2. Methodology
Explain what you did. Assume that the reader knows what the particular usability methods are (as described in this sheet) and their purpose. Include the number of subjects, the pre-test evaluation, task description, etc. You must provide a list of the tasks that you have developed, and why you included them. You must also provide the pre- and post- test questionnaires, and why you included each question.
Section 3: Observations
Summarize your observations. Where appropriate, use selected raw and collapsed data, paraphrasing, comments, questionnaire and interview results, etc. It is important to present as much information as possible with economy! This section should be at least 2 single spaced pages
Section 4: Interpretation: System strengths and weaknesses
Identify common and important problems and strengths of the system. This should be more than a checklist of all the problems seen. Try to generalize problems when necessary, although you can use examples to highlight them.
Section 5: Suggested improvements
Describe five important changes that you would make to the design of the system, with explanation. Refer back to your observations and the discussion on design as covered in class. Note that you must stay within the style of interface presented: for example, your modification cannot turn (say) a form fill-in system into a graphical map.
Section 6: Conclusion
Summarize what you found and the recommendations.
Appendix 1: Comparison of different techniques
For future usability studies, you want to tell your product team what worked well and what didn't in this usability study. Briefly summarize your experiences with each method, contrasting them for ease of use, the richness of the information obtained, their advantages, etc. Then recommend the methods you wish your group to use in the future. Which was most useful? Which was least useful? What would you keep? What would you throw away? This discussion should be about one type-written page. It should NOT include a description of what each method - we know that already. This should be purely analysis.
Pay close attention to the required sections - you will be graded on having the required elements for each section. The quality of your analysis is very important, so think critically about your conclusions. You will also be graded on writing; as graduate students, you should have no errors in spelling and grammar, and the paper should be well organized and presented.