Discussion of guidelines for user observation

From
User Observation: Guidelines for Apple Developers, by Kathleen Gomoll & Anne Nicol, January 1990

Introduction
User testing covers a wide range of activities designed to obtain information on the interactions between users and computers. Most user testing requires considerable expertise in research methods, as well as skill in using complex data collection tools. For example, user testing techniques include: interviews, focus groups, surveys, timed performance tests, keystroke protocols, and controlled laboratory experiments. Of the many user testing techniques available, user observation is one technique that can be used by anyone with a concern for including the user in the product development process.

User observation involves watching and listening carefully to users as they work with a product. Although it is possible to collect far more elaborate data, observing users is a quick way to obtain an objective view of a product.

When to observe users
User observation should be an integral part of the design process---from the initial concept to the product's release. Software design that includes user observation is an iterative process; user feedback provides the data for making design modifications. The iterative process assumes that preliminary human interface designs should exist prior to the development of underlying code. Interface designs should be tested frequently to determine which design should be implemented. Then, as the code develops, the entire product should be tested and revised several times.

Preparing for a user observation
Set an objective: Before you do any testing, you should take time to figure out what you're testing and what you're not. In other words, determine an objective for your test that focuses on a specific aspect of the product. By limiting the scope of the test, you're more likely to get information that helps you solve a specific problem.

Design the tasks: Your test participant will work through one or more specific tasks. These tasks should be real tasks that you expect most users will do when they use your product. The entire user observation should not run over an hour, so you should design tasks that focus on the part of the product you're studying. For example, if you want to know whether your menus are useful, you could design a task that requires the participant to access the menus frequently. After you determine which tasks to use, write them out as short, simple instructions.

Important: Your instructions must be clear and complete, but they should not explain how to do things you're trying to test. For example, if you want to find out whether users can navigate through your program easily, don't give them instructions for navigation. Or, if you want to know whether your interface is self-explanatory, don't describe how it works. This concept is extremely important to remember. If you teach your participants about something you're trying to test, your data will not be useful.

Decide upon the use of videotape: Although you can observe users effectively without using special recording equipment, you may want to use videotape to capture the entire session. By videotaping the session, you collect an enormous amount of valuable information that you can review and analyze after the test is over. If video equipment is not available, a tape recorder can be helpful for recording what is said during the test.

Determine the setting: The ideal setting for user observation is a quiet, enclosed room with a desk, the appropriate hardware and software, a video camera, and two microphones (one for you and one for the participant). Of course, you may not have all these things available when you need to observe; therefore, you should try to approximate the ideal setting as closely as you can. If you have to conduct the observation in a regular office, ask the people around you to keep the noise level down during the observation. The key is to make the environment as interruption-free as possible. Get the participants out of their offices, away from phone calls and people who might drop by.

Find representative users: When looking for participants, try to find people who have the same experience level as the typical user for your product. Don't ask people you work with regularly to be participants because they are probably familiar with your product or your opinions about the product. Generally, you should look for people who are familiar with the hardware you use but are not familiar with your product. You may want to ask pairs of people to work together on your tasks. You'll find that people working in pairs usually talk more than people working alone, and they also tend to discuss features of the product and explain things to each other.

10 steps for conducting a user observation
The following instructions guide you through a simple user observation. Remember, this test is not designed as an experiment, so you will not get statistical results. You can, however, see where people have difficulty using your product, and you can use that information to improve it. These instructions are organized into steps. Under most of the steps, there is some explanatory text and a bulleted list. The bulleted list contains sample statements that you can read to the participant. (Feel free to modify the statements to suit your product and the situation.)

Introduce yourself.
Describe the purpose of the observation (in general terms). Set the participant at ease by stressing that you're trying to find problems in the product. For example, you could say:
- You're helping us by trying out this product in its early stages.
- We're looking for places where the product may be difficult to use.
- If you have trouble with some of the tasks, it's the product's fault, not yours. Don't feel bad; that's exactly what we're looking for.
- If we can locate the trouble spots, then we can go back and improve the product.
- Remember, we're testing the product, not you.
Tell the participant that it's okay to quit at any time. Never leave this step out. Make sure you inform participants that they can quit at any time if they find themselves becoming uncomfortable. Participants shouldn't feel like they're locked into completing tasks. Say something like this:
- Although I don't know of any reason for this to happen, if you should become uncomfortable or find this test objectionable in any way, you are free to quit at any time.
Talk about the equipment in the room. Explain the purpose of each piece of equipment (hardware, software, video camera, microphones, etc.) and how it is used in the test.
Explain how to think aloud. Ask participants to think aloud during the observation, saying what comes to mind as they work. By listening to participants think and plan, you can examine their expectations for your product, as well as their intentions and their problem solving strategies. You'll find that listening to users as they work provides you with an enormous amount of useful information that you can get no other way. Unfortunately, most people feel awkward or self-conscious about thinking aloud. Explain why you want participants to think aloud, and demonstrate how to do it. For example, you could say:
- We have found that we get a great deal of information from these informal tests if we ask people to think aloud as they work through the exercises.
- It may be a bit awkward at first, but it's really very easy once you get used to it.
- All you do is speak your thoughts as you work.
- If you forget to think aloud, I'll remind you to keep talking.
- Would you like me to demonstrate?
Explain that you cannot provide help. It is very important that you allow participants to work with your product without any interference or extra help. This is the best way to see how people really interact with the product. For example, if you see a participant begin to have difficulty and you immediately provide an answer, you lose the most valuable information you can gain from user observations where users have trouble, and how they figure out what to do.

Of course, there may be situations where you have to step in and provide assistance, but you should decide what those situations might be before you begin testing. For example, you may decide that you can allow someone to flounder for at least 3 minutes before you provide assistance. Or, you may decide that there is a distinct set of problems with which you can provide help.

As a rule of thumb, try not to give your test participants any more information than the true users of your product will have. Following are some things you can say to the participant:
- As you're working through the exercises, I won't be able to provide help or answer questions. This is because we want to create the most realistic situation possible.
- Even though I won't be able to answer your questions, please ask them anyway. It's very important that I capture all your questions and comments on tape.
- When you've finished all the exercises, I'll answer any questions you still have.
Describe the tasks and introduce the product. Explain what the participant should do and in what order. Give the participant written instructions for the tasks.

Important: If you need to demonstrate your product before the user observation begins, be sure you don't demonstrate something you're trying to test. (For example, if you want to know whether users can figure out how to use certain tools, don't show them how to use the tools before the test.)
Ask if there are any questions before you start; then begin the observation.
Conclude the observation. When the test is over:
- explain what you were trying to find out during the test.
- answer any remaining questions the participant may have.
- discuss any interesting behaviors you would like the participant to explain.
Use the results. As you observe, you see users doing things you never expect them to do. When you see participants making mistakes, your first instinct may be to blame the mistakes on the participant's inexperience or lack of intelligence. This is the wrong focus. The purpose of observing users is to see what parts of your product might be difficult or ineffective. Therefore, if you see a participant struggling or making mistakes, you should attribute the difficulties to faulty product design, not to the participant.

To get the most out of your test results, review all your data carefully and thoroughly (notes, the video tape or cassette tape, the tasks, etc.).

Look for places where participants had trouble, and see if you can determine how your product could be changed to alleviate the problems.

Look for patterns in the participants' behavior that might tell you whether the product was understood correctly.

It's a good idea to keep a record of what you found during the test. That way, you have documentation to support your design decisions and you can see trends in users' behavior. After you've examined the results and summarized the important findings, fix the problems you found and test the product again. By testing your product more than once, you can see how your changes affect users' performance.
Web Accessibility