THE USABILITY TEST


Subjects were invited to the Beltsville Community Center where is a bimonthly meeting of 100-200 seniors. The test took place in a computer room with ~10 terminals. For the test, we provided a separate laptop to avoid installation issues. Unfortunately, the laptop did not have a regular mouse port, and the center did not have USB mice, so users were forced to use the touchpad on the laptop for the test. They were given instructions on how to use the touchpad, were told that the real application might be on a terminal or touchscreen, and were asked to ignore difficulties with the touchpad while evaluating the software.

The tests ran about 20 minutes each, in a one-on-one setting, with a PhotoGenic team member on hand to provide assistance if the user ran out of ideas for how to proceed. Users were encouraged to "think out loud," so that observations could be made about how they were interacting with the software. Since the subjects were not accustomed to usability tests, it was emphasized that this is a work in progress and that if something doesn't work as they expect, it is a flaw in the software, not in them. Even with that preparation, subjects still seemed reluctant to say that they could not figure out how to do something, because of an apparent perception that if they can't do it, they must be at fault.


The Tasks
Subjects were asked to complete the following tasks:

Placing a Phone Call
1. Go to the area to place a phone call.
2. Call Maggie at work. (During this test, the program will not actually place a call.)
3. Call Tonya Anderson at work. (During this test, the program will not actually place a call.)
4. Call Betty on the cell.

Searching for a Person
1. When was the last time you got an email from your daughter Lily?
2. When was the last time you sent an email to your friend Nikki?
3. How many times have you called Wayneson in the last month?
4. How many times has Wilma called you in the last month?
5. Select Cameroon to find outgoing phone calls to him. Cancel before the search is started.

Look at Pictures
1. Look through all of the pictures stored on the computer and find a picture of your niece Hilary.
2. Compare the Thumbnail layout vs. the Full size layout vs. the List of names on the left side. Which one is the easiest to look through? If you had a choice of only one selection method out of the three given, which one would you prefer and why?

Send an Email Card
1. Send your niece Hilary a card wishing her congratulations on his recent high-school graduation. (During this test, no email card will actually be sent)
2. Find your friend Tim's picture and send him a Thanksgiving note with a personalized message from yourself. (During this test, no email card will actually be sent)


The Subjects
A list of 11 seniors was compiled on a preliminary visit. When contacted, however, only four were actually available for the test. Of those four, only three came on the day of the study.

Two of the subjects were male, and one was female, and each received a questionnaire for demographic information. All three used a computer and email at least several times a week. One was aged 40-49, one was 50-59, and one was 60-69. When asked they or someone they know had the experience of seeing a picture of a close family member and not being able to recall their name, one chose "rarely" (the youngest one) and two chose "occasionally".

The subjects looked at photo albums ranging from a few times a month to almost never. All three buy hallmark cards, and two of three write a separate message on the card.


User Comments

Subjects were asked to fill out another questionnaire after the test. On a scale from 1-9, subjects marked their impression of the program and ease of use the same, as 6, 8, and 9. Scores for ease of understanding what different parts of the program (buttons, menus, selection boxes, etc.) were lower (7, 6, and 4 respectively), and layout of the program screens scored 4, 6, and 8. While there were only three subjects and their opinions were somewhat distributed, it is fair to say that they seemed to have some problem understanding what parts of the program did, and one had a problem with the layout of the program screens.

When asked what they liked about the program, subjects cited ease of moving between parts of the program, and the association of pictures with emails and calls. When asked about areas of the program that need work, subjects mentioned the adding ability to set a larger font size, larger picture size, and the order of operations for the Send an Email Card feature.

Subject suggested more categories for the Send an Email Card feature, selection of photos by letter to jump to a name, selection of a location by clicking an icon (cell phone, for example) in addition to clicking the radio button underneath, and having a picture associated with an incoming email.

Other comments included: "Great idea", and "This program would be nice to have for use at home or even at work. It would make life a little more simple for people that do not use a PC or laptop too often." One user reacted with "Oh, I like this" when the graphics for the email card came up on the screen.


Observations and Problems
One lesson we learned about working with seniors is that contact must be maintained. When the eleven subjects were not contacted for six weeks during development of the program, many forgot what they had volunteered for, or had lost enthusiasm that they had after the initial presentation. If a weekly progress email had gone out, they might have felt more involved and more inclined to come back for the actual study.

In general, the subjects were interested in the software, and thought that the idea was a good one. They seemed to understand the tasks, and they related to the tasks as things that they do on a regular basis anyway.

Two of the three users never clicked the help button, and the third only did it out of curiosity after filling out the post-test questionnaire. This might have been because a PhotoGenic representative was on hand for the test and so they did not feel the need. None of the three spent time on the welcome screen, but they were also not expressly asked to, and they were given verbal instructions instead.


Overall
The look and feel is inconsistent between the first two features and the last two. Specifically, the last two have an icon in the upper left hand corner followed by bold text with the current feature, and the first two do not. Importance 5, Effort 1.

Names were not alphabetically listed. Importance 5, Effort 1.

Subjects clicked the button on the left side of the screen in attempts to "reset" the view if they felt disoriented, and this did not work. Importance 4, Effort 2.


Look At Pictures
Clicking an item in the list did not select a person, which was complicated by the instruction: Select a person from the list or by clicking on their picture. In fact, selecting a person from the list did NOT select them until the user clicked "Select Person." This also led to confusion on the Send an Email Card screen. Importance 5, Effort 1.

It was not clear who the four people in the thumbnail view were (in the fullsize layout, the selection matched the currently visible person). Importance 4, Effort 1.

It was not clear if the caption on the button under the pictures was a description of the current state, or an explanation of the state that would be entered by clicking the button. Importance 4, Effort 2.

There was no integration between Look at Pictures and Send an Email Card, so if you wanted to send a card to someone whose picture you were looking at, you had to change modes using the buttons on the left. Importance 4, Effort 4.

One subject wanted names to be sorted by both last name and first name. Importance 4, Effort 3.

Subjects expressed a preference for seeing one picture at a time, but appreciated having both views to choose from, since sometimes the name is hard to remember, and sometimes the face is hard to remember.


Send an Email Card
The tabs on Send an Email Card were hard to find, and subjects clicked the tabs out of order if they found them at all. It was not easy to see which tab was currently selected. One user tried clicking an occasion, then double clicking it, and then looked perplexed at the screen. Importance 5, Effort 3.

Changing tabs before required information was entered showed the new tab with an error message, and then went back to the previous one, which was disorienting. Importance 4, Effort 1.

There were spelling errors in text of hallmark cards. Importance 5, Effort 1.

Suggested other categories: Congratulations, Thinking of You, Sympathy, Thank You, Invitations. Importance 3, Effort 3.


Make a Phone Call
All subjects first chose a place to call, which is located horizontally from the Make a Phone Call button (i.e. their eyes scanned across, rather than starting at the top of the screen). Subjects chose a place to call and then got an error when they clicked Call, at which time they started looking for how to choose a person. Even then it sometimes took a long time for them to figure it out. Importance 5, Effort 3.

All subjects tried to click the pictures of Home Work, and Cell rather than the radio buttons. Importance 5, Effort 1.

One subject was very frustrated by the lack of a "Back" button. The subject missed the desired name and had to cycle through the entire list to find it, only to miss it again! Importance 5, Effort 1.


View Diary
It was not immediately obvious what feature this button enabled, although subjects understood quickly once they clicked on it. Importance 3, Effort 2.

Subjects did not seem to understand what Cancel meant in the context of search. Perhaps "Clear" was what was intended? Importance 4, Effort 2.

One subject clicked the "Choose Person" to start from the beginning, which worked, but clicking next showed the next person from before the user clicked "Choose Person", i.e. where the subject had previously been in the list. Importance 5, Effort 1.


Response to Feedback

The following changes were incorporated into the different components as a result of the student suggestions and usability test results.

Framework
- A background color was added to the currently active component.


Figure 16. Revised Framework


Look at Pictures
- "Look and Feel" made a little more consistent to the other components by adding a graphic and title to the Browse section. (similar to "Phone Call" and "Diary")
- Names are alphabetically sorted on the basis of the first name.
- Clicking on a person in the list now selects them. (Earlier only double click worked but now both single and double click work)
- Button under the picture now states "Switch to Fullsize Layout" and "Switch to Thumbnail Layout" as opposed to merely "Fullsize Layout" or "Thumbnail Layout" to avoid confusion.
- List names are now displayed with a slightly larger size font for easier viewing.


Figure 17. Revised Browsing Screen


Send an Email Card
- Tabs for "Occasion" and "Send To" can be clicked in any order now.
- When clicking the "Message" tab, the user is merely prompted to select recipient and/or occasion and not taken to the specified tab by default in order to avoid disorienting the user.
- Spelling mistakes in the text for the E-Cards have been corrected.

Make a Call & View Diary
- Removed the choose person button, and introduced a new design with a list of names and a picture box beside it. The user selects a name from the list and the picture box changes to show the person along with the name.
- The pictures of locations were changed so that the associated radio button is selected when the picture is clicked on.
- In the "View Diary" section, the name of the category can now be clicked on to select that option. The "Cancel" button has been renamed to "Clear".


Figure 18. Revised Phone Call Screen



Figure 19. Revised Diary Screen