Two testers, Michael Yoo and Kevin Tom, conducted the
usability tests of the RISE interface.
Each tester worked with three test users, making six total test users. All testing was done with one tester and one
test user at a time. Test users did not
have any interaction with each other. Since
users would desire privacy during their use of the system, the usability tests
were done in a closed environment.
Testers minimized the number of distractions by closing windows and
doors, minimizing outside noise, and removing distracting materials from the
closed room. Testing of the interface
was performed on a desktop computer using the PDA emulator included for Visual
Basic in Microsoft Visual Studio .NET 2003.
All of the usability tests were conducted during the first weekend in
It was emphasized to users that the target of the testing was the interface, not the users themselves, and that they could ask questions or stop testing at any time. After signing the informed consent form, users were given a pretest questionnaire. This questionnaire was used to determine general information about a user’s history concerning their disorder and tasks that users used to help treat their disorder. Then users were given a list of seven tasks that covered the most important functionalities and critical features of the interface. During the usability test, users were instructed to “think-aloud” while navigating and using the interface. In the meanwhile, the testers actively watched the users perform each task and recorded their observations of the users in addition recording what users liked about the interface, and what caused them difficulty. Though there were significant concerns about testers creating an emotionally stressful situation by raising memories of the past disorder, no user showed distress or stopped their usability test. After the test was completed, the user filled out a posttest questionnaire and was thanked for their help. The tester would then immediately write a report for each user to ensure that no information was lost from a test. Once the tester had completed their three users, they generated a summary of problems as well as positive features of the interface indicated by users to be discussed with the other tester.
About the Test Users
The test users for the interface were all within the age
range of nineteen and twenty-three. Many
hospitals, clinics, and health centers were contacted for potential test users,
but they all did not want to have their patients tested for medical
reasons. As an alternative, the RISE
team searched for test users who had fully recovered from a depression or
anxiety disorder. Five users were found
to be suitable for usability testing. Each
of these five users had been fully recovered from their disorder for a minimum
of two years. This minimized the risk of
any emotional damage while still providing knowledgeable and helpful feedback
on the interface. Moreover, the
severity, length of time, and treatment of the disorders varied, helping
testers to gauge the importance and flexibility of certain functionalities for
different levels of the disorder. While users
did not currently suffer from a depression or anxiety disorder, their
participation was a valuable contribution to the development of the RISE
interface. The sixth user was a student
studying Computer Science at the
Users’ Response to Each Task
The first task was to utilize the immediate self-help functionality in the interface. There is a large button on the login screen of the interface that was supposed to be clicked to access these functions. Half of the users immediately saw this button and proceeded to find the directions to address their panic attack. Users checked off each step in the directions for using the self-help technique. However, three of the users failed to see the ‘self-help’ button immediately. One of these users attempted to login to the system by typing their name. However, the interface was not yet able to accept typed names in the login field and did not allow the user to proceed. This delayed the user and frustrated the user considerably. The other two troubled users took more than five seconds to find the button to click. If users are in need of immediate self-help, they would ideally need to access it quicker than they did during testing. The design of this button needs to be enhanced to make its functionality more evident and allow easier access to the self-help functions that users seek.
The second task was to complete a single entry in the daily log. Users had no difficulty finding the daily log functionality and easily utilized the interface. While having a portable daily log is essential to helping deal with depression, it was questioned whether the PDA provided data entry that was simple enough for an average user. Connectivity to a normal computer or a keyboard could greatly ease the difficulty of recording the daily log. This could be feasible since a daily log would most likely be written from the same location everyday. Creating this connectivity to ease logging and other input would be a significant improvement to the interface. Despite the issue of improving the method of input, users had no trouble finding the log functionality and completing the entry into the daily log.
The third task was to write an email to Dr. Smith. Users found the email function easily and several commented on the ease of use and clean look of the screen. One user noted the lack of a carbon copy function that could be used to email a nurse, psychiatrist, or family member with information that could be important. The carbon copy function has been added into the email interface. In general, all six users found the email capability of the interface to be well designed and very familiar.
The fourth task was for the user to make an appointment to meet with their psychiatrist. There was an error in the coding of the ‘minutes’ field. The ‘minutes’ field only offered the same options, from one to twelve, as the ‘hours’ field. Users were initially confused by this error but most simply picked an alternate time. Some users asked the tester to clarify what had happened. Besides this error, users had no problems making an appointment. The user with interface design experience noted the use of coded fields and the straightforward layout of fields on the screen. However, that user also suggested including a visual calendar system in a later revision similar to the calendar used in Microsoft Outlook or travel websites.
The fifth task was to track the medication usage on a particular day. This screen repeated the error in the ‘minutes’ field from the fourth task, creating an appointment. However, none of the users had any difficulty completing the fifth task once they overcame this error.
The sixth task was to evaluate the effectiveness of the medication and prescription. Users had no difficulty performing this task. Several users liked the use of the slider to evaluate the medicine. Since the slider started on the leftmost value, testers observed that users moved the slider from left to right, seemingly evaluating the medicine incrementally, instead of evaluating the medicine from the middle of the scale relative to the normal evaluation. Further usability testing might indicate a better starting value for the slider, near the middle of the scale.
The seventh task was to find an image of the user’s brother to help relax the user. Users had no difficulty navigating to the section with media content. This media content is uploaded by users who find specific pictures, songs, videos, and more relaxing resources. Users had no difficulty finding the picture or viewing it. Several users commented that the interface design was very easy to follow despite it being different from other sections in the interface due to its difference in functionality. The file navigation system used an expanding and collapsing list rather than a normal folder view that would require multiple-page flipping on a PDA. The navigation system seemed very effective, displaying many entries at once.
Problems Identified By the Test
During usability testing, several problems were observed by testers. Most of the users had similar problems or complaints about the interface and its functionality. One problem was a shortage of self-help techniques available. While the interface was developed enough to handle most basic functionality, two users noted a general shortage of self-help techniques. If a user needed a specific kind of self-help, they will be frustrated as the interface does not currently include a comprehensive library of techniques to help users. Creating a comprehensive library that would contain as many self-help techniques as possible is a necessary addition to the RISE interface. This is a very important addition which would improve user satisfaction and the interface’s functionality dramatically, though it will take the RISE developers a significant amount of research and time to complete. Currently, the RISE developers are focusing on improving the interface and getting all of the functionality working. Content additions would be made after the interface has been proven and fully functional.
The second problem was that the immediate ‘self-help’ button did not standout on the login screen. Buttons of higher importance need to be highlighted in a more effective manner to draw users’ attention. In a stressful situation, the self-help functionality must be blatantly obvious for users to see. This is a relatively easy design improvement. Improving the design of the ‘self-help’ button is extremely important towards improving the functionality of the interface.
Users also had an obvious problem navigating the interface after completing a task. The menu system was docked on the top of the interface. After users completed a self-help technique moving down the screen, they looked at the bottom of the screen to navigate to complete the next task. Not finding any navigational buttons on several screens, users were often confused. There was a large delay of five to twenty seconds before a user would look up to the top of the screen again to return to the menu. While users eventually learned to go to the top of the screen to navigate the interface, it was clearly an awkward visual adjustment that gave them difficulty. Moreover, there was no ‘back’ button. Though users were only one screen from the main menu, they looked for a ‘back’ button instead of the ‘main menu’ button. Correcting the phrasing and layout of the navigation system would drastically improve the navigation in addition to giving users a better mental picture of the interface. Recently, the menu system has been moved to the bottom of the screen and several navigational buttons were added to aide users. Additionally, rephrasing and reorganization have improved the navigation.
As noted earlier, two tasks revealed a coding error in the ‘minutes’ field. The range of values was from one to twelve instead of zero to sixty. Users were confused by this error, but it was a simple correction that has already been made.
One user attempted to login to the system by typing in their name. This functionality was not built into the interface and frustrated the user. While the system had a login screen in order to increase the amount of privacy and security, it has not been determined whether multiple users should be able to use the same system. Since a self-help system should be with a user at all times, it would seem impossible to allow multiple users. However, family members or others may need to have access to the system; creating a login screen would facilitate restricting access to other users besides the primary user. Typed login names can now be accepted by the interface along with multiple users, but user restrictions have not been implemented.
Users had difficulty navigating back to the main menu or self-help menu; the menu system was not intuitive. There are two menus from which all functionality is accessible. However, this design of the menu system was not readily apparent to the test users. Moreover, while many users wanted to go back to the previous menu they were on, they had trouble utilizing the menu system, instead looking for a ‘back’ button or arrow. Creating a menu system that is more obvious and easy to use would help users quickly find the information they need and access specific modules of the interface. Additionally, giving users the option of using the ‘menu’ button or ‘back’ button may be beneficial. Currently, the menu system has been improved and ‘ok’ buttons were added, but ‘back’ buttons have not been added.
Two users had slight difficulty logging out of the system. The ‘logout’ button was in one of the menus, but since the users had never seen or used the ‘logout’ button before, they failed to find the ‘logout’ button quickly. While this would be probably improved with repeated use of the system, it might be beneficial to have the ‘logout’ button directly on the interface. This problem has been addressed by the changes to the menu system and its placement.
Usability Test Summary
The usability tests were extremely helpful in identifying problems with the RISE interface. Addressing these problems and issues identified during usability testing will help users navigate the interface, find functionality, and further utilize self-help techniques for their disorder. A redesign of the interface has already occurred because of the usability tests.
While further usability testing and interface revisions could definitely be made to improve the RISE interface, several positive features were determined that should be maintained. The login screen was appreciated by several users. The use of colors did not distract for the user’s task and was said to be tasteful. Test users thought having check boxes next to exercises in the self-help directions helped them follow along. The general layout of most screens was very simplistic and consistent. Using the interface for the first time, the test users were never overwhelmed by the amount of information or layout of any screen. Additionally, test users agreed that using a PDA was a good hardware choice that they could easily adapt to despite the data entry difficulties. These positive features of the interface should be kept and reevaluated during any redesign.
In summary, users felt that the layout of the interface was clean, the use of colors was appropriate, and that most of the functionality was very easy to use. While there were some minor errors and significant navigational issues, test users were satisfied with the interface based on the first-time use. In the future, longer tests should be conducted to evaluate features that require more time, such as reading the daily log entries from the past week.
Peer Review Comments
After completing an initial draft of the final report, several peers reviewed the report and interface design. Several suggestions for improvement were gathered from their input. While some steps are written using numbers, others are written using letters. Additionally, some minor formatting and phrasing is inconsistent. These changes would be very easy to implement. Two peers applauded the navigational improvements that were made, but still felt that some of the button placement could have been improved. One person suggested using a slideshow viewer for pictures and movies that were uploaded. Overall, the interface was determined to have a clean design, work towards a meaningful purpose, and already contain many improvements from usability testing and redesign. However, the peer reviews indicated that significant improvements could still be made to the interface.