1	HCI Research Methods Ben Shneiderman ben@cs.umd.edu Founding Director (1983-2000), Human-Computer Interaction Lab Professor, Department of Computer Science Member, Institute for Advanced Computer Studies University of Maryland College Park, MD 20742
2	Scientific Approach (beyond user friendly) Specify users and tasks Predict and measure time to learn speed of performance rate of human errors human retention over time Assess subjective satisfaction (Questionnaire for User Interface Satisfaction) Accommodate individual differences Consider social, organizational & cultural context
3	Scientific Method - Controlled Experiment Practical Problem & Existing Theory Write a Lucid & testable Hypothesis Alter a small number of independent variables (treatment) Select & assign subjects Control other variables Measure small number dependent variables Apply statistical test Guidance for practitioners, refine theory, advice for experimenters
4	Scientific Method - Controlled Experiment Practical Problem & Existing Theory Write a Lucid & testable Hypothesis Alter a small number of independent variables (treatment) Select & assign subjects Control other variables Measure small number dependent variables Apply statistical test Guidance for practitioners, refine theory, advice for experimenters
5	Research Methods Controlled Experiments Theory-driven, hypothesis testing Modify Independent Variables à Measure Dependent Variables Ethnographic Methods Surveys & Questionnaires Logging & Automated Metrics
6	Usability Engineering User-Centered Design Processes Guidelines Documents and Processes Research-based (NCI, 2003) www.usability.gov/pdfs/guidelines.html User Interface Building Tools Expert Reviews and Usability Testing
7	Design Process – Data Gathering
8	Design Process - LUCID
9	Design Process - Contextual Design
10	Guidelines Document and Processes Social process for developers Records decisions for all parties to see Promotes consistency and completeness Facilitates automation of design Should contain philosophy and examples of: title screens, menus, forms, buttons, graphics, icons, fonts, colors, instructions, help, tutorials, error messages, … Multiple levels are desirable: standards, practices, guidelines Education, Enforcement, Exemption & Enhancement
11	Expert Reviews and Usability Testing Improved product quality Shorter development time More predictable development lifecycle Reduced costs Speed development Simplify documentation Facilitate training Lower support Fewer updates Improved organizational reputation Higher morale: staff and management
12	Expert Reviews Experienced reviewers Review every screen, menu, dialog box Spot inconsistencies and anomalies Suggest additions Disciplined approaches Heuristic evaluation: check if goals are being met Guidelines review: verify adherence Consistency inspection: terms, layout, color, sequencing Cognitive walkthrough: pretend to be a user following scenario Formal inspection: public presentation and discussion
13	Usability Testing Physical place and permanent staff vs. discount usability testing Focuses attention on user interface design Encourages iterative testing Pilot test of paper design Online prototype evaluation Refinement of versions Testing of manuals, online help, etc. Rigorous acceptance test Must participate from early stages Must be partners, not "the enemy” (Dumas & Redish, 1999; Nielsen, 1993)
14	Usability Testing
15	Usability Testing Videotaping valuable for later review & for showing designers or managers the problems that users encounter. Many variant forms of usability testing have been tried: Paper mockups Discount usability testing Competitive usability testing Universal usability testing Field test and portable labs Remote usability testing Can-you-break-this tests
16	Evaluation Methods Ethnographic Observational Situated Multi-Dimensional In-depth Long-term Case studies
17	Evaluation Methods Ethnographic Observational Situated Multi-Dimensional In-depth Long-term Case studies Domain Experts Doing Their Own Work for Weeks & Months
18	Evaluation Methods Ethnographic Observational Situated Multi-Dimensional In-depth Long-term Case studies
19	MILC example Evaluate Hierarchical Clustering Explorer Focused on rank-by-feature framework 3 case studies, 4-8 weeks (molecular biologist, statistician, meteorologist) 57 email surveys Identified problems early, gave strong positive feedback about benefits of rank-by-feature
20	MILC example Evaluate SocialAction Focused on integrating statistics & visualization 4 case studies, 4-8 weeks (journalist, bibliometrician, terrorist analyst, organizational analyst) Identified desired features, gave strong positive feedback about benefits of integration
21	Case Study Methodology 1) Interview (1 hr) 2) Training (2 hr) 3) Early Use (2-4 weeks) 4) Mature Use (2-4 weeks) 5) Outcome (1 hr)