1	The challenge of Information Visualization Evaluation
2	Information Visualization “The real voyage of discovery consists not in seeking new landscapes but in having new eyes” Marcel Proust Maturing field Design principles (e.g. overview, zoom and filter, detail on demand) Validated visualization techniques Many successful conferences
3
4
5	Two views of where field is Pessimistic view Still only few successful products Research ideas not transferred Evaluations often inconclusive Optimistic view Active field We have success stories Making progress understanding what works
6
7
8
9	"What can we do to" What can we do to Improve evaluations to guide potential adopters ?
10	A subliminal tour of a few “classics”
11
12
13
14
15
16
17	Treemap - Stock market, clustered by industry
18
19	CamTree - ConeTree
20	And more tools coming all the time …. e.g. for trees… Spacetree
21
22	"What can we do to" What can we do to Improve evaluations to guide potential adopters ?
23	"What can we do to" What can we do to Improve evaluations to guide possible adopters ?
24	"What can we do to" What can we do to Improve evaluations to guide possible adopters ?
25	Is Segway a better vehicle?
26	Is Segway a better vehicle?
27	Is Segway a better vehicle?
28	Is Segway a better vehicle?
29	"What can we do to" What can we do to Improve evaluations to guide potential adopters ?
30	"What can we do to" What can we do to Improve evaluations to guide potential adopters ?
31	Current evaluation practices Survey of literature Komlodi et al. (2004) UMBC 4 main groups Controlled experiments 1) comparing design elements 2) comparing 2 or more systems Usability evaluations Case studies of tools in realistic settings
32	Field Studies RARE Users in natural environments Real tasks Demonstrate feasibility and in-context usefulness Time consuming, not necessarily replicable or generalizable
33	Field Studies RARE Users in natural environments Real tasks Demonstrate feasibility and in-context usefulness Time consuming, not necessarily replicable or generalizable
34
35	Controlled experiments 1- Comparing design elements Researchers? Adopters?
36	Controlled experiments 1- Comparing 2 or more systems
37	Controlled experiments CAN provide convincing evidence 36 participants used either LifeLines or tabular display Series of tests
38	Strong results LifeLines twice as fast for some tasks (time interval comparison or inter-categorical information tasks) Better recall (out of 6 questions: 4.33 vs 2.83 correct) More accurate “1st impression”
39	Spacetree User Study
40	User Study 18 subjects, all used the 3 interfaces Order and task set counterbalanced 40 minute session
41
42
43
44	Subjective Preference: Attractiveness
45	Preference for Future Use
46	Time to return to previously visited node
47	Estimating size of subtrees
48	Estimating size of subtrees
49	Conclusions Preview icons and consistent layout improve node-link tree browsers Task analysis will help you chose the right browser www.cs.umd.edu/hcil/spacetree/
50	Common traps and the potential adopters’ point of view Using partial or poor implementations Can’t tell – confused by conflicting results Aggregating results for all tasks Can’t tell what works for THEIR tasks Using simple arbitrary tasks Wondering what would happen if users chose their tasks, worked on complex tasks, had more time, talked to colleagues Answered questions they didn’t know they had Combined with other visualizations, other tools Inadequate care of details (e.g. zooming, labeling, widgets) Can’t tell Inadequate training and user motivation What would happen with more training, or no training
51	Improving Evaluation Practices More qualitative field studies Looking at data from different perspectives, over a long time Answering questions users didn’t know they had Making discoveries The benefits of increased awareness Report on opportunistic real usage outside of formal tests (e.g. by developers, visitors using their real data…) “The goal of visualization is insight, not graphics” Document the insight generation process Quantify insight? quality versus effort?
52	Infovis Contests InfoVis 2003 Contest with J.-D. Fekete Pairwise Comparison of Trees 3 datasets Phylogenies (60 nodes) ; Classifications (200,000 nodes) File structure with usage logs (70,000 nodes) Rich task taxonomy Ask people to tell us WHAT THEY COULD SEE in the data Results in the InfoVis Benchmark Repository www.cs.umd.edu/hcil/InfovisRepository InfoVis 2004 Contest with J.-D. Fekete and G. Grinstein 10 years’ Anniversary of InfoVis Metadata about conference papers and their references Tasks: Show the history in one screen Topics and their evolution Relationships of authors and topics For each task report: Process, Images and Insights + Video One month left to submit…
53
54
55	Infovis Contests InfoVis 2003 Contest with J.-D. Fekete Pairwise Comparison of Trees 3 datasets Phylogenies (60 nodes) ; Classifications (200,000 nodes) File structure with usage logs (70,000 nodes) Rich task taxonomy Ask people to tell us WHAT THEY COULD SEE in the data Results in the InfoVis Benchmark Repository www.cs.umd.edu/hcil/InfovisRepository InfoVis 2004 Contest with J.-D. Fekete and G. Grinstein 10 years’ Anniversary of InfoVis Metadata about conference papers and their references Tasks: Show the history in one screen Topics and their evolution Relationships of authors and topics For each task report: Process, Images and Insights + Video One month left to submit…
56	Infovis Contests InfoVis 2003 Contest with J.-D. Fekete Pairwise Comparison of Trees 3 datasets Phylogenies (60 nodes) ; Classifications (200,000 nodes) File structure with usage logs (70,000 nodes) Rich task taxonomy Ask people to tell us WHAT THEY COULD SEE in the data Results in the InfoVis Benchmark Repository www.cs.umd.edu/hcil/InfovisRepository InfoVis 2004 Contest with J.-D. Fekete and G. Grinstein 10 years’ Anniversary of InfoVis Metadata about conference papers and their references Tasks: Show the history in one screen Topics and their evolution Relationships of authors and topics For each task report: Process, Images and Insights + Video
57	An Evaluation of Microarray Visualization Tools for Biological Insight Purvi Saraiya, Chris North, Karen Duca Virginia Tech
58	Motivations Bioinformatics microarray data analysis Data intensive, exploratory, want insight Many visualization tools What insight do they provide? Evaluating visualizations for “insight” Controlled studies, benchmark tasks, too narrow What is insight? How to measure insight?
59	Cluster/Treeview
60	TimeSearcher
61	HCE (Hier. Cluster Expl)
62	Spotfire
63	GeneSpring
64	Experiment Design 5 visualization tools 3 data sets Time series: 1000 genes, 5 time points Virus conditions: 850 genes, 3 viruses Lupus screen: 170 genes, 48 sick, 42 healthy 30 subjects Biology senior researchers Biology student researchers Bioinformatics software developers
65	Protocol Tool tutorial Dataset description List data questions Open-ended exploration About an hour Think aloud Periodic insight estimates
66	Insight Definition Insight = an individual data observation Fact Time to discover Domain value (importance) Hypotheses generated? Directed vs. unexpected Correctness Category (overview, patterns, groups, details) Can be recognized via Think-aloud
67
68	Insight Summary
69	Insight Curves
70	Insight Methodology Insight definition Open-ended insight Quantity, value, speed Difficulties: Labor intensive Requires domain expert Requires motivated subjects Short training and trial time
71	Encouraging adoption
72	Crossing the Chasm (Geoffrey Moore, 91) Innovators sell a new idea to small number of EARLY ADOPTERS who Enjoy new tools Try out every features Crossing the chasm is reaching the EARLY MAJORITY Pragmatists Want reliable, proven tools that solve real problems For us: Matching tools to users, tasks and real problems
73	Generality Paradox Generality can become an impairment Early Majority wants applications designed for them Researchers can invest into custom implementations Publish in specialized journals of chosen application domain Encouraging leap of faith with meaningful and realistic examples (pick a story to tell, use large enough datasets) FAA, DJJ
74	Generality Paradox Generality can become an impairment Early Majority wants applications designed for them Researchers can invest into custom implementations Publish in specialized journals of chosen application domain Encouraging leap of faith with meaningful and realistic examples (pick a story to tell, use large enough datasets) Report on observed opportunistic usage outside of formal tests (by developers, visitors - use their real data…) FAA, DJJ
75	Generality Paradox Generality can become an impairment Early Majority wants applications designed for them Researchers can invest into custom implementations Publish in specialized journals of chosen application domain Encourage the leap of faith with understandable and realistic examples (pick a story to tell, use large enough datasets)
76	Toolkits
77	Specialized toolkits
78
79	"3 case studies" 3 case studies Treemap Dynamic queries/Spotfire Dynamic maps
80	"3 case studies" 3 case studies Treemap Dynamic queries/Spotfire Dynamic maps
81	Convincing examples help adopters “cross the chasm” (example of Treemap and the Map of the Market)
82
83
84
85
86
87
88	Convincing examples help adopters “cross the chasm” (example of Treemap and the Map of the Market)
89	"3 case studies" 3 case studies Treemap Dynamic queries/Spotfire Dynamic maps
90	Versatility, but tailored (example of the FilmFinder and SpotFire)
91
92
93
94	"3 case studies" 3 case studies Treemap Dynamic queries/Spotfire Dynamic maps
95	Addressing Universal Usability (example of Dynamic map for US government statistics)
96
97
98
99
100	Addressing Universal Usability
101	Addressing Universal Usability Colorblindness
102	Addressing Universal Usability Colorblindness
103	Addressing Universal Usability Colorblindness