SEMVAST

Scientific Evaluation Methods for Visual Analytics Science and Technology

Participants

Catherine Plaisant - University of Maryland, College Park
Georges Grinstein - University of Massachusetts at Lowell
Jean Scholtz - Battelle Memorial Institute, Pacific Northwest Division

Students:
Loura Costello - University of Massachusetts at Lowell
Heather Byrne- University of Massachusetts at Lowell (REU student)
Albayrak, Adem - University of Massachusetts at Lowell (REU student)
Samiksha Piprodia - University of Maryland

Contact us

Project Description

ALL our activities are listed in this wiki page
https://wiki.cs.umd.edu/semvast

Visual analytics is the science of analytical reasoning facilitated by interactive visual interfaces. As new visual analytics methods and tools are developed an evaluation infrastructure is needed. There is currently no consensus on how to evaluate visual analytics systems as a whole. It is especially difficult to assess their effectiveness as they combine multiple low level components (analytical reasoning, visual representations, computer human interactions, data representations and algorithms, tools for communicating the results of such analyses) integrated in complex interactive systems that requires empirical user testing. Furthermore, it is difficult to assess the effectiveness without realistic data and tasks.
Our long term goals are to:
1.Provide benchmark data sets with ground truth and develop corresponding metrics and automated tools for evaluation at the system and component level

2.Seed an infrastructure for the coordination of long term evaluation activities across the multiple core research and application domains

Visual analytics builds on multiple core research fields (e.g. information visualization, knowledge discovery, data mining, cognitive science, intelligent user interfaces, human-computer interaction) and will impact many visual analytics application domains (e.g. intelligence and business analysis, bioengineering and genomic research, transportation, emergency response). A survey of visual analytics evaluation methodologies across those disciplines is needed. An evaluation infrastructure will be seeded with benchmark data sets with ground truth and corresponding tasks, metrics definitions and tools to automate measurements, and the beginning of an online community to encourage collaboration and sharing of qualitative and quantitative methods amongst researchers.

Community wide, systematic evaluations of visual analytic systems will produce better understanding of the issues in the core research fields involved in visual analytics as well as the issues that cross between those research fields. A sharable set of user centered evaluation methods, benchmarks and metrics will be developed, which will allow researchers to assess the utility of their own techniques, in their own application domain. New approaches to the preparation and use of datasets with ground truth for empirical evaluation will be devised. Tools will be available for facilitating measurements of utility.

Sponsor

This material is based upon work supported by the National Science Foundation under a Collaborative Research Grant to the following three institutions:
- IIS-0713087 Plaisant, Catherine University of Maryland, College Park
- IIS-0712770 Scholtz, Jean Battelle Memorial Institute
- IIS-0713198 Grinstein, Georges University of Massachussetts, Lowell
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.