Skip to main content


Metrics for the Evaluation of Visual Analytics
A Vis-2007 Workshop:

A full day workshop, October 28th at the VIS 2007 conference

Plan for the Day

See the Agenda and position papers

Call for Position Papers (closed)

Deadline was September 15

The field of visual analytics is now recognized as a research area in many universities and organizations. As new fields develop ways of assessing, progress in those fields also expands. In the field of visual analytics, we are fortunate in that we already have lessons learned about evaluating visualizations. Unfortunately, these lessons still point out that this is a difficult problem. Visual analytics compound this problem by adding more dimensions: not only are we concerned with some measure of the visualizations, but we are concerned with evaluating the impact these visualizations have in helping analysts in their work. User-centered evaluations are vital in visual analytics as they contribute greatly to adoption of research software. The issues we face in developing user-centered evaluations for visual analytics are selecting:

    • The task: the tradeoff is between simple tasks that are easily evaluated and developing more realistic tasks that consume more time and are much less straightforward to evaluate
    • The corresponding dataset: the same issues as above plus the issues of developing a publicly releasable dataset that resembles a realistic dataset
    • The system and environment: how much does the system or environment play a role in the utility or success of the task.
    • The participants: access to senior analysts or junior analysts in evaluations and ensuring that analysts are open to new technology
    • Training: how much training should be provided to analysts prior evaluations or whether analysts should be paired with technologists to operate the software
    • The metrics: what combination of quantitative and qualitative measures will be accepted? How can we ensure that qualitative measures are collected with and meet some rigor? How can we measure insights that were derived from the visualization and interactions with the visualization? This is especially problematic as not all analysts approach problems in the same fashion. Most importantly, what measures are most helpful to the analytic community and to the research community?

Selected participants will receive copies of all accepted position papers. These participants will present their ideas or current research during the morning (about 10-15 minutes each). Based on the position papers and these presentations, the organizers will develop a list of possible metrics. An initial list will be distributed to the participants prior to the workshop. After all the presentations, this list will be discussed and refined by the participants.

In the afternoon session, the organizers will provide representative examples of different types of visual analytics systems (solicited from the participants for this purpose) and the workshop participants will test the list of metrics by evaluating these systems using the metrics. A discussion session will follow to identify successes and difficulties, and refine the list of metrics. The organizers will generate a report evaluating the metrics based on the participantsí usage and the discussions.

We will produce a poster from the workshop which will be included in the poster session. The poster will focus on the metrics used during the workshop and the lessons learned for each.

We will also consider a joint journal paper or future conference paper with the workshop participants contributing to the various metrics proposed by the workshop.

Submission of position papers:

Position papers should be 2 to 4 page long and focus on metrics and methods for evaluating visual analysis environments. If participants have used these methods already, please include lessons learned and references when appropriate. If the method has not yet been tried, please provide some estimates of the efforts that would be needed to implement these. Position papers should be submitted to all three organizers (see e-mail addresses below) no later than September 15th. Participants will be notified of acceptance no later than September 30th.

Organizers:

Jean Scholtz
Pacific Northwest National Laboratory, 340 Northslope Way, Rockaway Beach, OR 97136
Jean.scholtz@pnl.gov

Georges Grinstein
University of Massachusetts Lowell, Lowell MA 01854
grinstein@cs.uml.edu

Catherine Plaisant
University of Maryland, College Park, MD 20742, U.S.A.
plaisant@cs.umd.edu

FAQ: How does this relate to Beliv'08 Workshop?

Beliv'08 will take place at CHI in Florence. Beliv stands for "BEyond time and errors: novel evaLuation methods for Information Visualization"
While the VIS 2007 workshop focuses on metrics and will be conducted with brainstorming and teamwork sessions where we produce ideas of new metrics together - and try to test them with systems, the Beliv'08 workshop is meant to present and discuss new methodologies for evaluation in general, and has an emphasis on the presentation and discussion of refereed papers on this topic.