SEMVAST

News

The Visual Analytics Benchmark Repository is now available. It allows you to find benchmarks but also add new ones, or descriptions of the use of existing benchmarks (e.g. analysis done useing them), and to upload or point to papers that report on the use of the benchmark datasets. If you are interested in helping us test it, please contact the PIs.

A blog now allows you to ask questions or publicize papers and relevant activities,

Subscribe to our MAILING LIST to receive emails about this topic (only a couple a month maximum)

Participants

Georges Grinstein - University of Massachusetts at Lowell
Jean Scholtz - Battelle Memorial Institute, Pacific Northwest Division
Catherine Plaisant - University of Maryland, College Park

Contact the PIs

Students:
Loura Costello - University of Massachusetts at Lowell (CS PhD student)
Heather Byrne- University of Massachusetts at Lowell (CS undergrad - REU student)
Albayrak, Adem - University of Massachusetts at Lowell (CS undergrad - REU student)
Swetha Reddy - University of Maryland (iSchool, Master student)
(past) Samiksha Piprodia - University of Maryland (iSchool, Master student)

Our project has focused on two activities: 1) making benchmarck data sets available and 2) seeding an infrastructure for evaluation.

The VAST Challenge serves as a Testbed for our activities.
SEMVAST PIs are co-chairs of the VAST Challenge, organized as a submission category of the IEEE Visual Analytics Science and Technology (VAST) Symposium. See the VAST 2008 Challenge and VAST 2009 Challenge. The challenges are powerful activities allowing the community to evaluate their tools with representative tasks and data sets, and allows us to test our metrics and automated evaluation tools on the materials submitted. Rather than emphasize “winners”, the VAST Challenge attempts to engage the community more widely by awarding multiple awards that recognize effective designs.

1. Making benchmark data sets with ground truth available

Our collection started with the released datasets of the VAST contest and Challenge datasets. Those datasets were developed by the team of Mark Whiting of Pacific Northwest National Laboratory in the context of the Threat Stream Generator Project of the National Visualization and Analytics Center. As part of the VAST Challenge organization we worked with Mark and his team to develop all the evaluation materials for those benchmarks.

VAST datasets:

The 3 VAST 2009 Challenge datasets (Badge and computer network traffic, Social network with geospatial component, video analysis)
The 4 VAST 2008 Challenge datasets (wiki edits, cell phone social network, spacio-temporal [boat migrations, and building evacuation])
And also previously available:
VAST 2007 Contest dataset (mostly text)
VAST 2006 Contest dataset (mostly text)

Other datasets:

We are also working with colleagues at other institutions to help them develop new data sets and make them available to the community. Examples of data sets we acquired are in the areas of financial transactions, and epidemiology. Other target areas are health records, sensor/ tracking data, and accident records in the petroleum industry. We are focusing on data sets where ground truth is known to at least some extent.

NEWS: The Visual Analytics Benchmark Repository is now available. It allows you to find benchmarks but also add new ones, or descriptions of the use of existing benchmarks (e.g. analysis done useing them), and to upload or point to papers that report on the use of the benchmark datasets. If you are interested in helping us test it, please contact the PIs. Please let us know how to improve it!

Using the VAST Challenge as our testbed, we developed new methodologies to evaluate Visual Analytics technology thru competitions. We developed automated metrics for some aspects of Visual Analytics systems, and guidelines will help researchers assess the subjective aspects of visual analytics environments.  Data from 2009 reviews and from experiments currently underway with professional analysts are the basis of those guidelines.  Lessons learned from the evaluation on the 2008 Challenge are available in 2009 Infovis Journal (see below in publications).

Examples of lessons learned include:

- By redefining the contest format into a challenge with multiple mini-challenges we were able to increase participation dramatically.
- While publicly available data sets are useful (and have been available for a long time), benchmarks need to include 1) data 2) embedded realistic ground truth scenarios and 3) tasks to be useful for adequately assessing the utility of Visual Analytics tools.
- The availability of benchmarks encourages research on the topic of the benchmarks.
- Users of those benchmarks learn from each others.
- Having both professional users (i.e. analysts) AND visual analytics experts act as judges is essential.
- Computing accuracy metrics in the context of realistic analysis scenarios is a challenge. Teams participating in mini-challenge 2 of the VAST 2009 Challenge (the social network mini-challenge), were able to obtain accuracy scores for the social network they identified from the data. Teams were allowed to submit and receive an accuracy measure three times prior to submitting their final entry. We were not 100% successful on computations but the participants were appreciative of the feedback.
- In addition to accuracy measures, we developed guidelines to help researchers assess the subjective aspects of visual analytics environments.  Data from this year's reviews and from experiments currently underway with professional analysts will form the basis of those guidelines.  Ground truth is rarely black and white. Participants are asked to provide not only “answers” but evidence from the data that supports their answers. As our data sets are often too large to be reviewed manually, participants may find additional material that provides interesting data we were unaware of. Judging needs to take those additional insights into consideration.

3. Seeding an infrastructure for the coordination of long term evaluation activities

Visual analytics builds on multiple core research fields (e.g. information visualization, knowledge discovery, data mining, cognitive science, intelligent user interfaces, human-computer interaction) and will impact many visual analytics application domains (e.g. intelligence and business analysis, bioengineering and genomic research, transportation, emergency response). We are working to make the methodolgy and tools known and available in the VAST community as well as outside this community.

We have developed several tools and services that others can use:

* The Benchmark Repository (to be made public soon)
* A submission and evaluation management website adapted to the needs of Visual Analytics competitions (e.g. multimedia submissions, automatic accuracy ratings, multiple types of reviewers, etc.). It is available at http://vastsubmission.cs.uml.edu (but of course requires login and password).

To stimulate the research community and increase attention given to the evaluation of Visual Analytics SEMVAST's PIs have led several activities:

* The VAST challenge: e.g. VAST 2008 Challenge , and VAST 2009 Challenge.
See Slides of introduction to the VAST Challenge (May 2009 - from HCIL workshop)

* As part of the ACM SIGKDD Workshop on Visual Analytics and Knowledge Discovery (VAKD’09) (June 28th, Paris France) two SEMVAST PIs co-chaired a challenge using our VAST 2008 dataset which resulted in several original papers reporting on that experience.

* The 2009 Visual Analytics Evaluation workshop: May 29th 2009 in conjunction with the HCIL symposium. The workshop gathered 20 designers, developers, evaluators, professional analysts and program managers from academia, industry or government.

* A special issue of Computer Graphics and Applications (IEEE CG&A) May/June 2009, was coedited by the PIs. The 4 accepted papers are listed in the publications section.

* Beliv'08: BEyond time and errors: novel evaLuation methods for Information Visualization, a full day workshop April 5th 2008 at the ACM CHI 2008 conference in Florence (Beliv'08 proceedings - ACM Digital Library)

* VAST 2007 workshop on Metrics for the Evaluation of Visual Analytics, a full day workshop we organized on October 28th 2007 at the IEEE VIS 2007 conference. Short Report: Media:infovis_workshop.doc

* We worked with Dr. Chris North from Virginia Tech (VT) to understand how to best use the VAST ground truth datasets in class projects. Chris has used VAST contest data in his VT fall 2007 graduate class and in his VT spring 2008 undergraduate class. He also used the 2009 data sets in an analyst evaluation program. He contributed the outline and results for his graduate course. and will continue in 2009 (e.g see CS3724 HCI Class and CS5764 InfoVis class) (see VAST 2009 paper in publication section). Jean Scholtz looked at the contributed results and is monitoring the progress on the undergraduate course. We are currently making sure that the datasets are in reasonable shape to release to other educators and exploring how to assess feedback from the educators in regards to both the datasets and the metrics.

* Dr. Grinstein has used the data sets for his class this fall and will be using another in his new Visual Analytics class to be taught either spring or fall 2010. He also used the data sets in the DHS COE CCICADA 2009 Reconnect workshop on visual analytics.

* Dr. Haesun Park used the VAST 2009 datasets for the Summer Intern Program under the NSF/DHS FODAVA (foundations of data and visual analysis) program.

*** Other activities may be listed in our wiki (not active at the moment)

Publications

Whiting, M., North, C., Endert, A., Scholtz, J., Haack, J., Varley, C., Thomas, J., VAST Contest Dataset Use in Education, to appear in Proc. of VAST 2009 Symposium. (PDF)

Costello, L., Grinstein, G., Plaisant, C. and Scholtz, J., Advancing User-Centered Evaluation of Visual Analytic Environments through Contests, Information Visualization, 8 (2009) 230–238 (TR version) (Palgrave final version).

Grinstein, G., Plaisant, C., Laskowski, S., O’Connell, T., Scholtz, J., Whiting, M., VAST 2008 Challenge: Introducing Mini-Challenges, Proc. of IEEE Symposium on Visual Analytics Science and Technology (2008) 195-196. (TR version) ( Published IEEE DL version)

Plaisant, C., Grinstein, G., Scholtz, J., Whiting, M., O'Connell, T., Laskowski, S., Chien, L., Tat, A., Wright, W., Gorg, C., Liu, Z., Parekh, N., Singhal, K., Stasko, J. Evaluating Visual Analytics: The 2007 Visual Analytics Science and Technology Symposium Contest IEEE Computer Graphics and Applications 28, 2, March-April 2008, pp.12-21 (2008) (final version in IEEE DL)

Whiting, M., Haack, J., Varley, C., Creating realistic, scenario-based synthetic data for test and evaluation of information analytics software, Proc. of BELIV’08, BEyond time and errors: novel evaLuation methods for Information Visualization, ACM (2008).(Published version)

Plaisant, C., Fekete, J. D., Grinstein, G., Promoting Insight Based Evaluation of Visualizations: From Contest to Benchmark Repository, IEEE Transactions on Visualization and Computer Graphics, 14, 1 (2008) 120-134 (TR version) (Published version in IEEE DL).

Grinstein, G.; Plaisant, C.; Laskowski, S.; O'Connell, T.; Scholtz, J.; Whiting, M.; VAST 2007 Contest - Blue Iguanodon, Proc. of the IEEE Symposium on Visual Analytics Science and Technology, VAST 2007, pp 231 - 232 (Final version in IEEE DL)

In preparation

Costello, L., Byrne, H., Ly, A. and Grinstein, G., A Survey of Contests in the Computer Science Community (draft version)

Slides

Overview of the VAST challenge , from the HCIL Visual Analytics Workshop, May 2009 (Plaisant)

Visual Analytics Evaluation from 2008 Summer Camp (Scholtz and Whiting)

Other relevant papers

Scholtz, J., Cook, K. A., Whiting, M., Lemon, D., Greenblatt, H., Visual analytics technology transition progress, to appear in Information Visualization (2009)

Plaisant, C., Laskowski, S., Evaluation Methodologies for Visual Analytics Section 6.1, in Thomas, J., Cook, K. (Eds.) Illuminating the Path, the Research and Development Agenda for Visual Analytics, IEEE Press, 150-157 (2005) (part of this book chapter)

Shneiderman, B., Plaisant, C., Strategies for Evaluating Information Visualization Tools: Multidimensional In-depth Long-term Case Studies, Proc. of BELIV’06, BEyond time and errors: novel evaLuation methods for Information Visualization, a workshop of the AVI 2006 International Working Conference, ACM (2006) 38-43 (TR version).

Plaisant, C., The Challenge of Information Visualization Evaluation, Proc. of Conf. on Advanced Visual Interfaces AVI'04 (2004), p.109-116. (TR version) ( Published version in ACM DL)

Papers from CG&A May/June 2009 (vol. 29 no. 3) Special Issue on Visual Analytics Evaluation (that we edited)

Intro to special issue on Visual-Analytics Evaluation
Catherine Plaisant, University of Maryland, Georges Grinstein, University of Massachusetts Lowell and Jean Scholtz, Pacific Northwest National Laboratory
pp. 16-17

Generating Synthetic Syndromic-Surveillance Data for Evaluating Visual-Analytics Techniques
Ross Maciejewski, Ryan Hafen, Stephen Rudolph, George Tebbetts, William S. Cleveland, David S. Ebert, Purdue University and Shaun J. Grannis, Indiana University
pp. 18-28

To Score or Not to Score? Tripling Insights for Participatory Design
Michael Smuc, Eva Mayr, Tim Lammarsch, Wolfgang Aigner, Silvia Miksch, Danube University Krems, and Johannes Gärtner, Ximes
pp. 29-38

Integrating Statistics and Visualization for Exploratory Power: From Long-Term Case Studies to Design Guidelines
Adam Perer, IBM and Ben Shneiderman, University of Maryland
pp. 39-51

Recovering Reasoning Processes from User Interactions
Wenwen Dou, Dong Hyun Jeong, Felesia Stukes, William Ribarsky, Heather Richter Lipford, Remco Chang, University of North Carolina, Charlotte
pp. 52-61

Acknowledgments

We also thank the National Visualization and Analytics CenterTM (NVACTM) located at the Pacific Northwest National Laboratory in Richland, WA, their their contribution to the VAST Challenge. The Pacific Northwest National Laboratory is managed for the U.S. Department of Energy by Battelle Memorial Institute under Contract DE-AC05-76RL01830. We also wish to thank Jereme Haack, Carrie Varley and Cindy Henderson of Pacific Northwest National Laboratory; Andrew Canfield from Mercyhurst College, the other students of the University of Massachusetts Lowell who worked to manage the VAST Challenge submission website, especially Shawn Konecni.

(last updated in January 2010)