SEMVAST

News

Subscribe to our MAILING LIST to receive emails about this topic (only a couple a month maximum)

Participants

Georges Grinstein - University of Massachusetts at Lowell
Jean Scholtz - Battelle Memorial Institute, Pacific Northwest Division
Catherine Plaisant - University of Maryland, College Park

Contact the PIs

Students:
Loura Costello - University of Massachusetts at Lowell (CS PhD student)
Heather Byrne- University of Massachusetts at Lowell (CS undergrad - REU student)
Albayrak, Adem - University of Massachusetts at Lowell (CS undergrad - REU student)
Swetha Reddy - University of Maryland (iSchool, Master student)
(past) Samiksha Piprodia - University of Maryland (iSchool, Master student)

1. Provide benchmark data sets with ground truth and develop corresponding metrics and automated tools for evaluation
Today our collection of released datasets consists of the VAST contest and Challenge datasets. An effort led by Mark Whiting of Pacific Northwest National Laboratory has contributed the development of six data sets in the context of the Threat Stream Generator Project of the National Visualization and Analytics Center.

The 3 VAST 2009 Challenge datasets (Badge and computer network traffic, Social network with geospatial component, video analysis)
The 4 VAST 2008 Challenge datasets (wiki edits, cell phone social network, spacio-temporal [boat migrations, and building evacuation])
And also previously available:
VAST 2007 Contest dataset (mostly text)
VAST 2006 Contest dataset (mostly text)

Lessons learned from the evaluation on the 2008 Challenge are available (TR version).
[TO BE POSTED SOON]We are working with colleagues at other institutions to help them make their datasets available to the community. Other data sets we are attempting to acquire/develop are in the areas of financial transactions, health records, sensor/ tracking data, and accident records in the petroleum industry. We are focusing on data sets where ground truth is known to at least some extent, and developing templates for researchers to contribute their benchmark data set and associated materials. The challenge is motivating people to the value of the donations and to attract a larger number of contributors.
We are also considering an alternative structure to the benchmark database to encourage the enrichment and growth of existing data sets. In this new approach individuals could add ground truth information in the form of comments to the data and saved as metadata. We would provide the ability to able to filter at various levels of metadata. For example a teacher may want to give his/her students the raw dataset with no metadata and make the metadata only available at the end of the project. This will be discussed at the Advisory Board meeting and with other educators and will be pursued in year two. A workshop is being organized in MAY 2009 on this topic.

2. Seed an infrastructure for the coordination of long term evaluation activities across the multiple core research and application domains

Visual analytics builds on multiple core research fields (e.g. information visualization, knowledge discovery, data mining, cognitive science, intelligent user interfaces, human-computer interaction) and will impact many visual analytics application domains (e.g. intelligence and business analysis, bioengineering and genomic research, transportation, emergency response).

To stimulate the research community to increase attention on the evaluation of Visual Analytics SEMVAST's PIs cochair(ed) several activities:

* The VAST 2008 and 2009 challenge: SEMVAST PIs are co-chairs of the VAST 2008 Challenge and VAST 2009 Challenge. The challenges are powerful activities allowing the community to evaluate their tools with representative tasks and data sets, and allows us to test our metrics and automated evaluation tools on the materials submitted. Rather than emphasize “winners”, the VAST 2008 Challenge attempts to engage the community more widely. The challenge consists in a Grand Challenge and several smaller mini challenges, which allow for multiple data types and problems and facilitate the participation of teams whose focus is only on specialized tools. A participant workshop organized during VisWeek allows teams who submitted an entry to any of the challenges to explore each other’s results, learn from each other. and participate in the preparation of the next challnege.

* Visual Analytics Evaluation workshop: May 29th 2009 in conjunction with the HCIL symposium.

* The PIs are guest editors of a special issue of Computer Graphics and Applications (IEEE CG&A) May/June 2009, The submission deadline as passed, and the 4 papers selected are listed below (in publications)

* Beliv'08: BEyond time and errors: novel evaLuation methods for Information Visualization, a full day workshop April 5th 2008 at the ACM CHI 2008 conference in Florence (Beliv'08 proceedings - ACM Digital Library)

* VAST 2007 workshop on Metrics for the Evaluation of Visual Analytics, a full day workshop we organized on October 28th 2007 at the IEEE VIS 2007 conference. Short Report: Media:infovis_workshop.doc

*** Other activities may be listed in our wiki (not active at the moment)

Publications

Grinstein, G., Plaisant, C., Laskowski, S., O’Connell, T., Scholtz, J., Whiting, M., VAST 2008 Challenge: Introducing Mini-Challenges, Proc. of IEEE Symposium on Visual Analytics Science and Technology (2008) 195-196. (TR version).

Plaisant, C., Grinstein, G., Scholtz, J., Whiting, M., O'Connell, T., Laskowski, S., Chien, L., Tat, A., Wright, W., Gorg, C., Liu, Z., Parekh, N., Singhal, K., Stasko, J. Evaluating Visual Analytics: The 2007 Visual Analytics Science and Technology Symposium Contest IEEE Computer Graphics and Applications 28, 2, March-April 2008, pp.12-21 (2008) (final version in IEEE DL)

Whiting, M., Haack, J., Varley, C., Creating realistic, scenario-based synthetic data for test and evaluation of information analytics software, Proc. of BELIV’08, BEyond time and errors: novel evaLuation methods for Information Visualization, a workshop of the AVI 2006 International Working Conference (2004), ACM (2008) .(Published version)

Plaisant, C., Fekete, J. D., Grinstein, G., Promoting Insight Based Evaluation of Visualizations: From Contest to Benchmark Repository, IEEE Transactions on Visualization and Computer Graphics, 14, 1 (2008) 120-134 (TR version).

Grinstein, G.; Plaisant, C.; Laskowski, S.; O'Connell, T.; Scholtz, J.; Whiting, M.; VAST 2007 Contest - Blue Iguanodon, Proceedings of the IEEE Symposium on Visual Analytics Science and Technology, VAST 2007, pp 231 - 232 (Final version in IEEE DL)

Earlier relevant papers

Shneiderman, B., Plaisant, C., Strategies for Evaluating Information Visualization Tools: Multidimensional In-depth Long-term Case Studies, Proc. of BELIV’06, BEyond time and errors: novel evaLuation methods for Information Visualization, a workshop of the AVI 2006 International Working Conference, ACM (2006) 38-43 (TR version).

Plaisant, C., The Challenge of Information Visualization Evaluation, Proc. of Conf. on Advanced Visual Interfaces AVI'04 (2004), p.109-116. (TR version)

Papers from CG&A May/June 2009 (vol. 29 no. 3) Special Issue on Visual Analytics Evaluation (that we edited)

Intro to special issue on Visual-Analytics Evaluation
Catherine Plaisant, University of Maryland, Georges Grinstein, University of Massachusetts Lowell and Jean Scholtz, Pacific Northwest National Laboratory
pp. 16-17

Generating Synthetic Syndromic-Surveillance Data for Evaluating Visual-Analytics Techniques
Ross Maciejewski, Ryan Hafen, Stephen Rudolph, George Tebbetts, William S. Cleveland, David S. Ebert, Purdue University and Shaun J. Grannis, Indiana University
pp. 18-28

To Score or Not to Score? Tripling Insights for Participatory Design
Michael Smuc, Eva Mayr, Tim Lammarsch, Wolfgang Aigner, Silvia Miksch, Danube University Krems, and Johannes Gärtner, Ximes
pp. 29-38

Integrating Statistics and Visualization for Exploratory Power: From Long-Term Case Studies to Design Guidelines
Adam Perer, IBM and Ben Shneiderman, University of Maryland
pp. 39-51

Recovering Reasoning Processes from User Interactions
Wenwen Dou, Dong Hyun Jeong, Felesia Stukes, William Ribarsky, Heather Richter Lipford, Remco Chang, University of North Carolina, Charlotte
pp. 52-61

Sponsor

(last updated in May 2009)