1	VAST 2006 Contest A Tale of Alderwood Georges Grinstein – University of Massachusetts Lowell Catherine Plaisant – University of Maryland Theresa O’Connell – National Institute of Standards and Technology Sharon Laskowski – National Institute of Standards and Technology Jean Scholtz – Pacific Northwest National Laboratory Mark Whiting – Pacific Northwest National Laboratory
2	VAST 2006 Contest A Tale of Alderwood Georges Grinstein – University of Massachusetts Lowell Catherine Plaisant – University of Maryland Theresa O’Connell – National Institute of Standards and Technology Sharon Laskowski – National Institute of Standards and Technology Jean Scholtz – Pacific Northwest National Laboratory Mark Whiting – Pacific Northwest National Laboratory
3	Contest Goals Push the field of visual analytics Develop evaluations metrics and benchmark datasets for visual analytics: overall metrics (quality of answers) user-centered component (quality of visualization/interaction) 2 components Offline contest (5 months to solve problems) A closed-door live session for winners working with professional analysts
4	Data Set Synthetic Data set created by PNNL - Mark Whiting’s team. (a base of real news + inserted human-created plot and elements) About 1200 news stories from the Alderwood Daily News Some images, maps, voter registry, phone call log (excel tables) Background information
5	Sample text Alderwood to probe voting machines - Story by Ellie Olmsen, Date Published to Web 11/16/2004 Republicans in Alderwood joined Democrats yesterday in criticizing the performance of the city's costly new high-tech voting system, saying that it may have disenfranchised voters in the Nov. 4 election. The Republican commission scolded the city board of elections for minimizing problems with the touch-screen machines that the city purchased this year for $1.5 million and asked Mayor Rex Luthor to investigate what went wrong before the machines are pressed into service again. Alderwood's touch-screen voting machines, which resemble laptop computers without keyboards, were supposed to simplify voting and tabulating results. But in a debut that mirrored many of the problems experienced last year in areas across the country, some voters found the machines confusing, and the reporting of vote tallies was delayed almost a day. Luthor responded that he would try to address the board's concerns. He said he has called for a public meeting of the three-member board of elections to go over the requests at 5 p.m. today. "I pledge that I will answer every question as soon as I possibly can in the proper fashion," he said.
6	Sample image
7	City Hall Phone Logs
8	Voter Registry
9	Tasks Determine the inappropriate activities taking place Report hypotheses and conclusions including people, places, events Identify the associated relevant documents Answer form (answers and process) Video 2 page summary (see compendium)
10	Who – What – Where – Debriefing - Process VIDEO – 2 page summary
11	Who – What – Where – Debriefing - Process VIDEO – 2 page summary
12	Who – What – Where – Debriefing - Process VIDEO – 2 page summary
13	Who – What – Where – Debriefing - Process VIDEO – 2 page summary
14	Who – What – Where – Debriefing - Process VIDEO – 2 page summary
15	Who – What – Where – Debriefing - Process VIDEO – 2 page summary
16	A taste of the activities Suspicious weekend calls from town hall to Switzerland Mad cow disease outbreak New lab created, headed by strange character Politician hanky-panky Voting machines irregularities And many more…
17	Schedule
18	Advertising and dissemination Numerous IEEE and ACM sites KDD, European visualization and graphics groups Personal emails Last year’s IEEE Visualization participant list Web search engines Other meetings where visualization was discussed
19	Submissions 6 submissions 3 student teams 2 companies 1 team of analysts (using a manual process)
20	Judging – the process Two full days of judging in two separate locations 10 judges Experts in visual analytics, human-computer interaction and visualization Professional analysts Scoring based on Correctness of answers and evidence provided Assessment of Displays Interactions (as seen on video) Support of analytic process Generalizations (scaling, different tasks) Clarity of explanations Contest chairs met, reviewed the results and made final awards
21	The evaluation process – what went well Enough entries to make the evaluation exercise valuable Ground truth allows use of some quantitative metrics Synthetic dataset found realistic (positive feedback from users) Even problems/errors found representative of real datasets Video is extremely useful, live action is even better
22	The evaluation process – what was difficult Judging remains a qualitative process Unknown possible subplots exist in the data Need metrics for visualizations, analytic product and for value added by software More quantitative measures for judging quality of demonstration Or perhaps ask for certain subtasks to be demonstrated Small errors in dataset may confuse analysis Dataset is unrealistic in that it is closed Cannot use outside sources to confirm individual pieces of data Difficult to create numerous large datasets to allow this (phone directories, maps of towns, etc.)
23	Three winning entries
24	First place - corporate category Oculus Info Inc. Pascale Proulx, Lynn Chien, Adam Bodnar, Kaleb Ruch, Bill Wright
25	First place - student category Georgia Institute of of Technology Summer Adams, Kanupriya Sunghal (with Susan Gov and Sheena Lewis)
26	Second place - corporate category SSS Research, Inc. Russell A. Lankenau, M. Andrew Eick, Alexander Decherd, Maxim Khailo, Phil Paris, Jesse Fugitt
27	and now … Each team will have 7-8 minutes to present their tool Then we will Summarize yesterday’s live session Open to a panel format for questions/answers
28	Presentations
29	The interactive session (late last night) Purpose: to allow winning teams access to a real analyst for several hours Each team = 1 professional analyst + 2 developers Analyze similar but smaller data set 1/2 hour training 2 hour work (videotape provided to team for later analysis) Debriefing Observers (1 per team + 3 roaming) Similar metrics collected (correctness of answers, e.g., number of subplots located, ease of use of system) but focus was more on process and learning from each other (contestants, analysts, and committee)
30
31
32	The interactive session (continued) An amazing experience For analysts, developers, and us For learning, improving tools, exchange knowledge, processes Lessons learned Can evaluate tools in 2 hours Need more time to preprocess data Working through the problem was For the analyst, substantially better than a demo For the developer, a real test of their support of the analytic process
33	Conclusions In the long term data complexity will be increased by adding more data types providing larger volumes of data adding uncertainty increasing deception increasing the complexity of scenarios providing data in increments We will refine and validate evaluation metrics We will automate some of the judging process We will make future contests appeal to larger and more diverse audiences One of our goals as a community should be to organize a TREC-like series of yearly evaluations or competitions
34	VAST 2007 contest One, more complex, data set with a new scenario Task similar – identify the five W’s (who, what where, when, why) Currently plan 2 levels of the contest Raw data (as we did this year) Pre-processed data (the essence of the data set for non-text processing community) Tagged objects (names, places, dates, …) with referenced documents and location in document Index of documents, available as a downloadable search tool or a static table (words → documents) Plan to work with KDD community and InfoVis contest Schedule approximately the same with mid-January announcement
35	Acknowledgements Partial Support from: The Disruptive Technology Office The National Visualization and Analytics Center^TM (NVAC^TM) located at the Pacific Northwest National Lab. in Richland, WA. (The Pacific Northwest National Laboratory is managed for the U.S. Department of Energy by Battelle Memorial Institute under Contract DE-AC05-76RL01830) Extra thanks to: The UMass Lowell students The judges and analysts All those who participated, for the effort they took on to participate in this contest Extra thanks to MicroSoft for support for the live event
36	To know more… Poster session tonight at reception Teams’ 2-page summaries in Compendium, and ACM DL (soon) 2-page contest overview in Proceedings http://www.cs.umd.edu/hcil/VASTcontest06/ All submitted materials posted Dataset will remain available for download Solution posted Questions grinstein@cs.uml.edu, plaisant@cs.umd.edu jean.scholtz@pnl.gov, sharon.laskowski@nist.gov theresa.oconnel@nist.gov, mark.a.whiting@pnl.gov