IEEE VAST 2008 CHALLENGE

RECENT NEWS

March 20: Data sets now available! Register to download the data sets

Coming soon: More details about what to submit and evaluation criteria

See also: The FAQ’s and History of news and updates

Overview of the event

The VAST Challenge is a participation category of the IEEE VAST 2008 Symposium (part of VisWeek 2008). The VAST Challenge continues in the footsteps of the VAST 2006 and 2007 contests with the purpose of pushing the forefront of visual analytics tools using benchmark data sets and establishing a forum to advance visual analytics evaluation methods. We also hope it will speed the transfer of VA technology from research labs to commercial products, and increase the availability of evaluation techniques.

[NEW in 2008] In order to provide more opportunities for increased participation we now offer an overall Grand Challenge as well as several smaller Mini Challenges. Teams may enter one or more mini Challenges independently of the entering the Grand Challenge. ALL teams submitting an entry to a VAST Challenge will be invited to discuss their work during a challenge workshop.

Entries will be judged on both the correctness of the analysis (based on the availability of ground truth) and the utility of the tools in conducting the analysis. Participants have several months to prepare their submissions.

Grand Challenge and Mini Challenges

The Grand Challenge consists of 4 heterogeneous data sets (Register to download): a dataset of phone records; a dataset of geo-temporal records; a dataset of Wikipedia edit data and history; and a dataset of location tracking. Grand Challenge participants are expected to integrate the results of the analysis of all Mini Challenges to understand the overall situation (but are not required to submit entries to the individual mini challenges). Grand Challenge participants are asked to find evidence of suspicious activities and answer specific who, what, when, and where questions and to provide the relevant evidence for each. They are asked to provide a debrief describing the situation and to provide their process description. Both quantitative and qualitative measures will be used to evaluate the entries.

Mini Challenges are focused in the areas of social network analysis (phone transactions), modeling data analysis (e.g. evacuation modeling), unstructured text analysis (wiki edit records), and geo-temporal analysis. For each of these mini challenges participants have to answer specific questions that can be deduced strictly from the data provided. Participants are asked to provide a process description highlighting the visualizations and interactions used to arrive at their conclusions. Both quantitative and qualitative measures will be used to evaluate the entries.

Although the grand and mini-challenges are currently focused on homeland security topics we want to emphasize that both the problems and their solutions are indicative of very broad applicability. The challenges are very similar and representative of data sets that deal with medical, health, financial, educational, transportation, or social data. In fact it is our strong belief that good tools that succeed in solving the mini or grand challenges will easily apply to a broader collection of data sets, and we encourage individuals and groups from all disciplines to not only participate in the challenges but in the workshop and future planning of the VAST challenge activities.

At the IEEE VAST symposium

A VAST Challenge workshop will be held during the conference week, opened only to Challenge participants and Challenge sponsors. This format will allow all Challenge participants to learn from each other and help advance the science of visual analysis evaluation. During this workshop participants will discuss their process and results, provide feedback on evaluation metodology and provide suggestions for the VAST 2009 Challenge. Deserving entries will receive “certificates of excellence” in various categories.

Grand Challenge entries will once again be eligible to participate in an interactive session during which they will work with professional analysts on a new smaller problem. This interactive session is held during an evening of the symposium week and has been found extremely useful by past participants.

Representative teams from the Grand Challenge and from each mini Challenge will be invited to participate in a panel held during the VAST symposium.

The two page summaries from all teams submitting an entry to any Challenge will be published online (location still to be determined, probably on a NIST website).

The two page summaries of the most deserving entries will be also published in the VAST proceedings or compendium (to be determined by the symposium chairs depending on the number of papers to be published); and we also hope authors will consider submitting to the CG&A Special Issue on Visual Analytics Evaluation.

2008 Timeline

February 15 Sample data available

March 20 Data sets available (register to download)- Mini challenge topics finalized

July 11 Submissions deadline

Mid August Results returned to participants

Early September Camera ready copies of two page summaries due for publication

Oct. 19 – 24^th VisWeek - VAST Symposium, and Workshop for challenge participants

Rules

The challenge is open to all, except for the challenge organizers and judges. All categories of competitors (academic and commercial) may participate. If in doubt, ask the chairs.

Student teams must have a faculty sponsor and provide the faculty's contact information with their registration.

Teams may use any existing commercial product or research prototype and, of course, may combine tools. We strongly encourage teams to find partners with complementary expertise (e.g. groups with text analysis or reasoning tools might want to seek partners with user interface expertise). We can assist you if you are looking for partners, please ask!

At least one participant of the teams receiving recognition will have to attend the Symposium, so please discuss travel support issues early with your advisor or supervisor. If you are a student register as a student volunteer as early as possible. In the past we have been able to offer a few free registrations to the best entries and hope to be able to continue this tradition but cannot guarantee it at this time.

Remember, everybody who submits is invited to the workshop! (but only those who submit...)

Submission Information

Coming soon, but we will ask for a type of submission similar to previous VAST contest, i.e. expect to have to prepare a debriefing with your answers and a video demonstration. For the mini-challenge a few specific questions will be asked and the video will be optional.

VAST Challenge Chairs

Georges Grinstein, University of Massachusetts Lowell
Catherine Plaisant, HCIL, University of Maryland
Jean Scholtz, Pacific Northwest National Laboratory

Challenge Committee Members

Theresa OConnell, National Institute of Standards and Technology
Sharon Laskowski, National Institute of Standards and Technology
Mark Whiting, Pacific Northwest National Laboratory

Journal paper about the VAST 2007 contest

CG&A Special Issue on Visual Analytics Evaluation. (deadline Sept 12, 2008)