IEEE VAST 2006
Contest
Update History and FAQ
February 2007: Preprocessed Data added
To allow people to practice for the 2007 contest we
now provide preprocessed data for the 2006 contest data. Just register as
usual and download the 2006 data set and you will find the preprocessed
data with it. Here is the direct link to
the readme file to have an idea of what we
provide.
August 8th
2006: Results announced, see home page.
See you at the conference…
June 29th: FTP info
We added the information about “how to submit” in
the main page.
June 16th: Answer Form
The template for the Answer Form has been posted. Note that the answer form has a less
questions than was originally listed in the webpage about the tasks (I.e. we do
not ask for a list of activities but obviously those activities will be
reflected in the briefing which summarize you answer to the general question: “What is the situation and what is your assessment of the situation?”
The “how to submit” info on
the home page has been expanded and clarified.
Soon we will also give you
the FTP site where you will be able to upload your submission. If you need it immediately, let us know…
June 15th : No
extension
Will we be giving extensions? NO. We have a tight reviewing schedule because
of summer, so we cannot give an extra week to submit. Sorry.
If you have problems with videos: let us know…
A flier
advertising the contest was also made available for download.
May 1st : 1 FAQ added
Q6. Can PNNL employees participate?
A6. No, sorry.
April 7th : 5
FAQs added (1-5)
Q1. Is this data set similar to those given to
real-life analysts?
A1.
The easy answer to this question is "yes" and
"no". Real-life analysts are
given anything from hundreds of thousands of network traffic records, to photographs
and excel spreadsheets, to forensic ballistic evidence. Much depends on what question that analyst is
being asked to answer, and the mission of the agency for which the analyst
works. The data set with which you are
working is a realistic one, and contains all manner of information, important
and not, through which a "real life" analyst would have to sift in
order to provide a hypothesis to a decision-maker. The plot of the dataset isn't one you might
find on the front page of the Washington Post, but then stranger things have
happened!
Q2. We're tool builders, not analysts. Can we participate without an analyst on our
team?
A2.
Yes! When it comes down to it,
analysis is simply using a systematic logical thinking methodology; like using
the "scientific method" for an experiment. There are standard
analysis methods that are well documented and that you could reference and use
in your efforts, for example: Link Analysis, Analysis of Competing Hypothesis,
Social Network Analysis.
If you are interested to learn more,
there are also a couple of books you might want to look at:
Heuer, Jr. Richard J, Psychology of
Intelligence Analysis
Morgan D. Jones, The Thinker's
Toolkit: 14 Powerful Techniques for
Problem Solving
A nice article about subjective
thinking and competing hypotheses is at:
http://www.dodccrp.org/events/10th_ICCRTS/CD/papers/126.pdf
(or our local copy of the paper)
or see the presentations at:
http://www.dodccrp.org/events/10th_ICCRTS/CD/presentations/126.pdf
Another site to explore:
http://www.insna.org/INSNA/na_inf.html
BUT remember that we are interested
in new approaches and ideas as well!
Surprise us… What really matters here is answering the questions.
Q3.
I found some interesting anomalies in the dataset. I should report these, right?
A3.
Possibly -- only report data anomalies if they are relevant to your
hypotheses and/or conclusions. For
example, if you found that the days of the news articles are only Monday,
Wednesday, and Friday, that might be an anomaly when just considering data, but
you shouldn't report that unless you can find a link to the scenario. However, if you found that stories about
"Sam" were always associated with Merino sheep and Merino sheep play
a part in your evolving hypotheses, then you should report this in some
way.
Q4.
Could you move the deadline to August? That would allow us to have a summer
intern to help out.
A4. Unfortunately the July deadline
IS firm as we need to determine who will participate in the live contest, and
also to collect the camera ready materials in time. Next year we will try to
announce and have the data set available as close to the end of the VAST
conference as we can manage.
Q5.
How much you are allowing teams to "build a tool to fit this data" –
i.e. we could wind up building a tool incrementally, trying to solve the
problem as we went thru it and bringing in custom pieces as needed, even
building more special tools to present the "answer" afterwards. I
wasn't sure if that was allowed. Some contests (like TREC) are not run that way
(e.g. TREC is more "run your best existing system on this test data",
you're not allowed to look at the test data). Then we definitely couldn't
participate, as we'd have to pull in a lot of things to extend our system to
work with this type of unstructured data.
A5.
Your approach is fine for the offline contest (i.e. not “live”). Just keep in mind that if your tool does well
and is selected for the live contest, it will be used for a different – but
similar – problem. The rules of the live
contest are not finalized yet, but we expect that a competing team will be
comprised of you (the tool builder) and a professional analyst, working
together to assess a new situation, in only a few hours. In any case it will be important that in your
contest submission describes the process by which you arrived at answers and
identify whether your success only can occur for this specific dataset (in
other words does your process generalizes?) If you feel it does then that's
great.
March 1st
Detailed description of tasks provided in the Dataset and Tasks page
1st draft of scoring methodology posted in the Judging
page
Feb 15th
The full dataset was posted.
The samples posted earlier were removed because they were not all part of the
final dataset and could be confusing.
We updated the schedule to reflect that we will post the 1st draft
of the scoring methodology by March 1st.
Feb. 1st 2006
Website is up, samples are posted.
Questions? Email
the Contest Chairs
Return to Dataset
and Tasks
Return to VAST
2006 Contest