Skip to main content

Carlea Holl-Jensen||


(December 2013)
Evaluation of visual analytics environments: The road to the Visual Analytics Science and Technology challenge evaluation methodology
Information Visualization 13, 4 (2014) 326-335 [Published Version]

Evaluation of software can take many forms ranging from algorithm correctness and performance to evaluations that focus on the value to the end user. This paper presents a discussion of the development of an evaluation methodology for visual analytics environments. The Visual Analytics Science and Technology (VAST) Challenge was created as a community evaluation resource, that is, a resource available to researchers and developers of visual analytics environments that would allow them to test out their designs and visualizations and compare the results with the solution and the entries prepared by others. Sharing results allows the community to learn from each other and to hopefully advance more quickly. In this paper we discuss the original challenge and its evolution during the seven years since its inception. While the VAST Challenge is the focus of the paper, there are lessons for many involved in setting up a community evaluation program, including the need to understand the purpose of the evaluation, decide upon the right metrics to use and the appropriate implementation of those metrics including datasets and evaluators. For ongoing evaluations it is also necessary to track the evolution and to ensure that the evaluation methodologies are keeping pace with the science being evaluated. The discussions on the VAST Challenge on these topics should be pertinent to many interested in community evaluations.

Q&A: Supporting Engagement and Peer Learning in a Classroom Setting Screenshot

Q&A: Supporting Engagement and Peer Learning in a Classroom Setting
More information

Tech Reports
Video Reports
Annual Symposium

Seminars + Events
HCIL Seminar Series
Annual Symposium
HCIL Service Grants
Events Archives
HCIL Conference Travel Award
Job Openings
For the Press
HCIL Overview
Become a Member
Collaborating Groups + People
Academic Visitors
Join our Mailing List
Contact Us
Visit Us
HCIL Store
Give the HCIL a Hand
HCIL T-shirts for Sale
Our Lighter Side
HCIL Memories Page
Faculty/ Staff
Ph.D. Alumni
Past Members
Research Areas
Design Process
Digital Libraries
Physical Devices
Public Access
Research Histories
Faculty Listed by Research
Project Highlights
Project Screenshots
Publications and TRs
Studying HCI
Masters in HCI
PhD in HCI
Visiting Scholars
Class Websites
Sponsor our Research
Sponsor our Annual Symposium
Active Sponsorship
Industrial Visitors

Web Accessibility