Skip to main content



HCIL-2010-08

Yeh, T., White, B., Davis, L., Katz, B. (May 2010)
Searching the Web Using Screenshots
HCIL-2010-08

Many online articles contain useful know-how knowledge about GUI applications. Even though these articles tend to be richly illustrated by screenshots, no system has been designed to take advantage of these screenshots to visually search know-how articles e ectively. In this paper, we present a novel system to index and search software knowhow articles that leverages the visual correspondences between screenshots. To retrieve articles about an application, users can take a screenshot of the application to query the system and retrieve a list of articles containing a matching screenshot. Useful snippets such as captions, references, and nearby text are automatically extracted from the retrieved articles and shown alongside with the thumbnails of the matching screenshots as excerpts for relevancy judgement. Retrieved articles are ranked by a comprehensive set of visual, textual, and site features, whose weights are learned by RankSVM. Our prototype system currently contains 150k articles that are classi ed into walkthrough, book, gallery, and general categories. We demonstrated the system's ability to retrieve matching screenshots for a wide variety of programs, across language boundaries, and provide subjectively more useful results than keyword-based web and image search engines.



Keshif:  Simplicity Driven Visual Faceted Browser Screenshot

Keshif: Simplicity Driven Visual Faceted Browser
More information

Tech Reports
Video Reports
Annual Symposium

News
Seminars + Events
Calendar
HCIL Seminar Series
Annual Symposium
HCIL Service Grants
Events Archives
Awards
HCIL Conference Travel Award
Job Openings
For the Press
HCIL Overview
Become a Member
Collaborators
Collaborating Groups + People
Academic Visitors
Join our Mailing List
Contact Us
Visit Us
HCIL Store
Give the HCIL a Hand
HCIL T-shirts for Sale
Our Lighter Side
HCIL Memories Page
Faculty/ Staff
Students
Ph.D. Alumni
Past Members
Research Areas
Communities
Design Process
Digital Libraries
Education
Physical Devices
Public Access
Visualization
Research Histories
Faculty Listed by Research
Project Highlights
Project Screenshots
Publications and TRs
Videos
Books
Products
Presentations
Studying HCI
Masters in HCI
PhD in HCI
Visiting Scholars
Class Websites
Sponsor our Research
Sponsor our Annual Symposium
Active Sponsorship
Industrial Visitors