As the popularity of digital photography has increased among the non-professional photographer, so has the need for effective ways to organize and manage those photos. This paper discusses a solution that we have implemented aimed at helping home users more easily browse and search their photo collections. We give a brief overview of the relevant research as well as describe a commercial product that partially, but not fully solves this problem. We will describe in detail our implementation and explain why we have made the choices that we did. Finally, we will discuss future work on this problem in general and for our solution in particular.
Storage of digital photographs by home computer users has been increasing in popularity over the past few years. There are two main reasons for this. First, with the decrease in price and increase in software and hardware support for digital cameras, digital photography has become more popular. Second, people are using scanners to digitize many of the photographs they have taken with conventional cameras. This presents an interesting situation for the digital photograph user. Is it possible for them to treat digital pictures similar to the ways that they would treat paper pictures? We would guess that a user would want to have the "good" parts of paper picture handling carry over and have the "bad" parts improved when they move to using computer software for handling digital photographs.
The "good" parts of handling paper pictures are that you can group the pictures into collections in various ways and then remember where individual pictures are based on the collections. The "bad" parts of paper pictures include the time it takes to shuffle through all the different pictures, as well as typically a poor way to organize and search for collections. This paper assumes the existence of good software for creating personal digital libraries of photo collections. We discuss the development of a software tool that attempts to improve the tasks of searching and browsing through this library of collections. We will review the relevant literature as well as discuss some commercial solutions to the problem and their deficiencies. Then we will more completely define the problem and present our solution to it. Finally, we will discuss future work in this area.
In general there has been very little research in the specific area of photo collection searching and browsing that we are exploring. The topics of annotating, searching, and browsing through a single set of photos have been discussed in numerous papers and addressed by many commercial software products. We will extract relevant parts of these papers and products to address the topic of photo collection browsing and searching.
Research has shown that one of the most common features desired by users is to be able to find a particular photo or photos from their library [Rodden99]. There are many different approaches that can be taken when searching for a photo or photos. One can search for photos by keyword using a simple textual query, or query based on image type. But, the main problem with specific querying devices is that the user is not given the ability to browse the photos. It has been shown that home users tend to retrieve information in a different way that professionals would. In general, home users tend to do less of the directed search activities and more browsing and riffling through collections of information [Kuchinsky99]. So for a photo collection system to be useful for home users, it must provide an interface that is useful for browsing, and not just searching.
In order to support browsing of a library, the objects in the library must be pre-arranged in some manner. In particular, "if the documents within the library are organized in some consistent, natural, comprehensible, orderly structuring method, then interfaces using a browsing approach are also applicable" [North96]. We believe that if photo collections can be organized in some way, such as by date or by subject matter, then they would lend themselves to browsing as well as searching. In both cases, users can make use of the structure of the organization to find what they were looking for, either explicitly by direct search or implicitly by browsing.
In another paper, an interesting discussion on the benefits of previews and overviews when browsing a large information space is presented. The basis of their argument is that browsing and searching are two very different activities. Browsing is a very interactive activity whereas searching is much more automated or batched. Therefore when browsing, one is limited by the human cognitive abilities. For this reason, overviews are presented as a beneficial tool to aid the browsing activity. This is true because, "an effective overview provides users with an immediate appreciation for the size and extent of the collection of objects the overview represents, how the objects in the collection relate to each other, and, importantly, what kinds of objects are not in the collection" [Greene97]. We believe that presenting an overview of all of the collections in a library organized by date would meet this goal.
In a paper dealing with thumbnail images and mouse-over text, Czerwinski et al, discuss qualities of human cognition that can be taken advantage of when browsing images, in this case thumbnail images of web pages. They note that people tend to remember spatially where things are located. They also note the benefits of a brief title (mouse-over text), the ability to group information, and the benefits of thumbnail images for remembering where pages are located and their content [Czerwinski99]. Because of this, we wanted to present our collection overview in a consistent, regular structure. This will allow users to always find collections in the same place. We also wanted to use thumbnails of a representative photo from each collection, and a mouse-over text displaying the title of the collection to serve as visual cues for each collection.
In addition to the research done on this topic, there are many commercial software packages that have attempted to provide a solution to the problem of managing photo collections for the home-user. In our examination we don't believe that any of the software packages provide a good solution for searching and browsing a library of collections. One of the most powerful of these packages is PhotoSuite. PhotoSuite allows users to annotate photos with metadata across a number of facets (e.g. category, place, camera lens). Users can then search for photos based on these facets. PhotoSuite also provides two tools for organizing photos into collections: an album tool and a slide show tool. Collections created with these tools cannot be annotated, and are searchable only by file name or creation date. There are a number of problems with these features. Searching is limited to single facets at a time, there is no support for annotating or searching at the collection level, and browsing an overview of collections is not supported. We believe that collection-level annotation is necessary to support collection-level searching and browsing. Further, we believe that users should have the ability to search and browse across multiple annotation facets at the same time.
Researchers in the University of Maryland Human Computer Interaction Laboratory Photo Library Project (HCIL Photo Library Project) have been studying various ways of inputting and cataloging metadata about individual photos and collections of photos. They believe that people are likely to want to organize groups of photos into collections based on some combination of when the photos were taken, who or what appears in them, and where they were taken. Such details are more personally meaningful and likely to be remembered in the future than file names or creation dates. If users annotate a collection of photos with these details when they create the collection, they can later search their library of collections for particular photos based on the annotation facets. Because the libraries are personal, users are likely to recall at least some details about the date, location, and/or content of the photos they want to find.
The goals of our project are to help users with the tasks of browsing their personal photo libraries and completing the first step of finding particular photos in their library by identifying which collection or collections might contain these photos. Our target audience is home computer users creating personal photo libraries who may or may not have much computer experience. We want to provide them with a tool that allows them to search across any or all of the annotation facets of a library of collections depending on what they remember about a search target, but without having to understand how to compose a complicated boolean query. We want the searches to be incremental and reversible so that users do not get lost when composing a search and can undo previous steps. We want the searches to take advantage of the visual representation of photos and to use other visual cues and feedback to offload some of the cognitive load of searching onto the vast resources of the human perceptual system. We want to provide an overview of the entire collection space to facilitate browsing, and tightly couple search actions with highlighted results in the collection space. In sum, we want to use the proven techniques of overview, zoom and filter, and details-on-demand visual search techniques by building our tool as an interactive information visualization.
We chose to implement our tool as a Java application using Java 1.2 so we could make use of the user interface widgets in the Swing development kit. We used a Microsoft Access database to store information about collections and photos and queried it in SQL via the JDBC-ODBC driver bridge. The database was local to our own machines because our facilities do not support an Access database server for remote connections. The photo thumbnails were also stored locally, but at paths relative to the database directory. However, our solution is fully generalizable to use remote photo thumbnails and databases of any type provided they are on a server accessible with a JDBC driver.
We began with the assumption that users would have a set of collections in their personal photo library annotated with the attributes identified by the HCIL Photo Library Group. These data are stored in a table in a database. The attributes include:
The title is a short summary of the contents of the collection while the description is a longer, free-text explanation of the collection. The location describes the city, state, and country of the photos in the collection, if known. The starting and ending date indicate the earliest and latest dates when the photos in the collection were taken, if known. Finally, the representative photo is a photo in the collection that the user picks to provide a good visual reminder of the contents of the collection. In addition to the metadata associated with a collection, we also assumed that users might annotate individual photos with similar attributes, including, but not limited to, the people in the photo. Collections were associated with the individual photos they contained via a table linking collection identification numbers with photo identification numbers. The people in a photo were associated with the photo via a table linking photo identification numbers with person identification numbers. For the prototype tool we developed, we used an existing HCIL Microsoft Access database containing 17 collections occurring in 11 different years from 1970 through 1998.
Users who create many collections over time are the most likely to benefit from a collection searching and browsing tool because of the large number of photos involved and the details forgotten when trying to retrieve a photo from an old collection. As a result, we focused on time as the primary facet for designing our tool. We wanted to present the entire set of collections over time to provide a good overview. We also wanted to fit as many of the collections in a single screen window as possible without having to scroll, with the constraint that a small but still identifiable thumbnail of its representative photo would identify each collection. We believe that users are likely to have collections organized at least by year, and perhaps by month, season, or event. We chose to arrange the collections in a calendar grid where each row was a year in the span of years covered by the library and each column was a month. Each row was labeled with the corresponding year and each column was labeled with the name of a month, both in ascending order.
This design allows for a fixed width window with no horizontal scrolling due to the fixed number of months in a year (12 plus 1 for collections with an unknown date). The size of the thumbnails meant that the grid cells would have to be roughly square. The horizontal width of the window was limited to roughly 1000 pixels to fit on the average computer monitor. This constraint resulted in grid cells of 70x70 pixels. With the addition of some searching widgets at the top of the window to be described below, this allowed us to fit roughly 10 years worth of photo collections on a screen, with subsequent years reachable by vertical scrolling. An alternative design might scale the grid to always fit on one screen according to the number of years in the photo library. However, we felt that this design would distort the thumbnail images too much for libraries spanning many years because of the small number of month columns relative to the number of year columns.
Each collection is placed in a year/month grid cell according to its starting date. Collections for which the starting month is not known are placed in an unknown column after the 12 month columns, and those for which the starting year are not known are placed in an unknown row after all the known years. We chose not to make use of the ending date attribute. Alternative designs might place a collection in every grid cell covered by the time span between the starting and ending date, or place a collection in the center of its time span with visual cues to indicate the beginning and end of the collection span. We chose not to explore these designs due to the complexity of managing multiple collections in the same grid cell.
The representative thumbnail takes up the majority of a grid cell. A textual count of the number of collections in the cell appears below the thumbnail so that users can see how many collections are in each cell. In cells with more than one collection, the small size of the cell only allowed for the display of a single thumbnail. We provided small arrow buttons on either side of the count text to page through the representative photos in each collection. The count text indicates the current collection number being displayed as well as the total collection count, and the left and right arrow buttons become disabled when the beginning or end of the collection list is reached. This design allocates as much space as possible to the visual representation of the photo to take advantage of users' perceptual skills in identifying collection contents with representative photos. However, the space taken up by text and arrows provides valuable information about and easy access to all the collections that appear in a grid cell.
Although it is not necessary for a month to appear in the grid if there are no collections in that month in any year, we have chosen to display all months to maintain a consistent grid size for libraries as collections are added or deleted. Research has shown that even without visual thumbnail cues, the spatial consistency of blank squares representing thumbnails helps users when searching [Czerwinski99]. Similarly, a year need not appear in the grid if if there are no collections in that year. We chose not to display rows for such years. This choice can result in a non-continuous list of year rows, which might be confusing to users, but also provides a huge savings of space for libraries with sparsely distributed collections over many years. An alternative choice might be to display a very thin row for each year in which no collections appear. This choice would maintain a continuous list of year rows at the expense of wasted space and irregularly sized grid elements.When the application is launched, a number of initialization steps must take place to properly display the grid and search widgets. First, the database is queried to find all of the people and locations that appear in the collections. We didn't want users to have to type in names of people and places when searching to avoid problems with name recollection, misspellings, and typos. Instead, we created pulldown lists of people and places to select from when searching. The people pulldown list is multi-selectable so that users can query for multiple people found in the same collection. The list is sorted in alphabetical order by last name. We assume that a collection only occurs in one location due to the design of the database, so the places pulldown is single selectable. The list is sorted in alphabetical order. However, the places pulldown is currently built from whatever a user has entered for the location attribute of a collection. Thus, it is possible that a location might be two or more places (e.g. "College Park, MD and Annapolis, MD") if a user creates a collection that takes place in two or more places. In this case, "College Park, MD and Annapolis, MD" will appear as a single choice in the pulldown.
The number of month columns in the grid is fixed at 13, but the number of year rows depends on the set of collections being viewed. We query the collection database for the number of distinct starting years and their values to determine the number of rows in the grid and their values. We then initialize the row and column labels accordingly and build the grid as a two dimensional array of fixed size. Each row and column label is built as a toggle button that can be set on or off to indicate whether or not the associated year or month should be included in the current search. For each collection in the database, we query the collection, photo, and linkage tables to create a collection object that holds the location, people, representative photo path, collection name, starting month, starting year, ending month, ending year, collection identification number, and collection description. Our application only makes use of the first six attributes right now, but future work might make use of other attributes. Each of these attributes except for the collection name also has a boolean value associated with it to indicate whether or not the attribute matches the search. All attributes are initialized to false to begin with. Each collection is then added to the appropriate grid cell based on its starting month and year.
The entire application is then drawn with the people and place pulldown menus at the top of the window and the grid below. Each grid cell with one or more collections is drawn with a representative photo thumbnail and collection count. The thumbnail is displayed as a square icon on a button. The button may be used for a future feature that will allow users to click on a collection image to view all the images in the collection. We chose to make all the thumbnail icons square, which slightly distorts rectangular photographs of either horizontal or vertical orientation. This preserves a regular grid structure at the expense of making some images slightly harder to see. An alternative choice would be to determine the orientation and shape of each representative photo thumbnail and scale it accordingly for display. At any time, a user can mouse over the thumbnail with the cursor to see a tool-tip that shows the title of the collection followed by the number of photos in the collection in parentheses. Left and right paging arrows are added to the element with more than one collection.
In Figure 1, the entire application is shown. The grid cell highlighted in white with the color photo contains a single collection. The text below the photo reads "1 of 1" to indicate that this is the first and only collection in the cell. In Figure 2, the cell highlighted in white contains more than one collection, so the arrow buttons appear and the text below the photo reads "2 of 3" to indicate that this is the second of three collections in the cell. In both figures, the cursor has been placed over the image in the highlighted cell showing the title of the collection represented by the photo. The default state of the application has no search criteria entered, so none of the collections in the library are highlighted as matching. This state is indicated in the widgets by setting all month and year buttons to the off state and setting the pulldown menus to their default values of "No People" and "No Places". This state is indicated in the grid by setting all grid element backgrounds to gray and disabling all of the thumbnail image buttons, which sets their background to gray. Figure 3 shows this default state. We added two buttons above the pulldown menus called "Reset Off" and "Reset On" to allow users at any time to put the application into this initial state or a state where all collections are selected. When the latter button is clicked, all year and month toggle buttons are set on, indicated by a depressed state and color change. The pulldown menus jump to values of "Any Person" and "Any Place". All grid elements containing collections are highlighted and all representative thumbnails are re-enabled. Figure 4 shows this state.
As stated earlier, the initial state of the application has no collections selected. To find a particular collection, users specify the year, month, people, and places they think specify the collection. Years and months are specified by clicking on the toggle buttons for rows and columns. People are specified by selecting one or more names from the people pulldown. Places are specified by selecting a location from the places pulldown. Any collections matching the search criteria will be highlighted in the grid. The search structure supports a limited form of boolean queries. Searches are disjunctive across year/month pairs, but strictly conjunctive otherwise. That is, for a collection to match a search, it must be in a grid element where the year and month are selected, contain all the people selected in the people pulldown, and take place in the location selected in the place pulldown. For example, in Figure 1, we show a search for collections in June of 1992 containing Ben Shneiderman that were taken in College Park, Maryland.
Although the search structure is fairly restrictive in requiring conjunctive matches, it can be relaxed to ignore one or more search criteria. Ranges of years and months can be selected by toggling multiple rows and columns to allow disjunctive search across multiple time periods. To search across all years or months, a user can toggle all the row or column buttons on. To search across all places, the user can select "Any Place" from the location pulldown. To search across all people, the user can select "Any Person" from the people pulldown. Finally, to select all collections, the user can click the "Reset On" button at the top of the window. Individual grid cells cannot be clicked on to include or exclude the collections in a cell in the current search. To do this would require all of the search widgets to be updated to reflect the people, place, month, and year in the cell. We felt that this would be confusing to the user because the values in the people and place pulldown menus would change without their knowing.
A grid cell with multiple collections is highlighted whenever at least one collection in the cell matches the search. However, not all collections in the cell may match. To indicate that a cell has at least one matching collection, the cell is highlighted. To indicate that the current collection displayed in the cell is one that matches the search, the thumbnail representative photo appears normal. For example, in Figure 5, the cell for January of 1991 contains a collection that matches the current search, so it is highlighted. The cell contains multiple collections, as indicated by the paging arrows. The current collection displayed matches the search criteria because its photo thumbnail is normal. To indicate that the current collection displayed does not match the search the representative photo is grayed out. In Figure 6 the same search is shown as in Figure 5. This time, the user has paged to a collection in the cell that does not match the search criteria. Its photo thumbnail is therefore grayed. Thus, a user can page through the collections in a highlighted cell to see exactly which collections match the search criteria.
This search structure has a number of advantages. First, it is fast because no typing is required. Everything can be done by using the mouse. Second, it is easy to specify searches because all and only the relevant criteria are presented on the screen. Users can see exactly what their choices are and these choices are guaranteed to match some collection. Third, it is easy to understand because it is rapid, incremental and provides immediate feedback. Each criterion is specified with a separate, independent widget that visibly and immediately indicates its state and the state of the grid when clicked. Changing one widget has no effect on any other widget. Changes to the grid are clearly indicated with colored highlighting. Finally, it is easy to control because all actions are reversible. Years and months are toggled on and off with a single click for immediate reversibility. The place and people lists can always be set back to their previous selections, provided the user remembers what they were. If not, the "Any Person" or "Any Place" options can be set to search across all people and places.
User studies are needed to test the utility of the tool and the quality of the interface. We would like to better understand which types and sizes of collection libraries are best suited for searching and browsing with this tool. To do so, we would like to have users attempt a variety of tasks using a variety of libraries. We would also like to see if the semantics of the widgets are comprehensible to novice users and powerful enough for more advanced users. If users indicate that the gaps between years caused when no collections appear in certain years is confusing, we would like to add these years back in to the window as very thin rows to provide context without wasting too much space. When users scroll the application window to see collections in later years, the pulldown menus, month toggle buttons, and reset buttons can disappear from view. If this proves confusing, we might consider placing these components in a separate frame and only allowing the grid to scroll. This would allow these components to always be visible.
There are a number of enhancements and improvements that could be made to this tool in the future. For collections annotated with text descriptions, it would be nice to support a free-text search over these descriptions. This search might have to be disjunctive from the rest of the search criteria since the descriptions would likely contain dates, people and places. We would like to be able to connect our application to other tools that support the final step in the photo search task of finding a particular photo within a collection. Ideally, we would like to be able to click on a selected collection to open a window of manipulable, searchable thumbnails representing the photos in the collection. This should be relatively straightforward if the other tool is using the same database because our application could simply pass the collection identification number to the other application.
Our grid structure is fixed to be organized by month and year, which we believe will be useful for many collection libraries. However, some libraries might be organized at a higher level of detail such as by season, or at a lower level, such as by day. It might be useful to extend this application to either pre-process libraries to determine the best presentation structure, or provide widgets to zoom in or out over the time granularity. Two simple improvements that could be made would be to add widgets to allow users to automatically select the sets of three months corresponding to the four seasons and to add widgets to allow users to indicate certain days or ranges of days to look for in a collection.
An individual grid cell provides a great deal of information: the number of collections in that month and year, a representative photo for each collection, whether or not any collections in the element match the current query, and exactly which collections match the query. While the textual information that shows the number of collections in the cell is useful, it might be better to have a more noticable visual cue that indicates this value, such as a different colored cell background for certain count ranges. The tool tip for a cell indicates the title of the collection and the number of photos it contains, but users have to actually mouse-over the cell to get this information. We tried adding a small, vertical "progress bar" next to the photo thumbnail that was filled to indicate the relative number of photos in the collection compared to other collections in the library. However, we found that this bar was often wasted when most collections had relatively few photos when compared with the number of photos in the largest collection. Another solution would be to adjust the size of the representative thumbnail photograph to indicate the size of the collection. More research would be needed to determine how small the thumbnail could be made to still be recognizable for this feature.
Our database did not have any unknown collection starting months and years, so we did not have to use the unknown month column or year row. We included the unknown month column anyway, but not the unknown year row. We might consider eliminating the unknown month column in the future if there are no unknown months. We did have unknown first and last names for people, and cities and/or states for places. We simply presented these in the pulldowns as they appeared in the database with "null" or "??" values. This presentation could be cleaned up a bit, but even the partial information is still useful for searching. Our tool is somewhat limited in the types of boolean searches allowed. For advanced users who understand the semantics of full boolean searches, a separate search tool that supported more complex searches might be useful. However, for the average home user who does not understand the boolean semantics of "and" and "or", searches using these operators have been shown to be confusing in the past. We opted in favor of simplicity at the expense of some power. We believe the semantics of the year and month buttons is fairly clear, but user testing is needed to determine if the people and place pulldowns make sense. Each time a new year or month is selected, the new choice is "or"-ed with the previous selections. However, each time a change is made to one of the pulldown menus, the previous selection is forgotten and the collections are re-tested using only the newly selected criteria. Users might expect a different semantics where their previous choice was "and"-ed or "or"-ed with the new choice.
Finally, there are some space management issues that might be improved in the future by other tools. Our application does not make efficient use of space for collections widely distributed over years and months. It is also not useful for collections that are highly concentrated in a certain time periods. In the first case, there is a large amount of empty space. In the second case, there might also be empty space, but there will also be many collections in certain grid elements that require a lot of paging. It might be useful to have an application examine the dispersion of collections and present them with our tool if the collections are fairly evenly distributed and another tool if they are more dispersed. Alternatively, the application could examine the dispersion of the collections and adjust the month and possibly day ranges in the display to better accommodate the collections. Improvements to our tool or other tools could also be made to indicate collections that span months and years in a coherent way without cluttering up the display.
As use of digital cameras for personal photography continues to grow, good software for organizing, cataloging, searching, and browsing photos will become more important. Recent research and commercial software development has focused on managing an individual collection of photos, but little work has been directed towards managing a library of these collections. Over the course of a lifetime, digital camera users may create many collections in their library. We have created an information visualization tool that allows users to browse and search such a library. Browsing is supported via an overview of all the collections in the library organized by date. Searching is supported with simple widgets for specifying the values of various facets that characterize the collection. For users who want to find a particular photo in their library, this tool can be used to identify which collection contains the photo. Other tools can then be used to search or browse the relevant collection.