Personal Media Exploration with Semantic Regions


Hyunmo Kang

Department of Computer Science and Human-Computer Interaction Lab
University of Maryland, College Park, MD 20742 USA


Computer users deal with large amount of personal media data and they often face problems in managing and exploring them. The paper presents an innovative approach, Semantic Regions that are rectangular regions directly drawn on 2D space with semantics so that their layout can form users’ various mental models toward the personal media data. A prototype personal media exploring application, MediaFinder, based on the concept of Semantic Regions is presented. Usability tests will be conducted to evaluate the Semantic Regions as a personal media management model including organization, search, navigation, indexing, meaning extraction, and distribution.


Personal Media Management, Graphical User Interface, Spatial Information Organization, Fling-and-flock


Computer users deal with large amount of personal media data such as images, audio clips, videos, web pages, emails, and various document files. It is often difficult for users to explore and manage them due to three major problems: increasing amount of data, rigid organizing metaphor, and difficulty in rapid data access. To tackle those problems, this paper presents an innovative approach, Semantic Regions, based on the hypothesis that “Spatial organization of information based on the semantics of personal media” and “Use of various users’ mental models for managing the personal media” will greatly improve task performance.


With the extraordinary progress in all of the “cyberization” technologies [1], there have been a number of research efforts about Personal Information Storage [1][5][6] and Retrieval [2][3][4]. A variety of approaches have been suggested for supporting more flexible document management against the file system approach. LifeStreams [3] uses a timeline as the major organizational metaphor for managing document spaces, but it just replaces one superordinate aspect of the document with another. The Semantic File System [4] introduces the concept of a “virtual directory”. The system is directly integrated into a file system and interprets virtual directory names as queries, but no interface features are introduced other than the filename/query language syntax. Presto [2] is a prototype document management system that interacts with documents through user-level document attributes. The 2D graphical interface is mainly used for querying but unable to visualize high-level attributes in a meaningful way.


Semantic Regions are rectangular regions drawn directly on a two-dimensional space. Users can specify the shapes, sizes, and positions of the regions in two-dimensional space and thus form the layout of the regions meaningful to them (Figure 1). Creation of Semantic Regions is straightforward: users simply click on the desired starting point of the semantic region and drag the pointer to the desired location of the opposite corner. Once the semantic region is created the semantics of each region can be specified with the combination of the attributes of personal media objects.

When users drag a collection of the personal media objects onto the Semantic Regions, the objects are automatically distributed and placed in the appropriate regions based on the semantics defined in each region. This metaphor is called fling-and-flock.




Figure 1. Semantic Regions Examples: (a) Map (b) Critical Path (c) Organization Chart (d) Monthly Calendar


The MediaFinder is a prototype interactive tool built to investigate the use of Semantic Regions for exploring and managing personal media data. The initial prototype has been built on top of PhotoFinder [7] to manage photos. In MediaFinder, users create and arrange the rectangular regions on 2D space and specify the semantics. In figure 2, each region represents a person, and the regions are grouped into 5 clusters to represent different friend groups. When photos are dragged onto the regions, they are automatically placed in the appropriate regions based on the annotations. This metaphor is called fling-and-flock; that is, users fling the objects and the objects flock to the regions. If photos don’t satisfy any of the semantics of regions, they are collected in the remaining items region located at the top of the panel. Users can save the current state of regions to apply it to other personal media data sets later.

A photo can be contained in multiple regions as long as it satisfies the semantics of regions. When the mouse cursor is over a photo (yellow circle), all the regions containing it are highlighted red, and when mouse is over the title of a region, all the regions containing any of photos in the region are highlighted. With this feature, social relations among the different friend groups can be easily visualized in figure 2.

In MediaFinder, users can combine a few different mental models simultaneously and observe the relationships among them. For example, three mental models such as US map, year calendar, and friends group can be combined to answer the queries such as “When and where did I meet this friend ?”, “Which states did I visit during the winter of 2001 and whom did I meet then ?”, “When did I visit the east coat states and whom did I meet there ?”, and so on.

MediaFinder supports the boolean operations of the regions to facilitate search and navigation. For example, to see all the photos of a friend group in figure 2, OR operation can be applied to the regions in a cluster. A new region is created on top of the selected regions with all the photos in the group. Users can see only the photos taken in MD by applying AND operation between the newly created region and MD region in US map. In MediaFinder, users can also annotate photos by dragging them onto the specific region.


Figure 2. MediaFinder prototype implemented on top of PhotoFinder. Each region represents a person and contains all the photos annotated with the name defined in it.


The MediaFinder prototype provides an encouraging proof of concept for Semantic Regions model, but more examples of application will be necessary to demonstrate the utility of this approach. Ongoing and future works are as follows.

·         Semantic Regions Extension: MediaFinder will accommodate more extended and heterogeneous personal media such as music files, video clips, email, web pages, and various document files, and it will lead to a design of metadata allocation table for personal media management system like the file allocation table for file system. The concept of Semantic Regions will be generalized to test whether it can provide more intuitive and understandable visual queries for general database systems. How to increase the expressive power without increasing the complexity of interface will also be a major issue here.

·         Software Architecture, Algorithms, and Data Structures: Users will handle tens of millions personal media in near future, and thus the performance will become a critical issue. How to make algorithms faster than O(mn) (m: number of personal media objects, n: number of regions in MediaFinder), what kind of data structures are needed for partial updates of regions, how to optimize the sequence of boolean operations to enhance the performance will be the major issues in this area.

·         Evaluation: Two usability tests have been planned. The first test is to demonstrate whether users really can use the Semantic Regions for managing the personal media. A few sample Semantic Regions templates will be given to users and their usability including organization, search, meaning extraction, navigation, and indexing will be observed. After revising MediaFinder based on the result of the first test, the usability of constructing Semantic Regions based on users’ various mental models will be investigated.


I would like to offer special thanks to my advisor, Ben Shneiderman, for his guidance and encouragement.


1.     Bell, C. G., The Cyber All Project: A Personal Store for Everything, Microsoft Research Technical Report, July, 2000.

2.     Dourish, P., Edwards, W. K., LaMarca, A., Salisbury, M., Presto: An Experimental Architecture for Fluid Interactive Document Spaces, ACM Transactions on Computer-Human Interaction, 6 (2), 1999.

3.     Fertig, S., Freeman, E., Gelernter, D., Lifestreams: An Alternative to the Desktop Metaphor, Proceedings of ACM SIGCHI96 Human Factors in Computing Systems, 1996.

4.     Gifford, D.K., Jouvelot , P., Sheldon, M.A., O'Toole, Jr., J.W., Semantic File Systems, 13th ACM Symposium on Operating Systems Principles, 1991.

5.     Lamming, M., M., F., Forget-me-not: intimate computing in support of human memory, FRIEND21, ’94, International Symposium on Next Generation Human Interface, 1994.

6.     Mayo, R.N., Reprint of the Factoid Webpage, Technical Note TN-60, Compaq Western Research Laboratory, July, 2001.

7.     Shneiderman, B., Kang, H., Direct Annotation: A Drag-and-Drop Strategy for Labeling Photos, Proceedings of International Conference Information Visualisation, 2000.


Web Accessibility