User Controlled Overviews of an Image Library: A Case Study of the Visible Human

Chris North*, Ben Shneiderman*^, and Catherine Plaisant

Human-Computer Interaction Laboratory,
*Department of Computer Science,
^Institute for Systems Research

University of Maryland, College Park, MD 20742 USA
+1-301-405-2725
{north, ben, plaisant}@cs.umd.edu

Proc. ACM Digital Libraries '96 Conf, ACM Press, 1996.
© Copyright 1996 by ACM, Inc.
(Also available as CS-TR-3550, HCIL-TR-95-20.)

ABSTRACT

This paper proposes a user interface for remote access of the National Library of Medicine's Visible Human digital image library. Users can visualize the library, browse contents, locate data of interest, and retrieve desired images. The interface presents a pair of tightly coupled views into the library data. The overview image provides a global view of the overall search space, and the preview image provides details about high resolution images available for retrieval. To explore, the user sweeps the views through the search space and receives smooth, rapid, visual feedback of contents. Desired images are automatically downloaded over the internet from the library. Library contents are indexed by meta-data consisting of automatically generated miniature visuals. The interface software is completely functional and freely available for public use, at: http://www.cs.umd.edu/projects/hcil/Research/1995/vhe.html.

Keywords

Browsing, Digital Library, Image Database, Information Exploration, Information Retrieval, Internet, Medical Image, Remote Access, User Interface, Visualization, World-Wide Web.

INTRODUCTION

The Visible Human

The National Library of Medicine (NLM), for its Visible Human Project [NLM90], is in the process of creating a large digital library of anatomical images of the human body. It contains MRI and CT scans, as well as cryosection images (digital color photographs of cross-sections, Figure 1). Two cadavers, one male and one female, were carefully chosen as the subjects. The male dataset MRI and CT images were captured using the respective medical imaging scanners. The MRI images were captured at 4mm intervals throughout the body. Each is 256x256 pixels in resolution with 12-bit gray level encoding. The CT images were captured at 1mm intervals to correspond with the cryosections. Each is 512x512 pixels, also using 12-bit gray level. To capture the cryosection images, the cadaver was first frozen solid inside a large block of blue gel. Then, 1mm thick slices were successively cut away from an axial cross-section (planar cut perpendicular to the longitudinal axis of the body), and digital color images were taken of each newly exposed cross-section. A total of 1878 cryosection images were taken, spanning the body from head to toe. Each is 24-bit color and has a resolution of 2048x1216 pixels. The total result is a 15 Gigabyte image dataset of the male body. The female dataset will be captured at 1/3rd mm slices and, thus, is expected to require three times the storage space of the male dataset. In addition, 70mm film of cryosections may be scanned at higher resolutions for details of various anatomical structures.

[SnapShot] Figure 1: A Visible Human cryosection image of the male's head. Library cryosection images are full color and have 2048x1216 pixel resolution, larger than most CRTs.
Click on this image to retrieve the full-size image (275K GIF).

The NLM provides public access to the Visible Human digital library via the internet. The entire dataset resides on a high capacity storage system with a high speed internet connection. A simple license agreement with NLM, intended for tracking data usage, allows any user with an internet capable computer to remotely access the full dataset. Each image is stored on the archive as a separate file with numerically indexed filenames ordered from head to toe. Using "ftp", a user remotely logs onto the archive machine with a special login ID and password provided with the license agreement, and downloads any number of image files by specifying the correct numerical filenames. The images can then be displayed locally using an appropriate image browsing software package. Also, several sample images are available to those without a license agreement on the Visible Human World Wide Web page at http://www.nlm.nih.gov/research/visible/visible_human.html.

Because the library is available to the public, the user community is diverse. Users are of varying backgrounds and cover a wide range of expertise in human anatomy and computer skills. The most prominent user groups are medical professionals and students interested in using the images for education and modeling purposes, and computer science researchers and software developers creating medical imaging applications. Other users include elementary school teachers, curious internet surfers, and even martial arts experts! Some users wish to simply explore the dataset in an open-ended fashion, and download a sampling of images to browse. Other users, who desire a specific portion of the dataset, use a more directed search to locate and download images. Some desire only a single image or small set of images, perhaps to use as overheads while teaching an anatomy class. Others download an entire continuous block of images representing some portion of the body. In this case, the images are often used for 3D modeling of anatomical structures. For example, a medical expert might download all the cryosection images containing the heart, and load them into advanced software, such as Mayo Foundation's ANALYZE(tm), to visualize ventricle chambers. Very few attempt to download the entire library, since it would require weeks of continuous internet activity as well as overwhelming storage capacity.

Unfortunately, using the Visible Human digital library can be difficult. The ftp interface merely presents users with a long list of sequential image filenames. Browsing the data requires a trial-and-error process of guessing a file, downloading it (which could take several minutes), and displaying the image locally. A Web page interface that displays thumbnails of the library images would allow the user to visually browse the contents of the library, but would require paging through hundreds of thumbnails that must be downloaded for each access.

Whether a user wants to simply browse the library or retrieve images for some other use, it is important to avoid unnecessary downloads. Because the dataset is extremely large and would take weeks to download, and it is desired that users with smaller computers with limited storage capacity have access, the ability to remotely explore a visual representation of the dataset and download only desired images is a requirement. Hence, there is a great need for an effective user interface to assist users in browsing the library and retrieving images.

Related Work

As digital libraries quickly increase in number, researchers are continually developing improved public access methods for the many different types of library information. The primary user interface approach, especially for libraries containing unstructured collections of heterogeneous documents, is the keyword query method. In this case, Information Retrieval systems preprocess documents in the library, usually using automated algorithms, to generate comprehensive indices (meta-data). Then, when a user enters a query, the system can quickly generate a list of matching documents. Since many of these libraries contain textual documents only, such as computer science technical reports [Van94], and use strictly textual queries, interfaces are often text-based for simple WWW or teletype access. More advanced GUI interfaces afford additional capability to manipulate and visualize the textual query [VN95,YS92]. Image libraries may also provide for query by image content [FBF94]. In this case, images are indexed by content features and, if available, textual annotations. These interfaces allow users to specify example images, interactively sketch objects, or use an image content description language to create a query. KMeD [CCT95] is an example of a multimedia library in the medical domain which combines both textual and medical image elements. Digital video libraries expand the query method further, using speech recognition on the audio tracks of videos for textual indexing [GAE94]. While the query method may be an effective means for retrieving documents based on keywords or image features, it does not provide the capability to "explore" or build a mental model of the contents of a library.

However, if the documents within a library are organized using some consistent, natural, comprehensible, orderly structuring method, then interfaces using a browsing approach are also applicable. Advanced user interface techniques for information exploration can be used to explore the library structure, usually specified by meta-data, and its contents. This gives users a way to learn about the library and locate interesting documents without relying on keyword search. Several common structuring methods exist [Lev95]. For example, Hypertext systems such as the WWW organize documents in a hierarchy or other graph representation. With an interface for browsing graph structured information (see [Kor95] for a review of several), users can utilize the links, perhaps representing relationships between documents, to discover related documents that might not be found using a keyword query. Implementations include utilizing the Dewey Classification hierarchy to browse a library catalogue [All94] and reference citations to browse articles [RPH95]. MeSHbrowse [KS95] is an interface for browsing a medical terminology hierarchy containing arbitrary inter-relationships (meta-data) as an index to a medical image digital library. Another structuring method organizes information by defined textual and numerical attributes. Doan applies dynamic query interface techniques to arbitrary distributed information of this form, allowing users to quickly learn overall contents, find trends, and locate documents of interest [DPS95]. Browsing computer science literature has been handled by attribute structure as well, using author names, publication dates and sizes, etc. [HHN95]. Some interfaces use the map motif to browse spatial information, in which case attributes represent geographical locations [KJ95, Pla93]. All of these systems are examples of how advanced information-exploration user interface techniques can capitalize on the structure of an organized library to add a new dimension, namely the capability to browse, to the user's tool set for the task of information retrieval.

The Visible Human digital library exemplifies another organizational method for structuring libraries. The medical imagery, although un-annotated, is concretely structured. That is, the images have a strict sequential ordering defined by the physical objects (the human body) represented. Hence, because of the previously described successes, we expect that a user interface, which utilizes this structure to provide a browsing environment, would be an effective means for users to retrieve desired images.

Guiding principles for designing a user interface for the Visible Human library come from related research of browsing techniques. Firstly, direct manipulation, with compelling success in many exploration interfaces, provides these principles: [Shn92]

Research in digital medical atlases has developed reconstruction and visualization techniques for annotated, segmented, 3D medical image data [HPR95]. Interface elements and strategies for browsing individual 2D images are thoroughly identified [PCS95]. In particular, a coordinated detail view and overview giving contextual feedback, as well as extraneous data download avoidance, are essential elements for remotely browsing very large images [CPH94]. Work on Query Previews [DPS95] shows the benefits of previewing downloadable data as a filtering process by issuing dynamic queries on meta-data. With this technique users drag sliders, click on options, and receive rapid visual feedback to identify desired data available for retrieval. With these principles as our motivation, we designed the interface described in the following section.

THE INTERFACE

We have developed a direct manipulation user interface, called the Visible Human Explorer (VHE), for remotely exploring the Visible Human digital library and retrieving images. It allows users to rapidly browse, on their own machine, a miniaturized version of the Visible Human dataset and, based on that exploration, download desired full resolution library images. The miniature dataset provides an overview of library content and acts as a preview mechanism for retrieval. The interface, along with the miniature dataset, is downloaded in advance and then used as an accessor to the archive. The payoff is quick, since downloading the VHE is equivalent to only about 3 image retrievals. With this paradigm, although implemented completely outside of any existing web browser, the interface can be thought of as an advanced, dynamic web-page application for browsing a medical image digital library.

The VHE direct manipulation interface (Figure 2) presents the user with a coordinated pair of orthogonal 2D cross-section views of the Visible Human body. The left view-window, or overview, displays a miniature coronal section, a front-view longitudinal cut of the body. It acts as an overview of the library, giving the user a general understanding of the contents of the body from head to toe. The images for this view were reconstructed directly from the cross-section images, taking advantage of the structural organization of the library, by sampling a single row of pixels from successive images to simulate a perpendicular cut. The resolution is decreased to approximately 150x470. The right view-window, or preview, displays a miniature axial section, the type of cut typically seen in medical atlas textbooks. The images for this view are reduced resolution thumbnail images (300x150) of the original library cross-sections and act as a preview of those higher resolution images. The magnification of this view is twice that of the overview, giving the user more detail of the library.

[SnapShot]
Figure 2: The Visible Human Explorer user interface, showing a reconstructed coronal section overview and an axial preview image of the upper abdominal region. Dragging sliders animates the cross-sections through the body.
Click on this image to retrieve the full-size image (120K GIF).

A horizontal indicator on the overview indicates the vertical position, on the body, of the axial section shown in the preview. The indicator is attached to a vertical slider widget spanning the height of the overview and can be dragged vertically across the body to sweep the cut, shown in the preview, through the body. As the user drags the indicator, the preview provides smooth, rapid feedback (< 100msec) reflecting the axial cross-section at the sliding cut plane, resulting in a dynamic animated effect of motion through the body. This gives the user the ability to easily explore the contents of the entire body. Likewise, a sliding indicator in the preview controls the view shown in the overview. For example, the user can slide the overview to show a coronal section nearer to the front or back of the body. (See video report [NK96] for a demonstration of the dynamic interaction.)

While exploring, when the user locates an axial section in the preview for which high resolution data is desired, pressing the Full-Size Image button opens the Retrieval dialog box (Figure 3). This dialog handles the details of accessing the Visible Human library archive. After users type their Visible Human login ID and password into the corresponding textboxes and press the Retrieve button, the system begins to download the requested full resolution cryosection image from the NLM digital library over the internet to the user's computer. Since retrieval may take several minutes, depending on internet traffic, a status meter provides feedback on the progress of the retrieval. Users have the option to cancel the downloading at any time. Once the meter reaches 100%, the download is complete and the image is then displayed in a large detail view.

[SnapShot]
Figure 3: The VHE Retrieval dialog box meters downloading from the Visible Human digital library.

Since downloading over the internet can be a time consuming process, the system does have provisions for users with computers containing ample storage space. It can accumulate retrieved high resolution images on the user's computer, thereby allowing the user to browse not only the remote library but also a local cache as well. A vertical bar aligned next to the overview image shows tic marks indicating which axial cryosections are contained in the local cache. When one of these cross-sections is previewed by placing the cut plane slider over a tic mark, a flag in the preview window highlights to indicate that the corresponding full-size image is in the cache. Clicking on the Display button will then display the image from the cache, saving the download time. Below the overview, a text box shows the name of the user selectable directory containing the local image cache. This feature is particularly useful for users who acquire their own copy of the Visible Human digital library, since they can use the VHE to browse it as well.

The VHE also provides the capability to download any user specified range of cross-section images into this cache at once. The user simply places range indicators next to the overview slider, and selects a button to begin retrieval using the Retrieval dialog box. Two smaller axial cross-section views display images of the top and bottom slices of the user specified range. Feedback on the number of images within the range and the total size, in megabytes, of those library images is also provided. This feature is useful for retrieving commonly examined portions of the body a priori, or for obtaining a continuous block of images for use in 3D modeling applications.

Also, using the icon buttons adjacent to the overview, the user can select amongst different overview image types, including the coronal section, sagital section (side view longitudinal section), and a simple front view of the body (no cut). This gives the user a variety of views of the overall search space. Menu options (Figure 4) enable selecting between the male or female subjects, and between the MRI, CT, and cryosection medical image types. Currently, only the male cryosection meta-data is loadable.

[SnapShot] [SnapShot]
Figure 4: VHE menus for selecting amongst datasets.

Implementation and Performance

The VHE is implemented in the UNIX X/Motif environment on a Sun Sparcstation. The key to implementation is the use of meta-data, consisting of the miniature Visible Human thumbnail images, which acts as the library index. The benefit is two-fold: Firstly, the meta-data provides a manageable dataset for interactively browsing the library. Secondly, each axial section thumbnail provides a preview of and an index to the corresponding full size library image for automated retrieval purposes.

For optimal browsing performance, the meta-data is stored locally on the user's machine and is preformatted in 8 bit XImage [Nye90] format. Hence, as a slider is dragged, new thumbnail images can be moved directly from disk to screen using X Windows shared memory. On an older Sparcstation 1+, images are updated at approximately 20 per second. As a result, we expect that an implementation on a PC or Mac with reasonable I/O capabilities could also yield acceptable interactive performance.

The meta-data is pre-generated and comes packaged with the VHE software. To create the meta-data, we developed several tools to automatically generate it from the library images. The thumbnail meta-data preformatted for the axial preview window is cropped, spatially sampled, and color quantized directly from library images, and requires under 20 Megabytes of storage space. The overview meta-data formatted as coronal images is spatially sampled and reconstructed from the axial meta-data, and requires under 5 Megabytes of space. The total cryosection meta-data is 1/600th of the total size of the full resolution cryosection library data.

Benefits

The VHE interface design was chosen to be in harmony with user tasks. The tightly coupled 2D views of the dataset, combined with rapid, dynamic user control of movement through the third dimension, provides a highly interactive interface yet avoids unnecessary complexities. The result is an elegant interface affording convenient user exploration of the image data, for both novice and expert users. The need for learning time is essentially eliminated. Feedback is fast enough to engage the user. Users can quickly learn about the entire library by sweeping the views through the body, absorbing 20 MB of data in just a few seconds. Then, slower smaller movements to carefully examine interesting portions of the library help locate optimal images for downloading. Also, when dragging a slider, visualizing the resulting motion of the structural patterns in the cross-section thumbnails provides additional insight over, for example, simply viewing thumbnails side by side or in a click-and-wait incremental fashion.

As has been discovered in exploration interfaces for many other types of data, a visual overview of the data space is extremely helpful [Shn94]. When users search for specific data, the VHE overview quickly guides them to the desired location. It provides context for the axial preview thumbnails, some of which would otherwise be difficult to interpret exactly. It also promotes user exploration of the library, by eliminating the penalty of the possibility of getting lost or not being able to return to a desired location in a timely fashion. The preview allows users to select images, filtering out unwanted information, and download and zoom only on desired detailed data on demand.

DISCUSSION

Orthoviewer

While developing the VHE interface, we also implemented another prototype, dubbed the Orthoviewer (Figure 5), that generalizes the VHE motif. It presents the user with all three orthogonal views (sagital, coronal, and axial) of the Visible Human body simultaneously. Each view contains both a vertical and horizontal indicator line which reflects the position, with respect to the view, of the cut planes of the other two views. Each of these six indicators can be dragged, as in the VHE, to slide one of the views through the body. Pairs of corresponding indicators are tightly coupled. For example, dragging the vertical indicator in the sagital section animates the coronal view and also slides the horizontal indicator in the axial section, which reflects the position of the same cutting plane. Interestingly, we discovered that the Orthoviewer interface confuses many users. The addition of a third view and four indicators makes it difficult to decipher which indicator manipulates which view. Users typically resort to a trial-and-error process to find the appropriate indicator to operate. Also, since each indicator is tightly coupled to 2 other components, when dragging an indicator, the user is distracted by the additional motion. The interface simply provides too much dynamic information for many users to process. Users' cognition is consumed by the Orthoviewer interface instead of the data itself. As a result, we designed the VHE interface with a distinct overview-preview pair, as used in many successful browsing interfaces in other domains [KPS95].

[SnapShot]
Figure 5: The Orthoviewer prototype interface shows three orthogonal views of the body. Dragging any of the six indicators animates corresponding views.

User Feedback

In July 1995, we publicly released the VHE system on our Web site, enabling users to download and use it to access the Visible Human library. Our access logs indicate that over 300 users, from all over the world, have downloaded the software, after just 2 months. As a result, we received some informal feedback about the interface and its usage from actual users. We received 34 responses. Of those who indicated their occupation or field of study, users were evenly distributed between medicine and computer science, and between professional and student. Others were simply curious internet users. The medical users identified their use of the library information being primarily for learning and teaching gross anatomy and as a reference for radiology. The computer scientists identified their primary usage as test data for image processing and computer graphics algorithms research. All of whom gave in-depth comments reported that the VHE interface was very helpful for visualizing the Visible Human library. One user stated: "Now I have a far better understanding of how the whole data-set looks." Users who need to download specific individual images were very pleased that, with the VHE interface, they were finally able to quickly locate and retrieve the proper library image. "Before using the VHE it was always a bit of a problem to get the right images." Short responses indicated the users' excitement with the interface's highly interactive nature, fast feedback, and quick learnability, trademarks of a direct manipulation interface. Also, some suggestions for improvement were offered. Multiple users suggested increasing the size of the preview image by a factor of 1.5 or 2, enabling visualization of most major organs. One user felt that simply displaying the existing preview images larger, without an increase in actual data resolution, would be sufficient. Several users were wary of the 25MB size of the meta-data. Another suggestion was to display MRI, CT, and cryosection previews simultaneously, enabling the user to browse all three portions of the library at once.

Generalizations and Limitations

The VHE interface demonstrates a browsing approach that could be generalized to other image sets in which images are sequenced in some meaningful progression. The Visible Human library contains spatially sequenced images, or volumetric data. While medical imaging is the most common form, the interface could be used to browse other cross-sectional or volumetric datasets, such as geological or atmospheric samplings, allowing users to visualize pattern changes in sedimentation layers or storm turbulence. Images could also be sequenced by time of capture, as in digital video. In this case, axial cross-sections would correspond to individual frames of the video sequence, and longitudinal sections would show motion patterns of video action across frames [ED94]. In fact, we originally encoded the VHE meta-data as an MPEG video, but opted for another representation for performance reasons. Also, the images themselves could be visual representations of more abstract scientific measurements. For example, astronomers could browse a library of spectrograms collected over a period of time to view changes in wave patterns. Applying the VHE interface to these different datasets would simply require creation of the miniature meta-data (we developed tools to automate this process for the Visible Human project), and alteration of interface dimensions to accommodate the appropriate image sizes.

Currently, the VHE interface approach is limited to libraries containing such sequenced images. For libraries containing heterogeneous collections of arbitrary, unrelated images or textual documents, the overview window would need to be redesigned to show a meaningful representation of the library. For example, chronological timelines or simple page numbers would provide additional understanding of library contents. In this case, the ability to quickly flip through many images or pages, like flipping through a book, would be very helpful when exploring large libraries.

Future Work

With respect to the VHE interface, improvements could be made, including those suggested in the User Feedback. Also, a helpful new capability might allow the user to browse higher resolution detailed data (retrieved from the Visible Human library) of a smaller portion of the body in the same fashion used to browse the initial miniature body. For example, if the user wishes to see the heart in higher detail, retrieving portions of all the library axial cross-sections containing the heart might be preferred over a single large axial image. A schema could allow the user to specify a small sub-region of the body to zoom on, then, after retrieval, the entire interface would be reused for exploring that sub-region in higher resolution. The overview and preview windows would display views of the zoomed sub-region, for example, a coronal and axial section (respectively) of the heart. Naturally, the user could return back to explore the initial miniature Visible Human body at any time. With this zooming method, users could browse high detailed data using the same interface techniques as described in this paper, instead of simply viewing high resolution data only as single large axial images.

In addition, investigation of other visual overview representation and browsing techniques for this type of digital library structure is needed. As computer hardware progresses, 3D representation techniques are becoming more tractable and attractive. Human factors research is required to determine usability measures of differences between 2D representations, of which the VHE is an example, and potential 3D representations.

In the broader picture, as the Visible Human digital library grows to include textual annotations, segmented images, attribute indexes, relationship graphs, video animations, and many other forms of information, additional user interface browsing techniques will apply. The graphical interface approach presented in this paper could be combined with Korn's textual browser intended for navigating a medical terminology hierarchy containing links to the Visible Human images [KS95]. If the two interfaces were tightly coupled, manipulating either the graphical or textual interface elements would immediately show effects in the other domain. Users could browse using a combined strategy, utilizing their knowledge of both medical terminology and visual appearance of human anatomy. We believe that integrating different interface approaches for various information types would provide a rich comprehensive browsing environment for an expanding Visible Human digital library.

ACKNOWLEDGMENTS

Chris North received a National Library of Medicine fellowship, administered by the Oak Ridge Institute for Science and Education. Additional funding was provided by a National Library of Medicine grant and National Science Foundation grant EEC 94-02384.

SOFTWARE

The VHE interface software is fully functional and freely available for public use. For information on obtaining it, see: http://www.cs.umd.edu/projects/hcil/Research/1995/vhe.html, or anonymous ftp.cs.umd.edu in /pub/hcil/Demos/VHP.

REFERENCES

Resources

north@cs.umd.edu

Web Accessibility