Send all correspondence to:

Kent L. Norman
Department of Psychology
University of Maryland
College Park, MD 20742-4411

kent_norman@lap.umd.edu

fax: 301 314-9566
 
 

An experiment is reported that compared expandable indexes providing full menu context with sequential menus providing only partial context. Menu depth was varied using hierarchies of 2, 3, and 4 levels deep in an asymmetric structure of 457 root level items. Menus were presented on the World Wide Web within a browser. Participants searched for specific targets. Results suggest that reducing the depth of hierarchies improves performance in terms of speed and search efficiency. Surprisingly, expandable indexes resulted in poorer performance with deeper hierarchies than did sequential menus.
 

Keywords: Information retrieval, menu selection, depth versus breadth in information design, web page design, hypertext, searching, expandable indexes.
 
 

Information retrieval from the World Wide Web (WWW) is becoming a daily activity in both work and leisure environments. In order to make information retrieval more efficient, it is necessary that indexes, menus, and links be carefully designed.

The goal of this study was to investigate the use of expanding hierarchical indexes. One advantage of using such indexes for menu selection is that they preserve the full context of the choice within the hierarchy. While the user browses through the hierarchical structure, the tree is fully displayed. Thus at any point, the user has access to the whole set of major and same level categories. Sequential menus, on the other hand, do not display the full hierarchical context as they drop down to deeper levels in the hierarchy. Only elements in the selected category are displayed as options for browsing. This is of particular importance on the WWW when the number of root level alternatives is large and the depth of the hierarchy is greater than two.

Depth versus breadth in hierarchical menu structures has been the topic of much research. The trade-off between menu depth and breadth is considered by some researchers as the most important aspects that must be considered in the design of hierarchical menu systems (Jacko, Salvendy, & Koubek, 1994). Miller (1981) found that short-term memory is a limitation of the increased depth of the hierarchy. His experiment examined four structures (64¹, 2⁶, 4³, and 8²) with a fixed number of target items (64). As depth increased so did response time to select the desired item.

Snowberry, Parkinson & Sisson (1983) replicated Miller’s study by examining the same structures but this time including an initial screening session during which subjects took memory span and visual scanning tests. They found that instead of memory span, visual scanning was predictive of performance, especially in the deepest hierarchies.

Kiger (1984) extended Miller’s research by doing an experiment that provided users with five modes of varying menu designs of 64 end nodes (2⁶, 4³, 8², and 16x4, 4x16). Performance and preference data were collected. The results of the experiment showed that the time and number of errors increased with the depth of the menu structure. The 4x16 structure had the fastest response times and the fewest errors. The participants ranked the menus with least depth as the most favorable (The 8² structure was favored).

An experiment by Jacko and Salvendy (1996) tested six structures (2², 2³, 2⁶, 8², 8³, and 8⁶) for reaction time, error rates, and subjective preference. They demonstrated that as depth of a computerized, hierarchical menu increased, perceived complexity of the menu increased significantly. Campbell (1988) identified multiple paths, multiple outcomes, conflicting interdependence among paths and uncertain linkages as four characteristics of a complex task. Jacko and Salvendy build on this framework to suggest that these four characteristics are present as depth increases, and the presence of these four characteristics is responsible for the increase in complexity.

Wallace et al. (1987) confirmed that broader, shallower trees (4x3 versus 2x6) produced superior performance, and showed that, when users were stressed, they made 96 percent more errors and took 16 percent longer. The stressor was simply an instruction to work quickly ("It is imperative that you finish the task just as quickly as possible"). The control group received mild instructions to avoid rushing ("Take your time; there is no rush").

Norman and Chin (1988) fixed the number of levels at four, with 256 target items, and varied the shape of the tree structures. They recommend greater breadth at the root and at the leaves, and added a further encouragement to minimize the total number of menu frames needed so as to increase familiarity.

Zaphiris and Mtei (1997) replicated Kiger’s (1984) structures but this time on the WWW using hyperlinks. Overall, their results were in agreement with those of Kiger (1984). They found that of the structures tested (2⁶, 4³, 8², and 16x4, 4x16), the 8² structure was the fastest to search.

Larson & Czerwinski (1998) carried out an experiment using 512 bottom level nodes arranged in three different structures (8x8x8, 32x16, 16x32). Subjects on average completed search tasks faster in the 16x32 hierarchy, second fastest in the 32x16 hierarchy, and slowest in the 8x8x8 hierarchy. Also, on average, subjects tended to be lost least often in the 16x32 hierarchy. Larson & Czerwinski calculated "lostness" through an analysis of the number of unique and total links visited in comparison to the "optimal" path.

One limitation with previous studies on the topic of menu selection has been the use of symmetric hierarchical structures which do not reflect typical menu structures in real-world applications. Based on a more ecologically valid approach, the present experiment involves an asymmetric hierarchical structure which is common in most practical applications (Norman, 1990).

While previous research has investigated the depth versus breadth effect, it has neglected the effect of type of menu presentation. Specifically, the comparison between expandable indexes with full context and sequential menus with limited context in hierarchical structures on the WWW has not received sufficient attention despite the fact that many designers are employing expandable indexes in current WWW design. It was predicted that expandable indexes would result in superior performance because they provide a full context for the choices at lower levels of the hierarchy. Furthermore, sequential menus are predicted to be less efficient and to result in more backtracking due to loss of context and forgetting on the part of the user.

Participants

Twenty-one students (14 males and 7 females) at the University of Maryland with at least an undergraduate degree in business and management took part in this study. The mean age of the participants was 27. They were all experienced computer and WWW users who had been using the WWW for at least one year and at least twice per week. All participated voluntarily and signed a consent form after agreeing to the terms of the experiment.

Menu hierarchies were constructed and hosted on the WWW. The bottom level nodes for the various hierarchical structures were taken from the 1996 Annual Survey of Manufacturers "Statistics for Industry Groups and Industries" of the United States Bureau of the Census. This survey consisted of a total of 457 bottom level nodes arranged in a specific hierarchy. This structure was used as the basis for designing the different WWW menu organizations used in this study.

Menu structures were generated that varied along two factors: depth of the hierarchy and method of display. Three levels of depth were used: 2, 3, and 4. The method of display used either expandable indexes (Expandable) or sequential menus (Sequential). A factorial design (3 x 2) was used to generate all six possible combinations of levels.

Pages were generated for display in a WWW browser (Netscape 4.0). A top hyperlink bar was also displayed that allowed users to move back to previous pages or to return to the top level of the menu hierarchy. Figure 1 shows a series of displays depicting the expandable index with a menu depth of 4.
 
 

Figure 2 shows an example of the same path through the sequential menu with a depth of 4.
 
 

Procedures

Each participant was first told the purpose of the experiment and was then presented with the basic instructions to follow throughout the experiment. Specifically, they were told that we were interested in determining the optimal depth in presenting links on the WWW for expandable indexes and sequential menus that varied the amount of context. In addition, they were given the following general instructions:

1.  They were to browse through the assigned pages until they felt comfortable that they found the answer to the assigned task questions.
2.  They were to perform all searches as quickly as possible while making as few mouse clicks as possible.
3.  They were to navigate through the web pages using only the links present in the pages and not the navigation buttons of the browser.
4.  They were instructed not to browse for any other information that they have not been asked to seek until they had finished all assigned tasks.
5.  They should read carefully the assigned tasks and instructions presented to them.

Participants were divided into 3 randomly selected groups (between subjects factor) and assigned to one of the three levels of menu depth. Menu type (Expandable or Sequential) was a within subjects factor and was counterbalanced by presenting different treatment orders. Half of the participants in each group worked with the Expandable Menu first and the Sequential Menu second. The remaining half worked with Sequential Menu first and Expandable Menu second. Each participant performed a total of 10 searches (5 in a Expandable Menu and 5 in a Sequential Menu). The search targets were counterbalanced over the orders and conditions and were selected in such a way that all pages visited had a similar breadth size. The browser height was set to just display 20 lines of text. Each trial’s target and hierarchy information was presented to the participant on paper with one target presented per page next to the computer so it was constantly available as a reference. At the ! start of each trial, the participant was asked to turn the page in order to see the new target and menu type to be used for the next trial.

Three kinds of data were collected for analysis: response times, search efficiency, and subjective ratings. Response times and retrieval efficiency data were collected from the WWW server log. Response time was calculated as the time taken by the participant to browse from the main page to the specific bottom level node. Efficiency of retrieval was calculated as the difference in mouse clicks between the total number of mouse clicks to reach the bottom level node minus the "optimal" number of mouse clicks required to reach the target bottom level node After the participants finished all 10 tasks assigned, subjective preference responses were collected for the Expandable and Sequential Menus by answering on a scale of 1 to 9 the following 3 subjective questions:

1.  Which of the two types of menus browsed would you prefer to use?
2.  Rate the ease of navigation.
3.  Rate your sense of orientation.

Response Times of Information Retrieval

Figure 3 shows that with the Expandable Menu participants completed search tasks fastest in the hierarchy of Depth 2, second fastest in the Depth 3, and slowest in Depth 4. Menu depth had less of an effect for the Sequential Menus. The average response time for Depths 2 and 3 was about the same but faster than the response time for Depth 4. Surprisingly, searches were slower for the Expandable Menus than for the Sequential Menus, particularly for hierarchy depths of 3 and 4.
 
 

The mean response times and corresponding standard deviations for each hierarchy are shown in Table 1.
 
 

A split-plot analysis of variance indicated that the main effect of Menu Type was significant (F(1, 18) = 13.91, p < .01). On the average, response times were nearly 50% longer for the Expandable Menus than for the Sequential Menus. In addition, there was a significant interaction between Menu Type and Menu Depth (F(2, 18) = 3.62, p < .05). The effect of Menu Depth was greater for the Expandable Menus than for the Sequential Menus. Finally, the main effect of Menu Depth was not significant (F(2, 18) = 1.34, P > .05). This was probably due to the low statistical power associated with the between-subject factor.
 

Efficiency of Information Retrieval

An analysis of the number of mouse clicks made above and beyond the most direct path to the target was performed. Search inefficiency is equal to the total number of mouse clicks to complete each task minus the "optimal" minimum number of clicks. On the average, participants in the Depth 2 hierarchies made fewer mouse clicks in the Expandable Menu than in the Sequential Menu. For Depth 3 hierarchies, performance was about the same for the two menu types. However, in the Depth 4 hierarchies performance was drastically worse with the Expandable Menu than with the Sequential Menu. The mean number of mouse clicks beyond the most efficient path for each hierarchy are graphed in Figure 4 and listed in Table 2 along with corresponding standard deviations.
 
 

A split-plot analysis of variance revealed a significant interaction between Menu Depth and Menu Type (F(2, 18) = 4.75, p < .05). The effect of Menu Depth was much larger for the Expandable Menus than for the Sequential Menus. On the other hand, the main effects of Menu Depth and Menu Type were not significant (F(2, 18) = 1.97, p > .05 and F(1, 18) = 1.34, p > .05, respectively) owing to the form of the interaction.

Subjective Ratings and Preference Measures

After finishing the search tasks, each subject completed a subjective satisfaction questionnaire composed of three ratings: preference to use, ease of navigation, and sense of orientation. Table 3 lists the preference ratings (means and standard deviations) for the Expandable Menu over the Sequential Menu. These results indicate that subjects had a slight preference for the Expandable Menus for depths of 2 and 3 but for depth 4 their preference is to use the Sequential Menu. However, none of these effects were significant (F(5,34) = 0.57, p > .05).

Table 3

Mean Preference Ratings for the Expandable Menu Over the Sequential Menu. (1 = preference for Sequential Menu. 9 = preference for Expandable Menu. Standard deviations are shown in parentheses.)

Table 4 lists the ease of navigation ratings (means and standard deviations) for the Expandable and Sequential menus. Participants tended to rate the Sequential Menus as slightly easier to navigate than the Expandable Menus for Depths of 2 and 4 but about equal for Depth of 3. However, again, the analysis of variance test indicated no significant effect due to ease of navigation (F(5,34) = 0.33, p > .05).

Table 4

Mean Ratings of Ease of Navigation (1 = low. 9 = high. Standard deviations are shown in parentheses).

 

 
 
 

Finally, when asked to rate their sense of orientation, subjects rated the Sequential Menu as providing them with a slightly better sense of orientation when working with the Depth 2 and 4 hierarchies than the Expandable Menu. However the analysis of variance test indicated no significant main effect with respect to sense of orientation (F(5,34) = 1.04, p > .05).
 
 

Table 5

Mean Ratings of Sense of Orientation (1 = low. 9 = high. Standard deviations in parentheses)

 

DISCUSSION

The response time and efficiency of retrieval data support previous research on the issue of depth versus breadth in hierarchical menu structures. Performance, both in terms of access time and efficiency, decreases as the depth of the menu structure increases. The present results extend this conclusion to asymmetric menus as well.

It was expected that menus employing the expandable indexes providing the full context of choices within the hierarchy would result in better performance than sequential menus that provide only limited context. However, this was not the case, particularly for deeper hierarchies where one would expect the context information to be even more beneficial. Surprisingly, tasks were completed significantly faster with Sequential Menus than with Expandable Menus. Efficiency of search data indicated a slightly different pattern, which may help to explain the unexpected result for response time. For Depth 2 hierarchies, the Sequential Menus resulted in slightly more search moves than the Sequential Menu; for Depth 3 performance was equal; but for Depth 4 the Expandable Menu resulted in significantly more search moves than the Sequential Menus.

A possible reason for the poor performance of the Expandable Menu has to do with the problem of long vertical lists on a screen showing only 20 lines of text. When the Expandable Menus were expanded in the Depth 4 hierarchies, they were very long and unwieldy. Users opened and closed more levels of the hierarchy and took longer to find the targets because the long indented indexes were hard to scan and scroll for context information. Furthermore, context may have been lost as the expanded index scrolled out of view in the limited size of the browser window.

User preferences tended to agree with user performance; however, none of these effects were statistically significant. Users tended to prefer to use Expandable Menus for depths of 2 and 3 but when the depth is increased to 4, they chose Sequential Menus as more desirable. Also users found the Expandable Menus difficult to navigate (except for Depth 3 hierarchies) and lost their orientation while browsing Expandable Menus (except for Depth 3 hierarchy where Expandable and Sequential Menus had equal ratings for sense of orientation).

Implications for WWW Designers

Suggestions for Future Research

 

ACKNOWLEDGEMENTS

The authors would like to thank all those that helped in recruiting volunteers and those that provided comments on different drafts of the paper, in particular, Chanda Harris and Betty Murphy. Finally, special thanks to all the volunteers that offered their time in order to participate as subjects in this experiment. This project was funded in part by the Statistical Research Division of the Bureau of the Census.
 
 

Web Accessibility