Expandable Indexes Versus Sequential Menus for Searching
Hierarchies on the World Wide Web
Computer Science Department
Kent L. Norman
Department of Psychology
University of Maryland
College Park, MD 20742
Send all correspondence to:
Kent L. Norman
Department of Psychology
University of Maryland
College Park, MD 20742-4411
fax: 301 314-9566
An experiment is reported that compared expandable
indexes providing full menu context with sequential menus providing only
partial context. Menu depth was varied using hierarchies of 2, 3, and 4
levels deep in an asymmetric structure of 457 root level items. Menus were
presented on the World Wide Web within a browser. Participants searched
for specific targets. Results suggest that reducing the depth of hierarchies
improves performance in terms of speed and search efficiency. Surprisingly,
expandable indexes resulted in poorer performance with deeper hierarchies
than did sequential menus.
Keywords: Information retrieval, menu selection,
depth versus breadth in information design, web page design, hypertext,
searching, expandable indexes.
Information retrieval from the World Wide Web (WWW) is becoming a daily activity in both work and leisure environments. In order to make information retrieval more efficient, it is necessary that indexes, menus, and links be carefully designed.
The goal of this study was to investigate the use of expanding hierarchical indexes. One advantage of using such indexes for menu selection is that they preserve the full context of the choice within the hierarchy. While the user browses through the hierarchical structure, the tree is fully displayed. Thus at any point, the user has access to the whole set of major and same level categories. Sequential menus, on the other hand, do not display the full hierarchical context as they drop down to deeper levels in the hierarchy. Only elements in the selected category are displayed as options for browsing. This is of particular importance on the WWW when the number of root level alternatives is large and the depth of the hierarchy is greater than two.
Depth versus breadth in hierarchical menu structures has been the topic of much research. The trade-off between menu depth and breadth is considered by some researchers as the most important aspects that must be considered in the design of hierarchical menu systems (Jacko, Salvendy, & Koubek, 1994). Miller (1981) found that short-term memory is a limitation of the increased depth of the hierarchy. His experiment examined four structures (641, 26, 43, and 82) with a fixed number of target items (64). As depth increased so did response time to select the desired item.
Snowberry, Parkinson & Sisson (1983) replicated Millerís study by examining the same structures but this time including an initial screening session during which subjects took memory span and visual scanning tests. They found that instead of memory span, visual scanning was predictive of performance, especially in the deepest hierarchies.
Kiger (1984) extended Millerís research by doing an experiment that provided users with five modes of varying menu designs of 64 end nodes (26, 43, 82, and 16x4, 4x16). Performance and preference data were collected. The results of the experiment showed that the time and number of errors increased with the depth of the menu structure. The 4x16 structure had the fastest response times and the fewest errors. The participants ranked the menus with least depth as the most favorable (The 82 structure was favored).
An experiment by Jacko and Salvendy (1996) tested six structures (22, 23, 26, 82, 83, and 86) for reaction time, error rates, and subjective preference. They demonstrated that as depth of a computerized, hierarchical menu increased, perceived complexity of the menu increased significantly. Campbell (1988) identified multiple paths, multiple outcomes, conflicting interdependence among paths and uncertain linkages as four characteristics of a complex task. Jacko and Salvendy build on this framework to suggest that these four characteristics are present as depth increases, and the presence of these four characteristics is responsible for the increase in complexity.
Wallace et al. (1987) confirmed that broader, shallower trees (4x3 versus 2x6) produced superior performance, and showed that, when users were stressed, they made 96 percent more errors and took 16 percent longer. The stressor was simply an instruction to work quickly ("It is imperative that you finish the task just as quickly as possible"). The control group received mild instructions to avoid rushing ("Take your time; there is no rush").
Norman and Chin (1988) fixed the number of levels at four, with 256 target items, and varied the shape of the tree structures. They recommend greater breadth at the root and at the leaves, and added a further encouragement to minimize the total number of menu frames needed so as to increase familiarity.
Zaphiris and Mtei (1997) replicated Kigerís (1984) structures but this time on the WWW using hyperlinks. Overall, their results were in agreement with those of Kiger (1984). They found that of the structures tested (26, 43, 82, and 16x4, 4x16), the 82 structure was the fastest to search.
Larson & Czerwinski (1998) carried out an experiment using 512 bottom level nodes arranged in three different structures (8x8x8, 32x16, 16x32). Subjects on average completed search tasks faster in the 16x32 hierarchy, second fastest in the 32x16 hierarchy, and slowest in the 8x8x8 hierarchy. Also, on average, subjects tended to be lost least often in the 16x32 hierarchy. Larson & Czerwinski calculated "lostness" through an analysis of the number of unique and total links visited in comparison to the "optimal" path.
One limitation with previous studies on the topic of menu selection has been the use of symmetric hierarchical structures which do not reflect typical menu structures in real-world applications. Based on a more ecologically valid approach, the present experiment involves an asymmetric hierarchical structure which is common in most practical applications (Norman, 1990).
While previous research has investigated the depth versus breadth effect, it has neglected the effect of type of menu presentation. Specifically, the comparison between expandable indexes with full context and sequential menus with limited context in hierarchical structures on the WWW has not received sufficient attention despite the fact that many designers are employing expandable indexes in current WWW design. It was predicted that expandable indexes would result in superior performance because they provide a full context for the choices at lower levels of the hierarchy. Furthermore, sequential menus are predicted to be less efficient and to result in more backtracking due to loss of context and forgetting on the part of the user.
Menu hierarchies were constructed and hosted on the WWW. The bottom level nodes for the various hierarchical structures were taken from the 1996 Annual Survey of Manufacturers "Statistics for Industry Groups and Industries" of the United States Bureau of the Census. This survey consisted of a total of 457 bottom level nodes arranged in a specific hierarchy. This structure was used as the basis for designing the different WWW menu organizations used in this study.
Menu structures were generated that varied along two factors: depth of the hierarchy and method of display. Three levels of depth were used: 2, 3, and 4. The method of display used either expandable indexes (Expandable) or sequential menus (Sequential). A factorial design (3 x 2) was used to generate all six possible combinations of levels.
Pages were generated for display in a WWW browser (Netscape 4.0). A
top hyperlink bar was also displayed that allowed users to move back to
previous pages or to return to the top level of the menu hierarchy. Figure
1 shows a series of displays depicting the expandable index with a menu
depth of 4.
Figure 1. Expandable index menu with hierarchical depth of 4. (Hand shows the selection at each level. Arrows show the series of selections but were not shown during actual trials.)
Figure 2 shows an example of the same path through the sequential menu
with a depth of 4.
Figure 2. Sequential menu with hierarchical depth of 4. (Hand shows the selection at each level. Arrows show the series of selections but were not shown during actual trials.)
Each participant was first told the purpose of the experiment and was then presented with the basic instructions to follow throughout the experiment. Specifically, they were told that we were interested in determining the optimal depth in presenting links on the WWW for expandable indexes and sequential menus that varied the amount of context. In addition, they were given the following general instructions:
Participants were divided into 3 randomly selected groups (between subjects factor) and assigned to one of the three levels of menu depth. Menu type (Expandable or Sequential) was a within subjects factor and was counterbalanced by presenting different treatment orders. Half of the participants in each group worked with the Expandable Menu first and the Sequential Menu second. The remaining half worked with Sequential Menu first and Expandable Menu second. Each participant performed a total of 10 searches (5 in a Expandable Menu and 5 in a Sequential Menu). The search targets were counterbalanced over the orders and conditions and were selected in such a way that all pages visited had a similar breadth size. The browser height was set to just display 20 lines of text. Each trialís target and hierarchy information was presented to the participant on paper with one target presented per page next to the computer so it was constantly available as a reference. At the ! start of each trial, the participant was asked to turn the page in order to see the new target and menu type to be used for the next trial.
Three kinds of data were collected for analysis: response times, search efficiency, and subjective ratings. Response times and retrieval efficiency data were collected from the WWW server log. Response time was calculated as the time taken by the participant to browse from the main page to the specific bottom level node. Efficiency of retrieval was calculated as the difference in mouse clicks between the total number of mouse clicks to reach the bottom level node minus the "optimal" number of mouse clicks required to reach the target bottom level node After the participants finished all 10 tasks assigned, subjective preference responses were collected for the Expandable and Sequential Menus by answering on a scale of 1 to 9 the following 3 subjective questions:
Response Times of Information Retrieval
Figure 3 shows that with the Expandable Menu participants completed
search tasks fastest in the hierarchy of Depth 2, second fastest in the
Depth 3, and slowest in Depth 4. Menu depth had less of an effect for the
Sequential Menus. The average response time for Depths 2 and 3 was about
the same but faster than the response time for Depth 4. Surprisingly, searches
were slower for the Expandable Menus than for the Sequential Menus, particularly
for hierarchy depths of 3 and 4.
Figure 3: Mean response times as a function of Menu Type (Expandable or Sequential) and Menu Depth (2, 3, or 4).
The mean response times and corresponding standard deviations for each
hierarchy are shown in Table 1.
Mean response times for Menu Type (Expandable or Sequential) and Menu Depth (2, 3, or 4). (Standard deviations are in parentheses)
A split-plot analysis of variance indicated that the main effect of
Menu Type was significant (F(1, 18) = 13.91, p < .01). On the
average, response times were nearly 50% longer for the Expandable Menus
than for the Sequential Menus. In addition, there was a significant interaction
between Menu Type and Menu Depth (F(2, 18) = 3.62, p < .05).
The effect of Menu Depth was greater for the Expandable Menus than for
the Sequential Menus. Finally, the main effect of Menu Depth was not significant
(F(2, 18) = 1.34, P > .05). This was probably due to the low statistical
power associated with the between-subject factor.
Efficiency of Information Retrieval
An analysis of the number of mouse clicks made above and beyond the
most direct path to the target was performed. Search inefficiency is equal
to the total number of mouse clicks to complete each task minus the "optimal"
minimum number of clicks. On the average, participants in the Depth 2 hierarchies
made fewer mouse clicks in the Expandable Menu than in the Sequential Menu.
For Depth 3 hierarchies, performance was about the same for the two menu
types. However, in the Depth 4 hierarchies performance was drastically
worse with the Expandable Menu than with the Sequential Menu. The mean
number of mouse clicks beyond the most efficient path for each hierarchy
are graphed in Figure 4 and listed in Table 2 along with corresponding
Figure 4: Inefficiency of information retrieval
as a function of Menu Type (Expandable or Sequential) and Menu Depth (2,
3, or 4).
Inefficiency of Information Retrieval (Standard deviations are shown in parentheses).
A split-plot analysis of variance revealed a significant interaction between Menu Depth and Menu Type (F(2, 18) = 4.75, p < .05). The effect of Menu Depth was much larger for the Expandable Menus than for the Sequential Menus. On the other hand, the main effects of Menu Depth and Menu Type were not significant (F(2, 18) = 1.97, p > .05 and F(1, 18) = 1.34, p > .05, respectively) owing to the form of the interaction.
Subjective Ratings and Preference Measures
Finally, when asked to rate their sense of orientation, subjects rated
the Sequential Menu as providing them with a slightly better sense of orientation
when working with the Depth 2 and 4 hierarchies than the Expandable Menu.
However the analysis of variance test indicated no significant main effect
with respect to sense of orientation (F(5,34) = 1.04, p > .05).
Mean Ratings of Sense of Orientation (1 = low. 9 = high. Standard deviations in parentheses)
The response time and efficiency of retrieval data support previous research on the issue of depth versus breadth in hierarchical menu structures. Performance, both in terms of access time and efficiency, decreases as the depth of the menu structure increases. The present results extend this conclusion to asymmetric menus as well.
It was expected that menus employing the expandable indexes providing the full context of choices within the hierarchy would result in better performance than sequential menus that provide only limited context. However, this was not the case, particularly for deeper hierarchies where one would expect the context information to be even more beneficial. Surprisingly, tasks were completed significantly faster with Sequential Menus than with Expandable Menus. Efficiency of search data indicated a slightly different pattern, which may help to explain the unexpected result for response time. For Depth 2 hierarchies, the Sequential Menus resulted in slightly more search moves than the Sequential Menu; for Depth 3 performance was equal; but for Depth 4 the Expandable Menu resulted in significantly more search moves than the Sequential Menus.
A possible reason for the poor performance of the Expandable Menu has to do with the problem of long vertical lists on a screen showing only 20 lines of text. When the Expandable Menus were expanded in the Depth 4 hierarchies, they were very long and unwieldy. Users opened and closed more levels of the hierarchy and took longer to find the targets because the long indented indexes were hard to scan and scroll for context information. Furthermore, context may have been lost as the expanded index scrolled out of view in the limited size of the browser window.
User preferences tended to agree with user performance; however, none of these effects were statistically significant. Users tended to prefer to use Expandable Menus for depths of 2 and 3 but when the depth is increased to 4, they chose Sequential Menus as more desirable. Also users found the Expandable Menus difficult to navigate (except for Depth 3 hierarchies) and lost their orientation while browsing Expandable Menus (except for Depth 3 hierarchy where Expandable and Sequential Menus had equal ratings for sense of orientation).
The authors would like to thank all those that helped in recruiting volunteers and those that provided comments on different drafts of the paper, in particular, Chanda Harris and Betty Murphy. Finally, special thanks to all the volunteers that offered their time in order to participate as subjects in this experiment. This project was funded in part by the Statistical Research Division of the Bureau of the Census.
Jacko, J., & Salvendy, G. (1996). Hierarchical menu design: Breadth, depth, and task complexity. Perceptual and Motor Skills, 82, 1187-1201.
Kiger, J. I. (1984). The depth/breadth tradeoff in the design of menu-driven interfaces. International Journal of Man-Machine Studies, 20, 201-213.
Larson, K. & Czerwinski, M. (1998). Page design: Implications of memory, structure and scent for information retrieval. In the Proceedings of CHI í98 Human Factors in Computing Systems, Los Angeles, CA, April, 1998, ACM Press, pp. 25-32.
Miller, D. P. (1981). The depth/breadth tradeoff in hierarchical computer menus. Proceedings of the Human Factors Society, 296-200.
Norman, K. (1990). The Psychology of Menu Selection: Designing Cognitive Control of the Human/Computer Interface. Ablex Publishing Corporation, Norwood, NJ.
Norman, K. & Chin, J. (1988). The effect of tree structures on search in a hierarchical menu selection system. Behaviour and Information Technology 7, 51-65.
Snowberry, K., Parkinson, S. R., & Sisson, N. (1983). Computer display menus. Ergonomics, 26, 7, 699-712.
Wallace, D., Anderson, N. & Shneiderman, B. (1987). Time stress effects on two menu selection systems. Proc. Human Factors Society, Thirty-First Annual Meeting (1987), 727-731
Zaphiris, P. & Mtei, L. (1997). Depth vs Breadth
in the Arrangement of Web Links. Available at http://otal.umd.edu/SHORE/bs04/