Dual Navigation of Computerized Self-Administered

Questionnaires and Organizational Records

 

Kent Norman, Laura Slaughter, Zachary Friedman, Kirk Norman, Rod Stevenson

 

Human-Computer Interaction Laboratory

Institute for Advanced Computer Studies

Department of Psychology

University of Maryland, College Park, MD 20742-3255

 

 

Introduction

 

Questionnaires for businesses and other organizations pose challenges that differ in a number of ways from questionnaires for individuals (Dillman, 2000). For instance, business questionnaires often have tight deadlines for completion due to time critical economic information being reported. One of the major obstacles that impede the successful completion of organizational questionnaires is that responses must often be retrieved from records rather than from personal opinion or memory. Respondents for business questionnaires are required to find specific information, make assumptions about the organization that they are representing, and at times calculate or synthesize responses from available data. Additionally, business and organizational surveys are often repeated annually or semi-annually making it necessary for the respondent to answer the same questions at every cycle and sometimes maintain a file of essential data.

 

Computerized self-administered questionnaires (CSAQ) can be an effective method of collecting data from organizations, and they can help to facilitate the process of data acquisition and overcome difficulties. On-line questionnaires offer a number of administrative advantages such as the ability to quickly disseminate and retrieve questionnaire data as well as having responses already in electronic form, preventing the need for manual entering of data. Furthermore, CSAQs can be formatted to a specific organization’s needs and the preferences can be stored for later use. In the case where respondents must search on-line organizational records, CSAQs can require less manual entry of data by automatically copying information. However, new problems arise with the interaction of CSAQs and organizational records. One problem is that of dual navigation of on-line records and the questionnaire in tandem. In order to understand dual navigation of organizational records and CSAQ’s, this research explores how respondents traverse a questionnaire and a web site containing information about an organization. This research is being conducted with the U.S. Bureau of the Census to help understand issues surrounding dual navigation of on-line questionnaires and organizational records.

 

The majority of research about questionnaire design focuses on questionnaires for private individuals. Norman et al. (2000) states that when questionnaires are answered from personal memory, they should be designed in a way that is congruent with human knowledge structures. They also concludes that when there is a match between the order of questions in a survey and the order of retrieval of information, processing is most efficient. In the case of organizational questionnaires, the structure of external information (as opposed to internal human memory) influences how the questionnaire will be processed and perceived by the respondent. The major problem with organizational surveys is that the respondent does not always have knowledge of the overall structure of the organizational records and it is difficult to design for the wide array of possible external information structures that different organizations have.

 

If an organizational CSAQ cannot be designed to take advantage of internal or external knowledge structures, it may nevertheless be possible to learn something from the navigation patterns of respondents. What design features are useful for organizational CSAQs when the user must navigate between the questionnaire and records? What types of navigational patterns do respondents develop to complete the survey?

 

Dual navigation: CSAQ’s and Records

 

Completing an organizational CSAQ requires that a respondent, who may not be familiar with either the organizational site or the questionnaire, search and retrieve answers that will be transferred to the questionnaire forms. This process requires dual navigation; the respondent must navigate not only the questionnaire but also files of on-line records.

 

A respondent may be completing the questionnaire for the first and the only time or repeatedly filling it out on a monthly or yearly schedule. Thus, there are four possible scenarios. A respondent may be 1) familiar with the organizational site, but new to the questionnaire, 2) familiar with the questionnaire but the organizational records are unfamiliar, 3) familiar with both the questionnaire and the organizational records, or 4) unfamiliar with both the questionnaire and the organizational records.

 

Respondents may be performing a number of different types of tasks while completing the questionnaire. The most common task is entering the data during a first pass through the questionnaire either in a linear or a non-linear sequence (perhaps due to idiosyncratic files of organizational information). Alternatively, a respondent may be making edits by going back to enter missing information or to re-enter data that was found to be in error. The task of either mentally or physically storing information that will be helpful for the next cycle may occur when the respondent has knowledge that the questionnaire will be re-administered.

 

Current exploratory research

 

This study is an extension to the work on the design of computerized self-administered questionnaires by Norman et al. (2000). The current research will analyze what can be learned from the systematic study of dual-navigation between a set of organizational records and a questionnaire. Two different questionnaire design alternatives, a form-based and an item-based, were constructed for the questionnaire. In the form-based design, question items were grouped into four discrete semantic sections with all the question items in each section shown on a single screen. The item-based design had only one question item on each screen. Both interface designs provided a side index for navigation to the sections of the questionnaire. The item-based questionnaire format was thought to be more difficult to navigate due to loss of context and number of operations required when navigating. Therefore, it was anticipated that this format would have a detrimental effect on the performance of respondents as they attempted to navigate the organizational records in addition to the CSAQ.

 

Our goal in this research is to develop design principles for constructing organizational CSAQ’s. We examined 1) completion time and accuracy of respondents, 2) patterns of navigation between the organizational site and the CSAQ, 3) subjective satisfaction with the use of an organizational CSAQ, and also, 4) comments that respondents made about the task. Navigational patterns for the item-based group were predicted to show more traversals back and forth between the organizational records and the CSAQ as respondents sought to find the answer to each question out of context. Respondents using the form-based design were expected to fill out the questionnaire in a non-linear sequence. It was thought that they would use the questions displayed for each section to get an overview and then use the majority of their time in the organizational records, thus transferring answers in the order that they appear in the records.

 

Method

 

Participants

 

Thirty-eight undergraduates (25 males, 13 females), ranging in age from 18 to 24 with a median age of 19, participated in this experiment. The participants were enrolled in an introductory psychology course at the University of Maryland and took part in the investigation in order to satisfy a course requirement.

 

Materials

 

Set-up

 

The study was administered to participants on either a Macintosh G3 or G4 computer with dual Apple Studio Display 15-inch flat screen monitors. Figure 1 shows a photograph of the experimental set-up. The participants were instructed to view the organizational records on one monitor and the computerized questionnaire on the second monitor. Participants worked independently through the experiment but were given the opportunity to ask questions at any point. The experiment was administered to each participant individually in a testing room. All of the experimental materials were web-based documents and all data was recorded to a web server using javascript and cgi scripts.

 

 

 

Computerized self-administered questionnaire design

 

The questionnaire constructed for this study consisted of 41 questions, arranged into 4 topic sections each containing 10 questions with 1 extra question that asked about the URL for the organizational records pages. The four topic sections were labeled: Information, Activities, Positions, and Finances. The majority of the question answers were directly available in the organizational records. Some questions required that the participant calculate answers from several pieces of information available in the records. For example, to answer the question “How many officers does the organization have?” the participant must count the number of officers from a list in the organizational records since the answer to the question is not explicitly written. A few questions could not be calculated precisely, but instead the participant needed to make an approximation using the organizational records. The questionnaire was created independently from the organizational records. The full questionnaire is shown in Appendix A.

 

Two interface designs were used to display questionnaire items. In both designs, an index to the questionnaire was shown in a frame on the left side of the screen. The index listed 1) an instruction section with directions on how to navigate and complete the survey, 2) four topic section links to questionnaire items, and 3) a link marked ‘Done’ that was required to submit the completed survey. The difference between the two interface designs used in the study pertained to the number of questions presented on the right side “questionnaire” page. In the form-based design, all the questions from each topic section (10 questions) were presented as a single scrollable web page. The participant was able to view all questions in a section by clicking on the topic links in the adjacent index. In the item-based design, the questions were listed one per screen with links provided at the bottom of each question item page labeled ‘next’ and ‘previous’ to allow forward and backward movement through each section of the questionnaire. When a participant clicked on a topic section link in the index, only the first question was displayed for that topic. The participant was required to click ‘next’ to view the other 9 questions within that particular topic section. Figures 2 and 3 show the form-based and item-based interface designs respectively.

 

Organizational records design

           

The organizational records were created as a set of web documents about a fictitious University of Maryland Break-Dancing Club. The records consisted of eight scrollable web pages. Each page contained information on one of the following eight topics: Introduction to the Break-Dancing Club, Officers, Events, Committees, Members, Minutes of Meetings, Budget and Financial Information, and Bylaws. An index in a frame on the left side of the browser window provided links to each section. Figure 4 shows the Introduction page for the Break-Dancing Club organizational records.

 

 

 

 

 

 

 

 

 

Procedure

 

Participants were randomly assigned to participate in the experiment using the form-based or the item-based questionnaire. The experimenter, prior to the start of the experiment, verbally explained all directions. The participant then read and signed an experimental consent form. Instructions were then given about how to operate the browser windows, particularly how to set up the records window on the left hand monitor and the questionnaire on the right.  Participants were informed of the steps required to complete the experiment, navigating between the questionnaire and the records. They were allowed to ask questions during any point of the experiment. They were told that they were to use their best judgment when answering questions that did not have a specific answer in the organizational records (such as answers that needed to be estimated from several pieces of data). The participants were then shown the link marked “Done” and instructed to submit the questionnaire once finished.

 

Participants initially filled out a pre-questionnaire questionnaire demographics form requiring age, gender, level of familiarity with computers, knowledge of the Internet, and previous experience with surveys. Following the pre-questionnaire questionnaire, participants were required to complete the experimental questionnaire, answering questions by navigating the organizational records. After completing the experimental questionnaire, participants answered a brief post-questionnaire questionnaire describing their subjective assessment of 1) the experimental questionnaire’s organization and navigation tools, 2) the organization of the Break-Dancing Club records, and 3) computerized questionnaires in general.  The questions for the pre- and post- questionnaire questionnaires can be found in Tables 1 and 2.

 

Data Recorded

 

The main goal was to examine the navigational data collected as participants traversed between the organizational record web pages and the computerized questionnaire. For each participant, these data included 1) the time (in seconds, starting from zero) when the participant clicked on a particular questionnaire item or organizational record page, 2) a record of visited nodes indicating which page of the organizational records or questionnaire item was clicked on and 3) the participant’s responses to the computerized questionnaire items. An independent groups design was used to make comparisons of participant performance and responses in terms of questionnaire type. The independent variable, questionnaire type, was defined as either form-based or item-based.

 

Results

 

Accuracy

 

An analysis of participant accuracy was assessed by scoring each set of responses using an answer key created by the experimenters. Each participant received a percentage score for the number of correct items answered. Participants in the item-based group (N=20) obtained a mean percentage correct of 56% with a range of 46.3% to 73.2% correct. Participants in the form-based group (N=18) obtained a mean percentage of 59% correct with a range of 43.9 % to 92.7% correct. There was no significant difference in accuracy between the two groups (t(36) = -0.91, n.s.).

 

Completion times

 

The mean time of completion for the form-based group was 26 minutes, 25 seconds. The mean time of completion for the item-based group was 24 minutes, 36 seconds. There was no significant difference between form-based and item-based groups (t(36) = -.9331, n.s.).

 

Pre- and Post- questionnaire items

 

The mean ratings for the pre-questionnaire questionnaire are listed in Table 1. Each of the question items was rated on a scale from 1 to 9 with “no experience” anchoring the right end and “very experienced” on the left. No significant difference was found between form-based and item-based for any of the pre-questionnaire questions.

 

The mean ratings for the post-questionnaire questionnaire are listed in Table 2. Each of the question items was rated on a scale from 1 to 9 with negative adjectives anchoring the right end and positive on the left. For each item, there was no significant difference between item-based and form-based.

 

 

 

 

 

 

 

Navigational Data

 

There were four aspects of the navigational data that were explored. Graphical representations provided an overview of the order in which participants answered the questionnaire items. Bar charts of the navigational data, sectioned into 5 time intervals, were created to illustrate the percentage of accesses made to each questionnaire item within each interval. Another analysis looked at when, during the five time intervals, participants were accessing primarily questionnaire items or accessing organizational records. The final analysis shows how often participants traversed within and between the organizational records and the questionnaire items.

 

Overview of questionnaire navigation patterns

 

Using a parallel coordinate chart (Inselberg, 1985; Manaster, 2000), a graphical representation of the participants’ navigation through the questionnaire was constructed. Each vertical axis represents a single access (a click of the mouse) to a questionnaire item. The first vertical axis on the left-hand side represents the first access, at time point 1 (T1), to a questionnaire item made by a participant and the right-most vertical axis represents the last access to a questionnaire item. The height of the vertical axes represents the questionnaire items from 1 (on the bottom) to 41 (at the top). Each participant’s navigation pattern is represented by a trace drawn across all the vertical axes.

 

Figure 5 shows the parallel coordinate chart for the item-based group and figure 6 shows the parallel coordinate chart for the form-based group. Although both groups essentially answered question items in the linear sequence presented, the item-based group was more rigid while the form-based group had more accesses to questions out of sequential order.

 

 

 

 

 

 

Detailed views of questionnaire navigational patterns

 

For each participant, accesses to questionnaire items were listed in temporal order and segmented into 5 intervals. Figures 7,8,9,10, and 11 depict accesses to questionnaire items within each interval. The length of each bar represents the percentage of accesses to a questionnaire item within the interval  (number of mouse clicks to each question item divided by the total number of mouse clicks for all questions within an interval for the group). These figures provide detailed views of item-based and form-based questionnaire navigational patterns. As was seen in the parallel coordinate graphs, the participants tended to answer questions in the same sequence in which they were presented. The Interval 2 and 3 graphs show that during those intervals, the form-based group were answering questions items that came later in sequential order (up to question 38) compared to the item-based group (up to question 22).

 

 

 

Navigation between the questionnaire and organizational records

 

For each participant, their list of accesses to both questionnaire items and organizational records pages were sectioned, in temporal order, into 5 intervals. Within each interval, the percentage of accesses to questionnaire items and organizational records was calculated (the number of question items accessed in the interval divided by the total number of accesses in that interval). Table 3 lists the mean percentage of accesses that were made to the questionnaire items as opposed to the organizational records. An analysis of variance was performed on the data for the five intervals and questionnaire type (item-based and form-based). This analysis yielded an interaction between intervals and questionnaire type, F (4,144) = 68.74, p<.001. This interaction is shown in Figure 12.

 

These results confirm that when beginning the questionnaire, the majority of accesses made by participants are within the organizational records. In the later intervals, the majority of accesses are made to the questionnaire items. The form-based group made the switch between primarily accessing organizational records to primarily accessing questionnaire items earlier than the item-based group.

 

Access patterns between and within organizational records/questionnaire items

 

Consecutive access points were combined in pairs to better understand how participants were navigating organizational records and questionnaire items. A traversal pair is a traversal from 1) an organizational record to another organizational record, 2) an organizational record to a questionnaire item, 3) a questionnaire item to an organizational record, or 4) a questionnaire item to another questionnaire item. For each participant, the percentage of each traversal pair was calculated. Table 4 displays, for both form-based and item-based questionnaire types, the mean percentage of each type of traversal pair. An analysis of variance was performed on these data and a significant interaction between questionnaire type and traversal pair was found (F(3, 108) = 5.71, p<.005). The item-based group made more traversals within the organizational records than did the form-based group. However, the form-based group had more traversals between questionnaire items. These results are displayed graphically in Figure 13.

 

 

 

 

                       

 

 

Comments

 

Comments about the CSAQ were collected during the post-questionnaire questionnaire. The most frequent comment in the form-based group was that it was difficult to find the answers to some of the questions. One participant stated that “the asked for calculations and stats that the record did not provide.  i had look through the record, remember the numbers and then calculate” and another stated that “the survey was easy to follow but all the information for the club was hard to find sometimes.” For the item based group, several comments and suggestions were made concerning the design on-line surveys. One participant said to “put the questions in on one page, (not a page a question...).” Similarly, another participant stated that designers should “put all the questions for each section on one page, so you can see all of them at once.  It is very hard to hit previous question and next question to try to find one you answered wrong.” The third participant gave more elaborate suggestions to “show several questions at once when it is a long survey... give a progress report to the user. I disliked not knowing how much of the survey I had completed at any given time.  it would have been easier if the survey floated on top of the browser in a corner, so that I would not have to go back and forth between two monitors. In this case, using frames or two separate windows would have been better.”

 

Discussion

 

There is a great potential for the use of computerized self-administered business/organizational questionnaires. Since various interface designs can bring about differences in the ways that respondents navigate between the questionnaire and organizational records, it is necessary to examine the effects of these designs. This study looked at two interface designs, the form-based and the item-based. The objective of this study was to observe differences in navigational patterns and determine if the design had an effect on completion time, accuracy or subjective satisfaction measures. What we found was that respondents had similar accuracy scores, completion times, and subject satisfaction measures regardless of the questionnaire interface design that they used. However, in looking at the navigational data, the different interface designs did affect the ways that respondents completed the questionnaire.

 

It was found that respondents in the form-based group were more likely to answer questions in a non-linear sequence. We believe that because this group had contextual information, that is, they had the entire page of questions for each section available to them, they were more likely to fill in questions as they located them within the organizational records. The item-based group answered the questions more linearly which may indicate that they knew exactly where to find the answers or that they were searching the organizational records each time they went to the next question.

 

One observation regarding the percentage of time that was spent on organizational record accesses versus questionnaire item accesses was that respondents in both groups seemed to spend considerable time at the beginning learning where information was located in the organizational records before they started to answer question items. During the first and second time intervals, respondents were mainly accessing organizational record pages. The form-based group began to answer questions earlier than the item-based group, starting in the third time interval. This supports the idea that respondents who have an overview of the questions can proceed to use the organizational records efficiently by answering questions as they find them during exploration of the records. By contrast, respondents using the item-based interface tended not to begin to answer questions until after the third interval. We speculate that since they did not know what to look for, they needed to spend additional effort learning the structure of the records. This is also reflected in the fact that the item-based group had more organizational record-to-organizational record traversals than did the form-based group. For the form-based group, they had less repeated searching through the organizational records. The item-based group needed to spend additional “familiarization time.”

 

The results that were obtained from this study show that the interface design significantly affects how respondents go about completing a CSAQ for an organization. The questionnaire and organizational records that were used for this study were straightforward and easy to follow. There were only 8 pages of organizational records and 41 question items. We expect that as records and questionnaires get larger and more complex, the navigational patterns found will affect accuracy scores, completion times, and subjective satisfaction. Future research should include questionnaires with skip patterns and larger sets of organizational records.

 

In this study, we examined how respondents approached an organizational CSAQ when they had no prior experience with either the questionnaire or the organizational records. Future research should also investigate changes as respondents gain experience of time with the records and/or the CSAQ.

 

References

 

Dillman, D. A. (2000). Mail and internet surveys (2^nd ed.), New York, N.Y.: John Wiley & Sons.

 

Inselberg, A. (1985). The plane with parallel coordinates. The Visual Computer, 1:69--91, 1985.

 

Norman, K.L., Friedman, Z., Norman, K.D., and Stevenson, R. (2000). Navigational issues in the design of on-line self-administered questionnaires. Laboratory for Automation Psychology technical report. LAP-TR-2000-1.

 

Manaster, C. (2000), Data Loom website. http://www.concentric.net/~Manaster/software/dataloom/index.html

 

Acknowledgements

 

This work was funded in part by a grant from the Statistical Research Division of the U.S. Bureau of the Census (Contract #43YABC917123). We thank Kent Marquis, Beth Nichols, Betty Murphy, and others at the Census for their guidance and direction in this research. We would also like to thank Ben Shneiderman and Catherine Plaisant from the Human-Computer Interaction Laboratory at the University of Maryland for their input

 

Appendix A

 

Questions in Each of the Four Sections of the Survey on Student Organizations

 

Information

 

1.      What is the name of the organization? (Name)

1.      Who is the president of the organization? (Name)

2.      When was the organization first formed? (Date)

3.      Does the organization have a savings account in a bank? (Yes/No)

4.      When were elections held for the officers of the organization? (date, month/day)

5.      Did the organization request and get funds from the Student Government? (Yes/No)

6.      Did the organization participate in the "First Look Fair" for student organizations? (Yes/No)

7.      Has the organization ever been disciplined or censured by the University? (Yes/No)

8.      Does the organization use office space in the student union? (Yes/No)

9.      Does the organization have a Web site? (Yes/No)

10.  If “Yes” to question 10: URL is _______

 

Activities

 

11.  How many members belong to the organization? (Enter number)

12.  How many events were sponsored during the year? (Enter number)

13.  How many members ran for offices in the last election? (Enter number)

14.  How many organizational meetings were held last year? (Enter number)

15.  What proportion of the events were fund raisers (give an estimate)? (scale from 0-1)

16.  What proportion of the members are active in the organization (e.g., attend meetings and events, contribute finanacially, etc.) (give an estimate)? (scale from 0-1)

17.  How many committees does the organization have? (Enter number)

18.  How many officers does the organization have? (Enter number)

19.  How many meetings must a member attend to remain on the active list? (Enter number)

20.  How many bylaws pertain to the unique activities of the organization? (Enter number)

 

Positions

 

21.  From the activities and positions of the statements please rate (scale from Academic/Scholarly to Sports/Entertainment)

22.  How diverse would you say the organization is from it membership list? (scale from Very Homogeneous to Very Diverse)

23.  Would you estimate that the membership of the organization is: (Predominantly Male, Slightly more male, About equal male and female, Slightly more female, Predominantly Female)

24.  Is this organization involved in community service projects?(Yes/No)

25.  In your estimation, how important is this organization to student life at the University of Maryland? (scale from Very Little Importance to Very Great Importance)

26.  In you opinion does this organization add to the quality of life of students on campus? (Yes/No)

27.  Would this organization be in anyway offensive to other student groups or to parents and alumni? (Yes/No)

28.  Does the organization seem open to change or is it closed? (Open/Closed)

29.  Are the organization's activities consistent with the Bylaws?(Yes/No)

 

Finances

 

30.  Do the minutes reflect participation of the membership or just the opinions of the elected officers? (Participation by members/Only elected officers)

31.  What is the total yearly budget for the organization?(Enter amount)

32.  What is the average contribution by members (give an estimate)? (Enter amount)

33.  How much money was raised by fund raiser events? (Enter amount)

34.  How much money was spent on advertising the organization? (Enter amount)

35.  Did the organization spend more than it collected through dues, contributions, and fund raisers? (Yes/No)

36.  What was the total amount spent on election campaigns by candidates for the presidency of the organization? (Enter amount)

37.  How much money was received by the organization from General Student Fees? (Enter amount)

38.  How much was spent on food and beverages for events by the organization? (Enter amount)

39.  What is the worth of office furniture, equipment, computers, supplies, etc. owned by the organization? (Enter amount)

40.  How much money did the organization have at the end of the year? (Enter amount)

 

 

.