Navigational Issues in the Design of On-Line Self-Administered Questionnaires:

The Effect of Training and Familiarity.

 

Kent L. Norman, Timothy J. Pleskac, & Kirk Norman

 

Human-Computer Interaction Laboratory

Institute for Advanced Computer Studies

Department of Psychology

University of Maryland

College Park, MD 20742-3255

 

 

 

Abstract

While completing a computerized self-administered questionnaire (CSAQ) for a business or an organization, a respondent may need to navigate between both the survey relevant database and records.  Dual navigation refers to the act of navigating information records to retrieve specific information, while simultaneously navigating through the survey.  In dual navigation the respondent may be either familiar or unfamiliar with the records on one hand and either familiar or unfamiliar with the survey on the other.  As familiarity with the records increase, respondents dual navigation may change.  In this study, two organizations were created:  the University of Maryland Break Dancing Club and the University of Maryland Knitting Club.  Additionally, each had a set of records for 1999 and 2000.  Participants were asked to complete the same organizational questionnaire twice for the 1999 Break Dancing Club and the 2000 Knitting Club, the 1999 Knitting Club and the 2000 Break Dancing Club, the 1999 and 2000 Knitting Club, or the 1999 and 2000 Break Dancing Club.  Accuracy, completion times, and navigational pattern were all recorded as well as subjective assessments.  Completion times and navigational pattern varied between conditions, reflecting a transfer of training from one questionnaire to the next. 

 

 

 

Conducting surveys of businesses and organizations using self-administered questionnaires pose unique challenges to surveyors.  Dillman (2000) notes, “Few survey undertakings are as difficult as defining, sampling, contacting, and obtaining responses to self-administered questionnaires from businesses or other organizations” (p. 323).  When both the survey and the records of the organization are on-line, respondents must often engage in dual navigation of both the records and the questionnaire  (Norman, Friedman, Norman, & Stevenson, 2000; Norman, Slaughter, Friedman, Norman, & Stevenson, 2000).  Dual navigation refers to the act of navigating information records to retrieve specific information, while simultaneously navigating through the survey.  Navigation is facilitated by familiarity with the system and its interface.  In dual navigation the respondent may be either familiar or unfamiliar with the records on one hand and either familiar or unfamiliar with the survey on the other. For example, a part-time employee at a small business may be given the task of completing a questionnaire on the general features of their employer.  Most likely the records they consult to answer the questions will be new and unknown to the part-time employee.   However, business and organizational surveys are often repeated annually or semi-annually, forcing the part-time employee to answer the same questions at every cycle, to maintain a file of essential data, and to become familiar with both the records and the survey.

As various researchers have noted (see Lazar & Pierce, 1999;  Synodinos, Papacostas, & Okimoto, 1994), the use of the World Wide Web and computerized self-administered questionnaires (CSAQ) has recently become a viable option to conduct survey research.  As such, it will be beneficial to investigate the psychological aspects of completing business and/or organization dual navigation CSAQs in order to design better interfaces.  Moreover, such research will add to our basic understanding of how respondents plan, learn, and perform question-answering tasks.

Using CSAQs and the Web can make the same part-time employee’s job of completing the questionnaire easier because surveys can be formatted to specific organizations, preferences can be stored for later use, or data from electronic files can be automatically copied.  Administratively, CSAQs can allow quickly dissemination, retrieval, and tabulation of questionnaire data. However, the use of CSAQs can pose a new challenge for survey designers as respondents search for answers to survey questions from different sources.  Respondents may answer questions from memory, from consulting paper records, or increasingly from digital records.  This research is an extension to the research of Norman, Friedman, Norman & Stevenson (2000a) and Norman, Slaughter, Friedman, Norman, & Stevenson (2000b).  Their research was conducted to assist the U.S. Bureau of the Census in developing principles and guidelines for interactive questionnaires.  Specifically, this research investigated the dual navigation of digital organizational records and a CSAQ and the transfer of training from one session to another session.

Dual Navigation

As previously mentioned, dual navigation refers to the act of navigating information records to retrieve specific information, which answers specific questions from the survey.   Dual navigation of records and survey suggest that performance on filling out the questionnaire can be influenced by both the design of the questionnaire and the database.[1]  As Norman et al. (2000a) note, “Both internal and external sources are organized in ways that dictate or influence in some way the order of retrieval of information” (p. 3).   When there is a match between the order of questions in a survey and the order or retrieval of information from an external source, processing is expected to be most efficient. 

When there is a mismatch, processing is less efficient and may result in errors (Norman et al., 2000a).  Norman et al. (2000a) demonstrated this by presenting participants with long distantia surveys partitioned in four ways:  a) whole/form-based, b) semantic/section-based, c) screen/page-based, and d) single item based.  Furthermore, the questionnaires were presented with or without an index.  Although neither completion times of the questionnaire nor subjective assessments differed among the eight versions, revision times[2] reflected ease of finding items in the structure of the survey and the use of an index to the sections of the questionnaire.

Performance during a dual navigation task can be influenced by the respondent’s familiarity with either the CSAQ or the records.  Norman et al. (2000b) classified four possible scenarios for experience of the respondents.  A respondent may be a) familiar with the organizational site, but new to the questionnaire, b) familiar with the questionnaire, but the organizational records are unfamiliar, c) familiar with both the questionnaire and the organizational records, or d) unfamiliar with both the questionnaire and the organizational records. 

As respondents interact with the records and the CSAQ, we hypothesize that they form a mental representation of the questionnaire and the database together.  This representation could lead respondents to complete the questionnaire in a nonlinear pattern the next time they saw the questionnaire and a similar record.  Figure 1 shows a schematic of  the first time participants complete a dual navigation task.  They do not have a mental representation of the record or the questionnaire.  Consequently, they answer the survey in a linear format and navigate through the record to answer each question.  Figure 2 illustrates the second time participants complete the dual navigation task with a database and questionnaire they are familiar with.  They are hypothesized to have formed a mental representation of the records and the locations of the specific questions to be answered.  Thus, they navigate through the survey in a nonlinear pattern as they answer several questions in different parts of the questionnaire that correspond to the current record they are viewing. 

 

 

 

An important question to be answered is what is the base for the mental representation of the participant.  For example, we hypothesized that respondents in our task would use the record as a base mental representation.  Thus, the questionnaire items are mapped onto specific areas of the record within the mental representation.  If respondents used the questionnaire as a base, the nodes of information would be mapped onto the questionnaire.  We hypothesized that subjects would use the database as a base for their mental representation because the record is the source of data and is the external store of information.  Participants should view the questionnaire as referring to the record, not the other way around.

Survey Design

            Lazar and Preece (1999) state, “Web-based surveys must follow many of the same rules for survey design as traditional paper surveys” (p. 63). For example, all survey researchers should verify that the questions are clear and well written by pilot testing the survey.  However, the use of CSAQs necessitates the understanding and implementation of the issues and factors involved in the computer interface environment (Norman et al., 2000a).[3]

            One aspect of interface design is the actual form of the questionnaire.  Norman et al. (2000a) conceptualized the design of questionnaires as existing on a design continuum.  At one end of the design continuum are form-based designs that present questionnaires as one long form in a scrollable window.  At the other end of the design continuum are item-based questionnaires that present only a single item at a time.  Additionally, each of the different designs from the continuum could also contain an index to the questions.  An index allows respondents to navigate through various sections of the survey by using a menu.  Norman and colleagues (2000a) found that when responding to the survey in a linear pattern the form of the questionnaire was extraneous to the task.  Furthermore, when responding in a linear form, indexes were found to be of little use to the participants.  In fact, they note that the indexes may have even served as a hindrance at times by disorienting the respondents when respondents only needed to go to the next item. 

The situation changes when respondents work through questionnaires in a nonlinear pattern.  Norman et al. (2000a) found that the interface needs to support nonlinear access to items, allowing respondents to reduce the amount of search for various required question items.   Thus, they concluded that a one long form with scrolling capability was superior to item-based forms when responding in a nonlinear format since respondents could quickly scroll up and down through the questionnaire.  Additionally, indexes produced significant differences in performance while completing the questionnaire in a nonlinear pattern. 

In a later study, Norman and colleagues (2000b) concluded that dual navigation of a CSAQ and digital records could produce a nonlinear completion pattern and that initially respondents spent time familiarizing themselves with the records and then answered the questions.  However, this behavior primarily occurred in a form-based questionnaire as compared with an item-based questionnaire.  An interesting result also occurred when Norman et al. (2000b) looked at the type of traversals respondents were making while completing the questionnaire.  Four types of traversals were defined by looking at consecutive access points in pairs:  a) an organizational record to another organizational record (within record), b) an organizational record to a questionnaire item (between record/survey), c) a questionnaire item to an organizational record (between survey/record), or d) a questionnaire item to another questionnaire item (within survey).  For each participant, the percentage of each traversal pair was calculated.  They found that participants who completed a form-based questionnaire had a much higher percentage of within survey traversals suggesting a nonlinear completion pattern (i.e., searching through the survey questions that were answerable dependent on the digital record they were currently viewing).  Furthermore, the item-based questionnaire had more within record traversals, signifying a linear completion pattern of the survey. 

Current Study

            The current study further investigated the effect of familiarity on completing form type questionnaires while navigating between organization records and an organizational survey.  Two organizations with digital records were created:  the University of Maryland Knitting Club and the University of Maryland Break Dancing Club.  Furthermore, two different years of records were created (1999 and 2000) for each organization.  Finally, a CSAQ made up of 41 questions was developed to ask questions pertaining to student organizations at the university.

            In the first part of the study, participants were asked to complete the questionnaire for either the 1999 Break Dancing Club or the 1999 Knitting Club, while navigating though their respective records.[4]  Completion time and type of traversals were not expected to differ between these first questionnaires.  Respondents then completed a second questionnaire.  Half of each group completed the questionnaire for the 2000 Break Dancing Club and the other half completed the questionnaire for the 2000 Knitting Club questionnaire.  All participants are predicted to complete the second survey in a faster time than their first one.  However, the participants who completed a questionnaire for the same organization in the year 2000 (similis [5]group) were predicted to have a faster completion time than those that completed a questionnaire for a different organization in the year 2000 (distantia[6] group).  Furthermore, the similis group was expected to have significantly more within survey traversals in the second survey than the distantia group.  This would indicate a more nonlinear completion pattern. 

Method

Participants

            Eighty-six undergraduates, from an introductory psychology course at the University of Maryland-College Park, participated in the experiment for course credit.   The participants ranged in age from 17 to 24, with a median age of 18. 

Materials

Computer.  The study was administered to participants on either a Macintosh G3 or G4 computer with dual Apple Studio Display 15-inch flat screen monitors.  Figure 3 shows a photograph of the experimental set-up.  The record was viewed on the left monitor and the CSAQ was viewed on the right monitor. 

Records.  The organizational records were created as a set of Web documents about a fictitious University of Maryland Break Dancing Club for the years 1999 and 2000.  Additionally, a set of organizational records was created for the fictitious University of Maryland Knitting Club for the years 1999 and 2000. 

            The records consisted of eight scrollable Web pages.  Each page contained information on one of the following eight topics:  introduction to the club, officers, events, committees, members, minutes of meetings, budget and financial information, and bylaws.  An index in a frame on the left side of the browser window provided links to each section.  Figure 4 and Figure 5 show the Introduction page for the Break Dancing Club and the Knitting Club, respectively.

 

CSAQ. The questionnaire constructed for this study is similar to that used in the Norman et al. (2000b) study.  The questionnaire had 41 questions, arranged into four topic sections each containing ten questions with one extra question that asked about the URL for the organizational records pages. The four topic sections were labeled:  information, activities, positions, and finances. The majority of the answers were directly available in the organizational records.  Some questions required that the participant calculate answers from several pieces of information available in the records.  For example, How many officers does the organization have? The participant must count the number of officers from a list in the organization records since the answer to the question is not explicitly written.  A few questions could not be calculated precisely, but instead the participant needed to make an approximation using the organizational records.  The questionnaire was created independently from the organizational records.  The full questionnaire is shown in Appendix A and Figure 6 demonstrates how the respondent saw the questionnaire. 

 

            The interface design for the CSAQ was a form-based design with an index on the left side of the browser window.  The index listed:  a) an introduction section with directions on how to navigate and complete the survey, b) four topic sections links to questionnaire items, and c) a link marked Done.  All the questions from each topic section (10 questions) were presented as a single Web page.  The participant was able to view all the questions associated with one of the topic section in a section by clicking on the topic links in the adjacent index.  Figure 3 shows part of the survey on the screen.

Procedure

                Participants were randomly assigned to participate in the experiment in either a similis group ((Knitting Club-1999, Knitting Club-2000) or (Break Dancing Club-1999, Break Dancing Club-2000)) or a distantia group ((Knitting Club-1999, Break Dancing Club-2000) or (Break Dancing Club-1999, Knitting Club-2000)).  The participant, prior to beginning the experiment, read the directions and completed an informed consent form. Instructions were then given about how to operate the browser windows, particularly how to use the records in the left-hand monitor and complete the questionnaire in the right hand monitor.

            Initially, participants completed a pre-questionnaire demographics form requiring age, gender, level of familiarity with computers, knowledge of the Internet, and previous experience with surveys.  Following the pre-questionnaire, participants were required to complete the organizational questionnaire, answering questions by navigating the organizational records.  Following the first organizational questionnaire, participants then completed their second organizational questionnaire.  After completing the organizational questionnaires, participants answered a brief post questionnaire asking for their subjective assessment of a) the organizational questionnaires and navigational tools, b) the organization of the records, and c) computerized questionnaires in general. 

Results

Pre-Questionnaire.  Participants in this study rated themselves as having moderate to high computer skills.  On a 9-point scale (1 = no experience, 9 = very experienced) the mean rating of overall experienced with computers was 7.03(1.37); use of the WWW, 7.10 (1.50); use of email, 7.71 (1.32).  They also rated themselves as having moderate experience in filling out surveys with a mean of 5.20 (2.23).

Accuracy.  An accuracy score of the information entered in the surveys was obtained.  This allows a check to be run on participants to assure that they were conscientious about answering all of the questions.  Participants scoring below 25% were removed from the results.  Nine participants were removed.  After removing their data, the rest of the data were analyzed using the seventy-seven remaining participants.  Additionally, the data from the four groups a) Knitting Club-1999, Knitting Club-2000, b) Break Dancing Club-1999, Break Dancing Club-2000, c) Knitting Club-1999, Break Dancing Club-2000, and d) Break Dancing Club-1999, Knitting Club-2000) were collapsed into two groups.  The similis group consisted of a) Knitting Club-1999, Knitting Club-2000 and b) Break Dancing Club-1999, Break Dancing Club-2000.  The distantia group consisted of c) Knitting Club-1999, Break Dancing Club-2000 and d) Break Dancing Club-1999, Knitting Club-2000.  There was not a significant difference for accuracy between the distentia and similis groups ( F(1,75) = 2.80, p > .05).  Within the groups, there was no significant difference in accuracy for completing either the first or second questionnaire ( F(1,75) = 1.24, p > .05).  Finally, there was no interaction for accuracy between group type and completing a questionnaire for the first or second time ( F(1,75) = .623, p > .05).

Completion Time.  As expected, there was a significant difference in completion times for the first and second survey ( F(1,75) = 292.66, p < .01).  Participants spent significantly less time on the second survey than the first.  However, there was no main effect for completion time for the similis/distantia groups ( F(1,75) = .75, p > .05).  Additionally, there was no interaction for completion time between group type and completing the first and second survey ( F(1,75) = 1.54, p > .05). 

 

            Navigational Access Patterns.  As previously noted, four types of traversal pairs were defined for dual navigation of questionnaires and records:  a) an organizational record to another organizational record (within record), b) an organizational record to a questionnaire item (between record survey), c) a questionnaire item to an organizational record (between survey record), or d) a questionnaire item to another questionnaire item (within survey).  For each participant, the percentage of each traversal pair was calculated. 

            There was a significant difference in the within survey traversals for group type ( F(1,75) = 5.73, p < .05).  Furthermore, the within survey traversals increased when the respondent completed the second survey ( F(1,75) = 27.72, p < .01).  The increase was dependent on the group type, as demonstrated by the interaction between group type and first or second survey ( F (1,75) = 8.31, p < .05).  Participants who took similis questionnaires were more likely to have an increase in within survey traversals in the second survey than those who took distantia surveys.

            The within record traversals decreased when the respondent completed the second survey ( F(1,75) = 57.85, p < .01).  However, there was no significant difference in the within record traversals for group type ( F(1,75) = .97, p > .05), nor was there an interaction ( F (1,75) = 2.13, p > .05). 

 

 

 

            The between record to survey traversals are perfectly complemented by the between survey to record traversals.  For every time a person navigates from the survey to the record, they have to navigate back from the record to the survey.  Thus, the results should and are complementary.  There was a significant difference in the number of between survey-record and between record-survey traversals for group type (F (1,75) = 7.11, p < .01 and F (1,75) = 5.77, p < .05).  There was no significant difference between the first and second survey in either the between survey-record or record-survey traversal ( F(1,75) = .001 and F(1,75) = .004, p > .05).  Finally, there was an interaction for both types of traversals ( F(1,75) = .5.27, p < .05 and F(1,75) = 5.528, p < .05).  This suggests that the levels between survey-record and record-survey traversals during the second survey were dependent on the group type.  In fact, the distantia group did make, on average, more of these traversals the second time.

            Post Questionnaire.  After completing each questionnaire (first and second), participants were asked to respond to several questions using a 9-point scale (  1 was very hard and 9 was very easy).  They were asked about moving through the survey, moving through the records, finding information in records, the organization of the survey, the organization of records, and to rate the overall rating of completing the questionnaire (see Table 1). 

There was no significant difference between distantia and similis or between the first or second questionnaire for moving through the survey (F (1,74) = 1.60 and F (1,74) = .017, p > .05, respectively).  Additionally, there was no interaction between the group type and the first and second questionnaire for moving through the survey (F (1,74) = .017, p < .05). 

There was a significant difference between distantia and similis when asked about ease of moving through the record (F (1,74) = 5.33, p < .05).  Additionally, respondents found that it was significantly easier to move through the second set of records (F(1,74) = 6.50, p > .05).  Finally, there was no interaction between the two factors (F(1,74) = 1.44, p > .05). 

There was no difference in respondents’ ratings of finding information in the records between distantia and similis (F(1,74) = .40, p > .05).  However, there was a significant difference between the first and second questionnaire for the rating of finding the information in the records (F(1,74) = 6.82, p < .05).  There was no significant interaction between the two factors (F (1,74) = .056, p > .05). 

Between distantia and similis, the ratings for overall ease of completion were not significantly different (F(1,74) = .31, p > .05.  However, respondents did significantly rate the second survey as overall easier to complete ( F(1,74) = 4.56, p < .05).  There was no significant interaction between the two factors (F(1,74) = 1.75, p > .05).

Respondents also rated the organization of the survey and the records (1 poorly organized to 9 well organized).  Not surprisingly there was not a significant difference in ratings for the organization of the questions of the survey for either between groups or between the questionnaires, nor was there a significant interaction (F(1,74) = .018, F(1,74) = .17, and F (1, 74) = .011, p > .05, respectfully).  In addition, there was not a significant difference in ratings for the organization of the records between group type or between the questionnaire, nor was there a significant interaction (F(1,74) = .12, F(1,74) = 1.20, and F (1, 74) = .59, p > .05, respectfully).

Discussion

            Holding the interface design constant for the questionnaire as a form allows several conclusions to be drawn from the navigational patterns of respondents as they become familiar with the records and the questionnaire.  When participants were presented with the task of completing a second survey, they transferred what they had learned the first time about navigating records and questionnaires and completed the second questionnaire in a faster time.  Furthermore, respondents were faster completing the second questionnaire for the Similis Group as compared to the Distantia Groups.  This indicates that respondents’ familiarity was influenced not only by previous experience with the similar questionnaire, but also by a previous experience with the same organizational records.

            The respondents’ navigational behavior also supports these conclusions.  Respondents transferred what they learned about the layout of the digital records (external memory source) when they navigated a similar set of digital records to complete the survey.  Their transfer of training might best be understood as forming a mental representation of the data records and how the questions map onto the nodes of the data record.  They completed the CSAQ in a nonlinear manner because the layout and organization of the digital records did not match the layout and organization of the CSAQ and the respondents had an understanding or mental representation of this from their previous experiences with the CSAQ.  Moreover, they were able to complete the questionnaire in a nonlinear format because the form based design with an index allowed them to do so.  This supports Norman and colleagues (2000a) suggestion that respondents find it useful to complete the CSAQ in a nonlinear formant and the interface should support such behavior.  

            Curiously, the Knitting Club Organization records were not all that dissimilar from the Break Dancing Club Organization records.  The content of the records was changed, but the format stayed the same.  In fact one respondent noted in her comments that “…it is curious that the knitting club listens to break dancing music while they knit.”  Although the records were generally all the same format, respondents seemed to treat them as entirely new and navigated them as if they were new. 

            These results can begin to help guide interface designs in the implementation of on-line questionnaires.  First, the design of the CSAQ should allow respondents to complete it in a nonlinear pattern.  As they become familiarized with the records and the survey and form a mental representation of the task, they begin to complete the CSAQ in a nonlinear format.  If the design of the questionnaire had been item-based, respondents would not have had the opportunity to complete the survey in a nonlinear form.  Second, data records should be treated as an external source of memory, which can influence the completion of the questionnaire.  This might be best understood by looking at the rating of the ease of moving though the records as compared to navigating the survey.  As was shown, respondents found it easier to move through a organizational records when they were from the same organization, but a different year.  Additionally, respondents found it easier to navigate the records during the second survey, regardless of the organization.  Respondents’ ratings of the ease of navigating the survey were not significantly different between groups or between questionnaires. 

            In future research, one could try to empirically study the transfer of training between surveys.  For example, using the same organization records, does a person transfer what they learned on filling out a general survey on the records to completing another different survey on the same organization?  Furthermore, research as to when the survey is used as a base in the mental representation versus when the records are used as the base might be of benefit, allowing designers to take advantage of knowing the mental representation and structuring the interface around it.

           

Works Cited

Dillman, D. A. (2000).  Mail and Internet Surveys (2^nd ed.), New York, NY:  John Wiley & Sons.

Lazar, J. & Preece, J. (1999).  Designing and implementing Web-based surveys. Journal of computer inforation systems, 39, 63-67.

Norman, K. L., Friedman, Z., Norman, K., & Stevenson, R.  (2000a).  Navigational issues in the design of on-line self-administered questionnaires.  Laboratory for Automation Psychology technical report.  LAP-TR-2000-1.

Norman, K. L., Slaughter, L., Friedman, Z., Norman, K., & Stevenson, R.  (2000b).  Dual navigation of computerized self-administered questionnaires and organization records.  Laboratory for Automation Psychology technical report.  LAP-TR-2000-2.

Synodinos, N. E., Papacostas, C. S., & Okimoto, G. M. (1994).  Computer-administered versus paper-and-pencil surveys and the effect of sample selection.  Behavior Reseach Methods, Instruments, & Computers, 26, 395-401.

Acknowledgments

This work was funded in part by a grant from the Statistical Research Division of the U.S. Bureau of the Census (Contract #43YABC039062).  We thank Kent Marquis and others at the Census for their guidance and direction in this research.
Table 1

Mean Ratings from Post Questionnaire Completed After Each Survey (Standard Deviations in parentheses)

Item	Condition	Mean  of Survey 1	Mean  of Survey 2
Ease of moving around survey.	Distantia	7.6 (1.6)	7.6 (1.2)
	Similis	7.2 (1.5)	7.2 (1.4)
Ease of moving around the records.	Distantia	7.3 (1.5)	7.5 (1.2)
	Similis	6.4(1.8)	6.9 (1.5)
Ease of finding information in the records.	Distantia	5.3 (1.7)	5.8 (1.7)
	Similis	5.1 (1.8)	5.5 (1.8)
Organization of questions and sections of survey	Distantia	6.6 (1.7)	6.5 (1.7)
	Similis	6.6 (1.6)	6.6 (1.7)
Organization of information in records	Distantia	5.7 (2.0)	6.1 (1.8)
	Similis	5.7 (1.8)	5.8 (1.6)
Overall ease	Distantia	6.0 (1.7)	6.1 (1.8)
	Similis	5.9 (1.5)	6.6 (1.6)