User Frustration with Technology in the Workplace

Jonathan Lazar1, Adam Jones1, Katie Bessiere2, Irina Ceaparu3, and Ben Shneiderman3



1Department of Computer and Information Sciences, &

Center for Applied Information Technology, & Universal Usability Laboratory

Towson University, Towson, Maryland, 21252


2Human-Computer Interaction Institute

Carnegie-Mellon University

Pittsburgh, PennsylvaniaA,  ??????15213

3Department of Computer Science, Human-Computer Interaction Laboratory,

Institute for Advanced Computer Studies & Institute for Systems Research

University of Maryland, College Park, Maryland 20742




Frustration, Usability, Technology Acceptance, Cost justification, Interface design, User satisfaction, Human-Computer Interaction



            When hard to use computers that are hard to use cause users to become frustrated, itthis can affect workplace productivity, user mood, and interactions with other co-workers. Previous research has examined the frustration that graduate students and their families face in using computers. To learn more about the causes and effects of user frustration with computers in the workplace, we collected modified time diaries were collected from 50 workplace users, who spent an average of 5.1 hours on the computer. In this experiment, users reported wasting on average, 42-4328-35% of their time on the computer due to frustrating experiences. The causes of the frustrating experiences, the time lost due to the frustrating experiences, and the effects of the frustrating experiences on the mood of the users are discussed in this paper. Implications for designers, managers, usersinterface designers, and employees a, information technology staff, and policymakers are discussed.




A September 2001 recent report released byof the National Telecommunications and Information Administration reports that as of September 2001, 56.7% adults (employed and over age 25) in the United States use a computer in the workplace (NTIA, 2001).  In addition, 81% of those employed in managerial and professional jobs, and 71% of those in technical, sales, and administrative support jobs utilized computers as part of their work environment.  This indicates that in ‘white-collar’ jobs, computer use is becoming very prevalent.

With the rising ubiquity of computer usage in American society in the home, school, and workplace, research has begun to focus on the possible consequences of such use.  Research on computer anxiety, attitudes, and frustration has shown that a disturbing portion of computer users suffer from negative affective reactions towards the computer, which can subsequently affect whether or not they use the computer, and whether or not they use the computer effectively.  Research on frustration, both in individuals and organizations, has shown that frustration can lead to maladaptive behaviors that can subsequently lower effective goal-oriented behavior.  In addition, research has shown that between one third and one half of the time spent in front of the computer was lost due to frustrating experiences -- when considering both the time it took to fix the problem and any additional time that was lost due to the problem (Ceaparu, Lazar, Bessiere, Robinson, & Shneiderman, 20032).

Because computers are so prevalent in organizations, it is important to examine the role of computers in the organization and the possible consequences arising from their use.  In this experiment, 50 workplace users recorded their frustrations with computers through the use of modified time diaries. There are solutions to the causes of user frustration.—these are not impossible to solve. However, the first step is to understanding the causes themselvesof user frustration, which can lead to experimental testing of improved interfaces to address these frustrations, and then implementation of these solutions in industry. Computers play an important role in affecting the performance of individuals within organizations, therefore, this research should be of great interest to businesses and other organizations, because improved interfaces can improve the bottom-line and corporate profit.


Background Research


Frustration is often defined in different ways, making the subject itself somewhat ambiguous creating an ambiguity that surrounds the term.  Frustration was first introduced by Sigmund Freud as a concept both external and internal in nature and related to the concept of goal attainment.  Frustration occurs when there is an inhibiting condition which interferes with or stops the realization of a goal.  All action has a purpose or goal whether explicit or implicit, and any interruption to the completion of an action or task can cause frustration.  For Freud, frustration included both external barriers to goal attainment and internal obstacles blocking satisfaction (Freud, 1921). 

This concept of frustration as a duality is continued in the analysis of frustration as both cause and effect (Britt & Janus, 1940).  As a cause, frustration is an external event, acting as a stimulus to an individual and eliciting an emotional reaction.  In this case, the emotional response is the effect, and the individual is aroused by this external cause and a response is often directed towards the environment. 

Dollard et al. (1939) define frustration as “an interference with the occurrence of an instigated goal-response at its proper time in the behavior sequence” (Dollard, Doob, Miller, Mowrer, & Sears, 1939).  Because an instigated goal response entails only that the goal be anticipated, frustration is due to the expectation and anticipation of a goal, not the actual attainment of the goal (Berkowitz, 1978).  If the goal is unfulfilled, frustration is experienced because satisfaction was not achieved and hopes were suddenly thwarted.  The thwarting or hindrance -- terms often used synonymously with frustration -- is not limited to the actual activity in progress, but relates to what the individual is expecting (Mowrer, 1938a). 

Frustrations, in all cases, are aversive events (Ferster, 1957) having as their main defining feature the element of a barrier or obstruction.  This barrier can take the form of an actual barrier, or an imaginary one such as the response to anticipated punishment or injury (Mowrer, 1938b).  A frustrating situation, then, is defined as any “in which an obstacle – physical, social, conceptual or environmental – prevents the satisfaction of a desire” (Barker, 1938).  These blocks to goal attainment may be both internal and external (Shorkey & Crocker, 1981), similar to the duality proposed by Freud.  Internal blocks consist of deficiencies within the individual such as a lack of knowledge, skill, or physical ability.  External blocks could include the physical environment, social or legal barriers such as laws or mores, or the behavior of other people.


Factors Affecting Level of Frustration

The level of frustration experienced by an individual clearly can differ depending on the circumstances surrounding the frustrating experience and on the individuals themselves.  One major factor in goal formation and achievement is goal commitment, which refers to the determination to try for and persist in the achievement of a goal (Campion & Lord, 1982).  Research on goal theory indicates that goal commitment has a strong relationship to performance and is related to both two factors:  1) the importance of the task or outcome and 2) the belief that the goal can be accomplished (Locke & Latham, 2002).

Individuals will have a high commitment to a goal when the goal is important to them and they believe that the goal can be attained (Locke, 1996).  The importance of the goal, in addition to the strength of the desire to obtain the goal (Dollard et al., 1939), will affect the level of goal-commitment as well as the strength of the subsequent reaction to the interruption.  Self-efficacy, the belief in one’s personal capabilities, can also affect goal commitment (Locke & Latham, 1990) in that the belief about how well a task can be performed when it involves setbacks, obstacles, or failures may affect how committed individuals are to that goal (Bandura, 1986). 

Judgments of efficacy are related to the amount of effort expended, how long they persist at the task, and resiliency in the case of failure or setback (Bandura, 1986, 1997b).  Self-efficacy influencesaffects emotional states as well; how much stress or depression people experience in difficult situations is dependent on how well they think they can cope with the situation (Bandura, 1997a).  The level of frustration that people experience, therefore, would be affected influenced by how important the goal was to them, as well as how confident they are in their abilities.  “Because goal-directed behavior involves valued, purposeful action, failure to attain goals may therefore result in highly charged emotional outcomes,” (Lincecum, 2000) including frustration.

Cultural factors can also play a role in the level of frustration experienced by individuals when coming across obstacles to their path of action.  Social Learning Theory (Bandura, 1973) states that “rather than frustration generating an aggressive drive, aversive treatment produces a general state of emotional arousal that can facilitate a variety of behaviors, depending on the types of responses the person has learned for coping with stress and their relative effectiveness” (p. 53).  Ways of coping with frustration are therefore learned from the society and are governed and constrained by the laws of a society. This can contribute to the level of frustration tolerance that individuals have, which is also affected by their prior experience and task specific self-efficacy.

According to Freud, it is not simply the nature of the frustrating incident that determines how people will react to it.  Rather, there is an interplay between the situation and the psychological characteristics of individuals.  The level of maturity of the individual also plays a part in the reactions to frustration (Barker, Dembo, & Lewin, 1965).  With maturity, there is an increase in the variety of responses to a situation employed by individuals, in the control of the environment, and in their ability to employ problem-solving behavior and plan steps to obtain the goal.  It would appear that learning, which is culturally determined, is a major factor in developing socially acceptable responses to frustration. 

Two additionalfinal factors that may influenceaffect the force of the frustration are the severity of the interruption and the degree of interference with the goal attainment (Dollard et. al. 1939).  All obstructions are not equally frustrating, and the severity and unexpectedness of the block will also factor into the strength of the response.  In addition, if individuals perceive that the thwarting was justified by socially acceptable rules, as opposed to being arbitrary, the frustration response may be minimized (Baron, 1977).  This may be due to the lowering of expectations because of extra information available to the individual. As stated above, it is the anticipation of success that affects frustration, and not the actual achievement of the goal.  Therefore, if individuals expect to be thwarted or have a low expectation of success, frustration may be minimized.


Responses to Frustration

The responses to frustration by individuals can be either adaptive or maladaptive (Shorkey & Crocker, 1981).  Adaptive responses are constructive and are implemented to solve the problem that is blocking goal attainment.  They may include preemptive efforts to avoid the problemblock, or once the block problemi is encountered, problem solving strategies to overcome or circumvent the problem.  Freud lists two types of adaptive responses:  1) transforming stress into active energy and reapplying this energy towards the original goal, and 2) identifying and pursuing alternative goals.  Maladaptive responses, on the other hand, are characterized by a lack of constructive problem solving and often make the frustrating experience worse by creating additional problems.  These maladaptive responses may be further categorized into objective (aggression, regression, withdrawal, fixation, resignation) and subjective (extrapunitive, intropunitive, impunitive) responses (Britt and Janus 1940). 


Organizational Frustration

            Organizational frustration has been defined by Paul Spector in a very similar fashion, and refers to an interference with goal attainment or maintenance that is caused by some stimulus condition within the organization (Spector, 1978).  It has been further narrowed to be defined as the interference with an individuals ability to carry out their day to day duties effectively (Keenan & Newton, 1984).  The sources of organizational frustration put forth by Spector include the physical environment (both natural and man-made), the organizational structure and climate, the rules and procedures of the organization, and individuals both in and out of the organization.  In addition, the concept of situational constraints (Peters & O'Connor, 1980) has been hypothesized to contribute to organizational frustration (Storms & Spector, 1987).  Spector (1978) suggested four reactions to organizational frustration:  1) an emotional response of anger and increased physiological arousal, 2) trying alternative courses of action, 3) aggression, and 4) withdrawal.  Of the behavioral reactions, only the secondfirst one – that of trying alternative courses of action to obtain the goal – is an adaptive response, while the other two three are maladaptive.  It is likely that the emotional reaction accompanies one of the three behavioral reactions, although the emotional reaction may be maladaptive by itself and become a further impediment to goal attainment.  Clearly, should an individual become frustrated, it is in the best interests of the organization to have the individual respond in an adaptive way and attempt to find another solution to the problem in a clear decisive manner.  Spector also put forth the idea that some mild forms of frustration may be seen as challenges rather than problems for some individuals, thus causing a motivational effect rather than a hindering effect and increasing the likelihood of an adaptive response rather than a maladaptive one.

             Behavior exemplifying two of the threetwo  maladaptive responses, in an organization, are described by Spector in his model.  Examples of withdrawal behavior in an organization could include the abandonment of a goal, absenteeism, or turnover.  Examples of organizational aggression include interpersonal aggression, sabotage, and withholding of output.  Both of these maladaptive responses are thought to lead into a decrease in job performance.  However, evidence for the frustration-performance link is mixed, as some cases of mild frustration are found to increase task-performance presumably due to increased arousal (Spector, 1975), whereas other studies find that frustration actually inhibits both task performance and learning of a new task.

            Other relationships with organizational frustration have also been tested.  In a sample of employed individuals, significant relationships were found between both self-reported sabotage and interpersonal aggression with level of frustration as measured by the Organization Frustration Scale (Spector, 1975).  Frustration was also found to be strongly correlated to a self-reported desire to leave the place of employment.  In another study of 401 employed engineers, Keenan and Newton found that organizational climate, role stress, and social support all correlated positively with environmental frustration (Keenan & Newton, 1984).  Additionally, they found that frustration was significantly related to angry emotional reactions, latent hostility and job dissatisfaction. 

Additional research has shown that organizational frustration is positively correlated with several negative behavioral reactions - aggression, sabotage, hostility and complaining, withdrawal, and intent to quit (Storms & Spector, 1987).  In an effort to examine the antecedents of the response choice (adaptive or maladaptive) Storms & Spector also tested for the moderating effect of locus of control, hypothesizing that individuals with an external locus of control would exhibit more counterproductive behavior during times of frustration than those with internal locus of control.  This hypothesis was supported, externals increased their counterproductive behavior as frustration increased, whereas the reactions of internal stayed constant.  [I1] 

Using the same Organizational Frustration scale, Jex and Gudanowski examined the role of self-efficacy beliefs and work stress (Jex & Gudanowski, 1992).  They found that individual efficacy beliefs were significantly negatively correlated with level of organizational frustration, indicating that those with less belief in their abilities at their job suffered more frustration than those with high efficacy beliefs.  However, they did not find that efficacy beliefs mediated the relationship between stressors and frustration, indicating that self-efficacy does not affect the level of frustration experienced due to external stressors such as situational constraints.


Situational Constraints 

The concept of situational constraints was introduced in 1980 by Peters and O’Connor in response to the perceived hole in the human performance literature (Peters & O'Connor, 1980).  They argued that it has long been assumed that the characteristics of the work setting play a role in performance, but it had never been empirically tested.  As such, they introduced a framework for the study of such situational constraints that might affect task performance, which takes into account the idea that individuals who are otherwise capable and motivated to perform may be inhibited by characteristics of the situation.  The situational factors that they believed to be relevant to performance, using a critical-incidents method, were job-related information, tools and equipment, materials and supplies, budgetary support, required services and help from others, task preparation, time availability, and work environment. 

As such, Peters and O’Connor hypothesized a direct link between situational constraints and performance, as well as a direct link between situational constraints and affective reactions such as job satisfaction or frustration.  In addition, they thought that the severity of the constraints would affect performance differentially, in accordance with expectancy theory (Vroom, 1964).  Persons who work in situations where severe constraints are the norm may develop the belief that additional effort on their part will not increase performance.  With regards to goals, this indicates that a long history of experience with situational constraints inhibiting progress towards a goal (in their model, performance) would reduce expectations of goal achievement and inhibit productive goal-oriented action.  For example, research has found that abilities are positively related to affective responses when situational constraints were low, but negatively related when they were high, indicating that the level of situational constraint on the individual affects their levels of job satisfaction and frustration (Peters, O'Connor, & Rudolf, 1980).

Subsequent research has examined the relationships between situational constraints and such dependent variables as task performance, job satisfaction, frustration, turnover, and goal commitment.  In addition, several studies reported the correlations between the negative affect caused by situational constraints and other outcome variables such as performance, job satisfaction, and turnover to demonstrate the direct link between affect and these variables, as well as the indirect link to situational constraints. 

In another study of  237 employed individuals in a range of managerial and non-managerial jobs, O’Connor et al. examined the relationship between situational constraints and the negative emotional reactions of job satisfaction and frustration (Peters, Chassie, Lindholm, O'Connor, & Kline, 1982).  Their results indicated that the higher the severity of overall situational constraints on the job, the greater the reported frustration and dissatisfaction.  In addition, they also found that the average constraint score was negatively correlated with level of effort (p<.001), motivation (p<.01) and organizational commitment (p<.001). 

Peters et al (1982) performed another experiment to examine the link between situational constraints, performance, and goal setting, hypothesizing that the link between goal difficulty and goal performance would be mediated by the presence of situational constraints.  Their analysis found that there was a direct link between situational constraints and performance, but that goal difficulty was unrelated to performance in the high constraint condition, but positively related to performance in the low constraint setting.  This indicates that the effect of goal difficulty and goal setting is dependent on the absence of constraints, showing that situational constraints also have an indirect effect on performance through this relationship.  In other words, highly motivated individuals (those who have set difficult goals) are unable to perform at the level anticipated in the presence of situational constraints.

Another study examined the link between goal commitment and performance, and the impact of situational constraints on this relationship (Klein & Kim, 1998).  They based this study on the idea that as employees become frustrated with constraints, they cannot perform as well as they feel they ought, and subsequently lose motivation because they no longer expect to perform well.  This also fits in with the idea that reduced expectations would lower goal commitment.  Their study of 105 salespersons found that situational constraints was negatively related to goal commitment, indicating that the presence of situational constraints negatively impacts the commitment to goals.

Phillips and Freedman performed another study of possible indirect links between situational constraints and motivation and satisfaction, and whether or not degree of perceived personal control over the job had any relationship (Phillips & Freedman, 1984).  They found a direct link between situational constraints and motivation and satisfaction.  They also found that in individuals feeling a high level of personal control, the perceived presence of high constraints actually increased the motivation and satisfaction of individuals in jobs that were characterized by a low motivating potential.  This suggests that individuals who perceive that they are in control of their job (internal locus of control) will perhaps find that mild constraints increase the challenge of an otherwise boring job.  This could be because these individuals continue to feel in control of the outcomes of the situation, and that a constraint does not necessarily inhibit the achievement of the goal and is seen as challenging rather than problematic. 

Locus of control was also examined as a factor in a field study by Storms and Spector (1987), where they found that situational constraints were positively related to perceived frustration, and that frustration was positively related to negative behavioral reactions.  As discussed above, locus of control was found to moderate the relationship between frustration and negative behavior, in that individuals with an internal locus of control did not increase their negative behaviors as a result of frustration, whereas externally located individuals did. 


Frustration and Situational Constraints

Clearly, situational constraints can have many consequences for an employee, ranging from frustration and dissatisfaction to negative behaviors.  Situational constraints not only have a direct effect on these negative consequences, but they also have an indirect effect through the affective reaction of frustration.  Constraints in the environment cause frustration in individuals, and the subsequent frustration also has a direct effect on satisfaction, motivation, performance, and the increase in negative behaviors.  A meta-analysis undertaken in 1993 on the effect of constraints and work related outcomes reports that situational constraints have a direct effect on turnover (.21), performance (-.14), frustration (.39), satisfaction (-.32), and commitment to the job (-.22) (Villanova & Roman, 1993).  In addition, frustration has an additional effect on performance (-.08) and satisfaction (-.31).  It would appear that both the situational constraints themselves, as well as the affective reaction towards it (which can be moderated by such factors as locus of control and efficacy beliefs) can influence organizational outcomes. 


Computer Anxiety                                                                                     

The reactions of people to computers have been studied extensively, particularly attitudes towards the computer (Loyd & Gressard, 1984; Murphy, Coover, & Owen, 1989; Nash & Moroz, 1997) computer anxiety (Cambre & Cook, 1985; Cohen & Waugh, 1989; Glass & Knight, 1988; Maurer, 1994; Raub, 1981; Torkzadeh & Angulo, 1992), and computer self-efficacy (Brosnan, 1998; Compeau & Higgins, 1995; McInerney, McInerney, & Sinclair, 1994; Meier, 1985).  Each of these variables, combined with the factors listed above, can affect how frustrated individuals will become when they encounter a problem while using a computer. 

The number of times a problem has occurred before can affect their perception of the locus of control, and therefore influence their reaction as well.  This may be related to anxiety, as people with low computer self-efficacy may be more anxious (Brosnan, 1998; Meier, 1985) and more likely to view the computer suspiciously and react with great frustration when something occurs, especially when they have run into it before.  Different levels of anxiety will affect performance when something unforeseen or unknown occurs, causing anxious people to become more anxious (Brosnan, 1998).  On the other hand, the level of experience may temper this if the prior experience increases computer self-efficacy (Gilroy & Desai, 1986) by lowering anxiety and reducing frustration when a problem occurs.  The perceived ability to fix problems on the computer, as well as the desire to do so may also affect levels of frustration.  If instead, these problems are seen as challenges rather than problems,, they may not be as frustrating, which is most likely directly related to level of prior experience as well as computer self-efficacy.



Computer Frustration

Frustration with technology is a major reason why people cannot use computers to reach their goal, hesitate to use computers, or avoid computers altogether. A recent study from the Pew Internet and American Life study found that a large percentage of people never go online, because they find the technology to be too frustrating and overwhelming (Pew, 2003). Currently, 42% of Americans do not use the Internet, in large part because they find it to be frustrating and confusing.  This is not surprising; previous research on user frustration found that users wastedr nearly one-third to one-half of the time spent on the computer, due to frustrating experiences Research on computer frustration (Bessiere, 2002; Bessiere, Lazar, Ceaparu, Robinson, & Shneiderman, 20032).

Unfortunately, computer applications are often designed with interfaces that are hard to use, and features that are hard to find. Even government web sites, which are supposed to provide easy access to government information for all citizens, are frequently hard to use and produce high levels of user frustration (Ceaparu, 2003; Hargittai, 2003). Frustration with technology can lead to wasted time, changed mood, and sufferedaffected interaction with colleagues. When users in a workplace are frustrated with their computers, it can lead to lower levels of job satisfaction (Murrell & Sprinkle, 1993). In some cases, user frustration with technology can even lead to increased blood volume pressure and muscle tension (Riseberg, Klein, Fernandez, & Picard, 1998)

 Research on computer frustration has shown that that computer self-efficacy and attitudes play a significant role in reducing the frustration levels in computing.  Level of comfort with the computer and the determination to fix a problem, which are associated with a high level of computer self efficacy, both appear as important factors in both the immediate experience of frustration as well as the overall frustration level after a session of computer use.  In this study, cIn the previous study on computer frustration, computer attitude variables mediated the experience of frustration but experience did not.  Simply using a computer, therefore, does not lessen user frustration;, rather it is one’s attitude towards it and comfort with it.

There is a measurable benefit to improved usability of user interfaces for lower user frustration (Bias & Mayhew, 1994). Many well-known companies, such as IBM, Staples, the National Football League, and Macy’s, do  focus on improving their interface design, which leads to measurable improvement ofin  the bottom line (Clarke, 2001; Tedeschi, 1999). For instance, when Macy’s made their web site search engine easier to use, the conversion rate (the rate at which site visitors are “converted” into buyers) went up 150% (Kemp, 2001). used feedback from users to improve their online registration pages, to make them easier to use. After improving the usability of the registration pages, the registration drop-off rate (the number of people who begin registering but fail to complete the registration) decreased by 53% (Roberts-Witt, 2001). After losing market share, AOL yielded to customer complaints and removed a majority of the pop-up advertisements from their service (Hu, 2002). Companies that have re-designed interfaces for log-on screens and for user forms have seen improvements in employee productivity that can be measured, in tens or hundreds of thousands of dollars (Nielsen, 1994).

 Having a proactive attitude towards the computer, particularly seeing the problems with the computer as challenges rather than problems, was shown to decrease frustration levels.

The consequences of negative attitudes towards the computer have not been studied extensively within organizational settings.  However, Murrell and Sprinkle examined the relationship between feelings of frustration and confusion about the use of computers and found that these were associated with lower job satisfaction (Murrell & Sprinkle, 1993).  This indicates that the consequences of negative attitudes towards computers may extend to the organizational level, as the literature on situational constraints would also suggest.


Organizational Frustration, Situational Constraints, and the Computer

            The growing ubiquity of computers and information technologies as part of the organizational environment suggest the increased role that equipment and tools, and situational constraints more generally, will play in both organizational level and individual level frustration.  The dearth of literature examining the causes and associated factors surrounding computer frustration make it difficult to hypothesize about the links between computer frustration and organizational frustration and other negative consequences.  However, the literature on situational constraints gives some indications for areas of future research.  For instance, we might speculate that the complexity of the computer as a tool leads to greater opportunities for situational constraints to produce organizational frustration.

Proposition 1:  The computer, as a piece of equipment and a tool necessary to accomplish many jobs, will contribute to levels of organizational frustration as a situational constraint.


 In addition, the increasing reliance on technical support adds another dimension to the idea of the computer as a situational constraint – insufficient support and help in the face of computer problems will also hinder the individuals’ progress towards their goals in the workplace. 

Proposition 2:  Increasing reliance on technical support both within and without the organization will serve as an additional situational constraint.


If the computer becomes a situational constraint, as well as the increased reliance on other individuals to keep this equipment running properly, all the attendant organizational consequences of situational constraints will therefore apply to the computer.

Proposition 3:  Frustration with computer problems will lead to decreased job satisfaction, increased organizational frustration, a decrease in performance due to both frustration and loss of time, a loss of motivation, and the exhibition of maladaptive goal-attainment behaviors.


However, as the literature on frustration indicates, particularly the literature on computer frustration, greater computer self-efficacy will lead to a tendency to view computer problems as challenges rather than problems, and will moderate the relationship between computer frustration and the resultant consequences.

Proposition 4:  Greater computer self-efficacy will lead to adaptive goal-attainment behavior and moderate the relationship between computer frustration and the resultant consequences.


Literature on the role of training and experience indicates that these lead to greater computer self-efficacy, but do not have a direct link to the consequences of negative attitudes towards the computer.  However, since training and experience increase computer self-efficacy, and computer self-efficacy moderates this relationship, training and experience are deemed necessary for the development of proactive attitudes.

Proposition 5:  Training and experience will increase computer self-efficacy.




            As the reliance upon computers as tools in organizations becomes more prevalent, it becomes more important for individuals to have positive attitudes towards the computer in order for them to have adaptive reactions to the problems that they are sure to encounter.  The computer, as a piece of equipment and a tool, is a situational constraint that has consequences for affective reactions to the organization and job, as well as behavioral consequences of decreased performance and possible interpersonal and organizational aggression.  As such, it is important to look at the possible factors that could lessen these maladaptive reactions, and find ways to lessen the impact of computer problems.  One such method might be to increase training and experience of users in order to heighten their computer self-efficacy and subsequently lower the negative consequences.  However, to ignore the problem of the computer would be detrimental to an organization, especially as technology becomes more complex and harder to learn.  As such, it is vital that organizations give the support to their computer users that is needed to lower the negative attitudes and frustration, which could have devastating organizational impact.




To learn more about user frustration with technology in the workplace, data was collected through the use of modified time diaries. With a modified time diary, usersUsers recorded data about their frustrations as the frustrations occuroccurred. Surveys would not be an appropriate data collection methodology for this research, since users trying to recall frustrations from their past experiences might over-estimate or under-estimate the level of frustration and the time wasted (Fowler, 1993). In addition, data logging cannot effectively measure frustration, since data logging would only work for system errors, or other occasions when the systems indicated an error state. There are many events that are frustrating for users (such as spam or pop-up advertisements), and occur when the system is operating in a correct state. This same methodology was used in the previous study of computer frustration in students (Ceaparu, Lazar, Bessiere, Robinson, and Shneiderman, 2003).


Subjects in this experiment study were encouraged to perform their typical work-related tasks, and record, as a part of their time diaries, any frustrating experiences. Tasks are not pre-assigned to subjects, because user frustration is correlated to the importance of the task (Bessière, Ceaparu, Lazar, Robinson, & Shneiderman, 2003). When tasks are important to users, users report higher levels of frustration than when tasks are not important. Pre-assigned tasks would therefore not accurately model the user frustration in an average workday. The following protocol was used:


1. Fill out demographic information (age, gender, computer experience, etc.)

2. Fill out a pre-session survey (noting current mood) (Appendix A)

3. Perform work-related computer tasks of their choosing, for a minimum of one hour total.

4. Fill out frustration experience forms, whenever the subject feels frustrated. These forms describe the cause, nature, and severity of the frustrating experience. (Appendix B)

5. Fill out a post-session survey (measuring frustration after the session ended) (Appendix C)

6. After completing the post-session survey, subjects fill out a reimbursement form and return all of the materials via postalsnail-mail to the researchers.






            Data collection took place from mid-2002 until 2003.


A total of 50 subjects took part in the research experimenstudy. Each of these subjects was a workplace user of computers, and each subject was paid $25 for their participation. The workplaces represented in this study include healthcare (15), law (3), education (8), information technology (11), non-profit-other (5), for-profit-other (2), government (3), and 3 subjects did not indicate their workplace. The average age of users was 35.95 years (with a range of 23 to 76 years old). The average number of years of computer experience was 2.38 years (with a range of less than a year, to 25 years of experience). A total of 149 frustrating experiences were reported, with each participant reporting between 1 and 6 experiences. Users recorded their experiences, in time diaries, for a period of 5.1 hours, on average. This paper reports the causes and severity of the frustration, highlighting the responses to frustration, as well as the time lost. A separate paper will address how the frustration impacted on the individuals mood and interaction with others.


Figure 1: Demographic information

Average age of subject

Average Experience for Subject

High / Low Age

High / Low Experience

36 years

2.38 years

(23 years / 76 years)

(0.5 years / 25 years)


WWeb browsing and ord processingemail and e-mail produced the largest number of frustrating experiences, probably reflecting that these applications were used most often (Table 1). Table 1 reports the applications that were the source of the frustrating experience. Often the frustrating experience affected the entire system, and was during activities that participants had previously completed successfully a number of times without error. There were several frustrating experiences involving moving data from one application to another type of application, such as email content into word processing and even moving content among similar applications, such as Word to WordPerfect. Many frustrating experiences were inhibiting but did not ultimately prevent the task from completion.




Table One: Summary and Demographic information from the study


Average age of users


35.94 years

(high was 76, low was 23)

Average computer experience

2.38 years

(high was 25, low was < 1)

Percent of total time lost


(35% including outliers)

Time lost outlier High (to error; to recovery)

540 minutes; 540 minutes

Time lost outlier Low (to error; to recovery)

0 minutes; 0 minutes

Most common problem source

Word Processing

Most common problem category

System-Wide Crashes

Most common solution


User knew how to fix from previous experience

Least common solution

Consulting Manual or Book

Most common level of frustration

9 Very Frustrating (Highest level)



  Participants came from a wide variety of backgrounds and work environments. There were 50 participants with a total of 149 frustrating experiences. With each participant reporting between 1 and 6 experiences. The most common response to errors was to reboot or restart the computer, implying that the problem would fix itself eventually.



Table On1eTwo: Applications that were the source of frustrating experiencesProblem Sources of reported incidents



Problem source

Frequency of problem sources



Web browsing




Other Internet Use


Video/Audio Software


Word Processing


Chat and Instant Messaging


File Browsers


Programming Tools


Spreadsheet Programs


Graphic Design Programs


Presentation Software


Database Programs








Table 2 reports the solutions that users took in the attempt to solve their frustrating experiences. Most participants were already familiar with the frustrating experience from previous experiences and know how to solve it (Table 2). Most solutions involved simply redoing the task either orafter a restarbooting and then redoing the task or twice in a row. Other solutions involved work-arounds and as a last resort finding help externally. The type of solution taken was independent of other demographic informationdifferences. Int is interesting to note that only one frustrating experience did the user consult a manual, and only in two experiences did the user consult online help.

1oneProblem sources were based on those listed in a previous study.  Often the problem affected the entire system, and was during activities that participants had completed successfully a number of times without error. There were several incidents involving moving data from one application to another type of application, such as email content into word processing and even moving content among similar applications, such as word Word to WordPerfect. Many problems were inhibiting but did not ultimately prevent the task from completion.








Table 23:  Solutions taken by participants



         Solution taken

Frequency of solutions



I knew how to solve it because it happened before


I figured out a way to fix it myself


I was unable to solve it


I ignored the problem or found an alternative


I tried again


I restarted the program


I consulted online help


I asked someone for help


I rebooted


I consulted a manual or a book






Note: some responses were left blank by participant



TIn this study, the time lost due to frustrating experiences was one of our key measuresd. In terms of time lost, one frustrating experience was considered to be an outlier. The one outlier frustrating experience was reported as 540 minutes to fix the problem, and another 540 minutes to recover the problem. The cause of the frustrating experience was a hardware problem, where the user reported that they would assign IRQs to hardware, and every time that the computer re-booted, the operating system would re-assign those IRQs. The user reported disabling the problem devices so that they could complete their current tasks. Due to the large amount of time wasted, we therefore felt that this one frustrating experience should be separated out as an outlier. The user reported two other frustrating experiences, but those were well within the typical range reported. The other frustrating experiences from the same subject are therefore included.. Users, in general, spend more time recovering from an incident than initially working through the incident. Both the initial time spent on responding to the frustrating experience, as well as the time to recover from any work lost due to the problem, contribute to the total time lost. The method for computing percentage of time lost is as follows:


       Percent Time Lost = (MS + MR) / MT


Where MS is  

percentage of time lost=

(minutes spent to solve the problem, MR is) +( minutes spent to recover from lostany work loss due to the problem, and MT is)


 total minutes spent on the computer


 (Ceaparu, Lazar, Bessiere, Robinson, and Shneiderman, 2003).


For each user, the amount of time lost to respond to the initial problem, as well as the time lost to recover from the problem, was added for all frustrating experiences reported by that user and then divided by the overall time spent by that user on the computer. The final figure represents the percent of total time lost by that user of the time that they spent on the computer. . Table 3 gives a sample of these data calculations from one user.






Table 3. Sample of time lost statistics for one user



Minutes Lost Spent tTo InitialSolve Problem

Minutes LostSpent to To Recover Lost Work

Total Minutes Lost

Total Minutes Spent on the Computer

Percent Time Lost tTo InitialSolve Problem

Percent Time Lost to Recovering Lost Work

Percent Total Time Lost

A sample user










The percentages for time lost dueto solve to initial the problem, time lost to recover lost work, and total time lost, after being normalized for each user, were then averaged over the population of 50 users (. The resulting averages are displayed in tTable 4). Each column in table 4 is calculated from the data itself and not from the previous numbers in the table. Therefore, percentage time lost is not exactly equal to the sum of the figures in the first two columns. FiguresPercentages are given for all users, and for all users minus the one outlier frustrating experience frustrating experience (see paragraph below).


One user reported an extremely long duration frustrating experience, which we considered to be an outlier. The one outlier frustrating experience was reported as 540 minutes to solve the problem, and another 540 minutes to recover lost work. The cause of the frustrating experience was a hardware problem, where the user reported assigning IRQs to hardware, and every time that the computer re-booted, the operating system would re-assign those IRQs. The user reported disabling the problem devices to complete the current task. Due to the large amount of time wasted, we therefore felt that this one frustrating experience should be separated out as an outlier. The user reported two other frustrating experiences, but those were well within the typical range reported. The other frustrating experiences from the same subject are therefore included (Table 4).



Most Participants were already familiar with problems from previous experience. Most solutions involved simply redoing the task either after a restart or twice in a row. Other solutions involved work arounds and as a last resort finding help externally. The type of solution taken was independent of other demographic information.



Table a4: Breakdown of time spent and lost by participants


Total Minutes Lost To Initial Error

Total Minutes Lost To Recover

Total Minutes Lost

Total Time Spent on the Computer

Percent Wasted Time for Errors

Percent Wasted time Recovering

Percent total wasted time









Table 3b: Breakdown of time spent and lost by participants (with Outliers)


Total Minutes Lost To Initial Error

Total Minutes Lost To Recover

Total Minutes Lost

Total Time Spent on the Computer

Percent Wasted Time for Errors

Percent Wasted time Recovering

Percent total wasted time










Table 43c.: NEW Breakdown of averageAverages for percent time lost due to initialsolve problem, average time to recover lost work, and average total time lost for all users time spent and lost by participants




Time Lost

 to Solve Problem Percent Wasted Time to Respond to Initial Problemfor Errors

Percent Time to Recover Lost WorkPercent TWasted time Lost Recovering

Percent Ttotal Time Lostwasted time

Total wWithout outlier




Total wWith outlier



43.72 %





Table 3d: NEW Breakdown of time spent and lost by participants (with Outliers)


Total Minutes Lost To Initial Error

Total Minutes Lost To Recover

Total Minutes Lost

Total Time Spent on the Computer

Percent Wasted Time for Errors

Percent Wasted time Recovering

Percent total wasted time







43.7 %



Table 3e: Time Statistics

Avg. min lost per user To Initial error

Most min lost by user to Initial error

Least min lost by user to Initial error

19.5 minutes

540 minutes

0 minutes




Table 5 reports the total minutes lost by application source. Problems with word processing cost participants the most amount of time in total (1225 minutes lost), followed by problems with email (666 minutes lost) (Table 5). This did correlate towith the applications that caused the highest number of frustrating experiences. The applications that were uncommon sources of frustrating experiences (such as programming tools, database software, and presentation software) often required more time per incident, as the problems were rare and complex to solve. Tables 3a-d represent important statistics with and without the outliers of the study. A and B were calculated by first adding up the times of all participants and then calculating percentages. C and D were calculated by finding the percentages for each participant and then adding up and averaging those results. Table 3e lists some other miscellaneous statistics.



Time lost was divided by the time to fix the problem and the time required to get back up to speed. Participants spent more time recovering from an incident than initially working through the incident. This includes recovering lost work and the time to reaccomplish the initial task. Time spent was based on start and stop times recorded by the user. The percentage is the amount of time that could have been utilized, had the incident not occurred.[ADAM: WE NEED TO SHOW TWO DIFFERENT ANALYSES HERE: WITH THE OUTLIER, AND WITHOUT THE OUTLIER]



Table 55: Breakdown of total minutes lost and average time minutes lost per frustrating experienceincident by application problem source

Problem source



Total minutes. lost

#  of fFrustrating experiencesE

Averageg minutes lost per FEfrustrating experience





Web Browsing




Other Internet Use




Word Processing




File Browsers




Video/Audio Software




Programming Tools




Graphic Design Programs




Database Programs




Chat and Instant Messaging




Presentation Software




Spreadsheet Programs












Note: FE=number of frustrating experiences


            Table 6 lists the specific causes of the frustrating experiences.            Problems with word processing cost participants the most amount of time, followed by problems with email. This did correlate to the two most common problem sources. The uncommon sources often required more time per incident as the problems were rare and complex to solve. In some cases minutes lost was also upinfluenced by to  external factors such as help desks and tech support. More often than not time would not have been saved with a backup copy of the data taken just before the error and restoring afterwards. Categories of frustrating experiences were based loosely on a previous study (Ceaparu, Lazar, Bessiere, Robinson, and Shneiderman, 2003) with minor changes to accommodate terms used by our participants (Table 6). Major categories were grouped by the behavior described in each frustrating experience. System crashes were the most commonly-reported frustrating experience, accounting for 21 of the 149, and were caused by specific programs as well as the operating system itself. While many of these problems are hardware-related or technical-related (such as printing problems and system freezescrashes), there were a number of frustrating experiences that were caused by interface-related issues (such as uncontrollable pop-up windows, hard to find features, and unpredictable behavior of application, and unclear error messages). For instance, there were 19 experiences with missing/hard-to-find/unusable features, 4 experiences with uncontrollable pop-up windows, and 5 experiences with unclear error messages. These interface-related causes of user frustration are easily solvable, when attention is paid to appropriate user interface design. As discussed in previous portions of the paper, when these interface improvements are made, it leads to improved user productivity and organizational profitability.






Table 66:  Specific causes of frustrating experiencesBreakdown of 5 main categories and many subcategories of problems



Operating System



Sending / Receiving Email and accessing attachments

{ 6 } Lost / Dropped Connections

{ 7 }

Missing / Hard to find Unusable Features

{ 19 }Unknown File Format

{ 1 }

System FreezeCrash Caused by Operating System

{ 21 }File Browser Operations

{ 2 }

Printing Problems

{ 10 }

User Kicked from System

{ 5 }Virus / Malicious Program

{ 1 }

Sending / Receiving Email and accessing attachments

{ 6 }Lost / Dropped Connections

{ 7 }

Program-Only Crash; System okApplication Crash

{ 11 }

File Browser Operations

{ 2 }System-Wide Crashes caused by OS

{ 21 }

Hardware Conflicts

{ 3 }

Multi user File Access and Permission Issues

{ 5 }Password Not working

{ 2 }

Uncontrollable Pop-up window

{ 4 }Browser Failure

{ 2 }

Buggy, Incorrect behavior of program

{ 10 }System-Wide crashes caused by Program

{ 4 }

Multitasking Failure

{ 1 }

Device Failures

{ 3 }

Password Not working

{ 2 }Multi user File Access and Permission Issues

{ 5 }

Plug-in Failure

{ 1 } Internet Login Failures

{ 3 }

Excessive Slow Operation

{ 8 }



Virus / Malicious Program

{ 1 }Local Network Connection Access Failure

{ 1 }

Browser Failure

{ 2 }Uncontrollable Pop-up window

{ 4 }

Unpredictable Response of program

{ 6 }Missing / Hard to find Unusable Features

{ 19 }



Local Network Connection Access Failure

{ 1 } Power Failure

{ 1 }

File Download Failures

{ 2 }

Unclear Error Messages

{ 5 }Unpredictable Response of program

{ 6 }



Power Failure

{ 1 }User Kicked from System

{ 5 }

Plug-in Failure

{ 1 }Internet Login Failures

{ 3 }

Installation Issues

{ 4 } Unclear Error Messages

{ 5 }





Application Crash that Froze the entire System

{ 4 }Buggy / Undesirable behavior of program

{ 10 }





Installation Issues

{ 4 }Unknown File Format

{ 1 }




Note: some responses were left blank by participant



The participants expresses strong eTables 7 and 8 display the data on the emotional reactions to the frustrating experiences (. Table 7) reports on how users felt after the frustrating experiences. For instance, in 60 of the frustrating experiences, users felt angry at the computer, in 34 experiences, users felt helpless/resigned, and in 15 experiences, users felt angry at themselves. It is important to note thatS since users may have more than one emotional reaction, these numbers for table 7 will add up to more than the 149 frustrating experiences reported.  Table 8 reports the level of frustration for each of the frustrating experiences. It is important to note thanUsing a 1 to 9 numeric scale, 106/149 of the frustrating experiences were reported to have frustration levels of 7, 8, or 9 , very high frustration levels, indeed(. Figure 1) displays the same frustration levels as table 8. These high levels of frustration can have an impact on the human bodyphysiological variables. For instance, in a previous study of user frustration, it wasresearchers found that when typical users get frustrated with their computer, it affects blood volume pressure (Riseberg, Klein, Fernandez, & Picard, 1998).Categories were based loosely on a previous study. Major categories were grouped by the behavior described in each incident. System crashes were the most commonly frustrating, and were caused by specific programs as well as the operating system itself. Most of these problems were non repeatable and subcategories involved more than one participant. For example all of the pop-up window errors did not belong to one person.  


Table 76:  User feelings per incident

Expressed Feeling

Number of


Angry at the computer


Angry at yourself


Helpless / Resigned


Determined to fix it






Note: Some participants had multiple feelings per incident




Table 8767:  Number of incidents for each level of frustration


Level of Frustration

Number of Frustrating Experiences

1 Not Very Frustrating
















9 Very Frustrating




Note: some responses were left blank by participant



























Figure 17: Bar graph of reported frustration levels








Unfortunately, it seems that these frustrating experiences are not rare occurrences that happen on an infrequent basis. From the frustrating experiences reported, users were asked to indicate whether this same event had occurred previously, and if so, how often (. Table 8)9 displays their responses related to the frequency of the problem.. For instance, for 25 of the frustrating experiences, users reported that the same event occurs more than once a day. For 21 of the frustrating experience, users reported that the event occurred several times a week. It is clear that users repeatedly face problems, and these problems interrupt their day.

Most users reported high levels of frustration, and some level of effect on their coworkers. The more frequent problems and the ones that took the most time were associated with higher levels of frustration. Unimportant tasks had lower levels of frustration.




Table 898: Frequency of problems

Frequency of problem

Number of reports

More than once a day


One time a day


Once a week


Once a month


Several times a week


Several times a month


Several times a year


First time it happened



Note: Some responses left blank by participant




From the results of this study, it is clear that user frustration is a problem in the workplace. Since uUsers lose largemore than 40%  quantities of their time, and these frustrationg experiences have an impact on both the individuals, and their time lost has an impact on the organizations. From an individual point of view, users waste a large amount of time, which impacts onslows their completion of work, limiting their time with family, friends, and co-workers. It also can affect their emotional state. TAnd these frustrating experiences also impactharm on the organizations which these individuals work for. Just think about how much more profitable a company would be if more of that time was productive. And this isn’t the workers slacking off or wasting their time—this is the users that are trying to complete their tasks, but are facing frustrating experiences, leading them to take more time on their tasks. The time wasted due to poor computer interface design is staggering. The time wasted has a large monetary value to organizations. by undermining productivity, lowering quality, and raising stress levels. The moneyinvestment in spent to improvinge user interfaces iswould yield large payoffs; several studies suggest more than justified by the monetary value of time saved. In some cases, the  that the cost of the interface improvement is made up 5, 10, or 20 times over (Bias and Mayhew, 1994).






It is clear from this study and previous work that user frustration is a major problem. In the previous study of 111 subjects, the subjects were university students, and their friends and family members. This current study was limited to workplace users. However, the results of the two studies were surprisingly similar.

Next Section: A Comparison of the Workplace data and the student data


Table 9 reports the The top 3 problem applications causing a frustrating experiencesources  from both the previous study and the current study. The applications causing the most frequent frustrating experiences for the student frustration studystudy  were web browsing, email and word processing. In the workplace frustration study, the top 3 application problems sources encountered by the users were the same, but in reverse order: word processing, email, web browsing.


The top 3 solutions taken by the participants to solve the problems that occurred in the student study were: for the student study - they knew how to solve it from previous experience, they figured out a way or they were unable to solve it. For the workplace study, the top 3 solutions cited were; for the workplace study -  that they knew how to solve it from previous experience, they rebooted, or ignored the problem/found an alternative.


Table 9:  Applications causing the largest numbers of frustrating experiences


Student Study-Causes of Frustration

Workplace Study-Causes of Frustration

1. Web Browsing

1. Word Processing

2. E-mail

2. E-mail

3. Word Processing

3. Web Browsing




In both studies, the levels of frustration wereis at the high end of the scale. For instance, 7,8, and 9 are the highest scores on the frustration scale, and in both studies, large percentages of subjects reported their frustrations being in that range. Levels 7, 8 and 9 of frustration are the top 3 levels for both the student study (7: 68, 8: 77, 9: 91) In the student frustration study, 63.3% (236 out of 373) of the frustrating experiences caused high levels of frustration. In the workplace study, 71.1% (106 out of 149) of frustrating experiences caused high levels of frustration. and the workplace study (7:23, 8:33, 9:50).  




The amount of time lost was also similar in the different frustration studies. Table 10 describes the time lost in the various studies, both with and without outliers. The numbers are very similar. In the previous study with students, there were two phases: self-reports and observations. These data points are listed separately. In the previous study with students, the average percentage of time lost ranged from 38.9% (for self-reports without the 5 outliers) to 50.1% (for self-reports with the 5 outliers). In this workplace study, the average percentage of time lost ranged from 42.7% (without outliers) to 43.7% (with outliers). We think that the difference in ranges was logical, due to the numbers of subjects involved in the two studies. In the previous student study, 111 subjects took part, whereas in this workplace study, only 50 users took part. With a larger number of users taking part, it logically follows that there will be more outliers, and therefore, a wider percentage spread.


Table 10109: Time lost in the variousthree studies, with and without outliers.


Average time lost per user

(with outliers)

Average time lost per user (without outliers)

Student study self-reports



Student study observations



Workplace study




From the analysis in the student study, we get the following results in terms of time lost: the average time lost per individual for UMD reports was 47.8% (37.9% without the outliers) and for Towson 53.1% (43.5% without the outliers); the average time lost of 50.1% (38.9% without the outliers) from self-reports and 49.9% (41.9% without the outliers) from observations.

From the analysis in the workplace study, the average time lost per individual is 28.1%.


When looking at the specific causes of the frustrating experiences categories and subcategories of problems that occurred, the student study finds that the top 5 were: error messages, timed out/dropped/refused connections, application freezes, missing/hard to find/unusable features, long download time. The workplace study finds that the top 5 were: OS crashes, missing/hard to find/unusable features, application crashes, hardware problems, buggy/undesirable behavior of program. Many of these problems (such as error messages, hard-to-find features, and undesirable behavior) are caused by poorly-designed interfaces, and therefore, can be solved with more usability testing and more user involvement in the interface development. It is interesting to note that some of the causes of frustration for the student participants, such as timed out/dropped connections, and long download times, did not appear as frequent frustrations for workplace participants. It is likely that the network connection at a workplace is of higher quality and speed, therefore less likely to cause frustration due to either response time or dropped connection. However, many of the software applications are the same, regardless of user population or location, and are highly likely to cause frustration.



ConclusionSolving the Causes of User Frustration


It is clearThis study with 50 workplace users adds to the growing evidence that user frustration is a major problem. Further studies with a narrower focus may isolate and measure contributing factors, but there is enough evidence to encourage change in the industry.

This study, in a new population of workplace users, helped to confirm the problem of user frustration. What can be done about this problem? BadImproving user interface design, causing user frustration, is a solvable problem is one clear opportunity because the payoffs will be immediate and benefit many users. To build better interfaces, more user involvement is needed in the interface design process. Designers should follow the interface guidelines that exist. AndUser tra users needs to be provided with more training and documentationwill also help, especially if it addresses problem solving strategies that will help build self-efficacy. Even small changes in the interface can make a big impact on user satisfaction. For instance, in recent studies of the FedStats web site, changing the interface of a governmental web site increased user satisfaction and performance nearly 100% (Ceaparu and Shneiderman, 2004). But these frustrations can actually be “engineered-out” of the system. And many of these frustrations are easily solvable. For instance, it is noted in the data thatSincereported beingare  , interface designers should be directed to review all messages and instructions. The cause of these specific frustrations are generally unclear wordIf link titles are unclear for users, or are not where the users expect, they may not be able to complete their tasks (Daniel and Lazar, 2004). Unclear wording has been found as a major problem in interface design, regardless of the user population or the task. For instance, in a usability study of a university web site, 5 users all failed to find the information that they were looking for (current course schedules) because the information was listed under an unclear heading (Student Life”). From a technology coding point of view, changing the words displayed is relatively simple. In addition, the usability methods needed to find out that the wording is unclear, are also relatively simple. Paper-based usability testing methods such as card sorting or paper prototypes can help find flaws in interface wording. Since many users reported being frustrated by unclear error messages or by hard-to-find features, interface designers should be directed to review all messages and instructions. Good guidelines for error message design have existed since 1982, but these guidelines are rarely followed. (Shneiderman, 1982). Error messages should be positive, provide information for users (in their language) on what occurred, and offer suggestions on how to continue. Current error messages rarely assist users (see figure 2 for an example of this). Improved error messages can reduce user frustration while making users more satisfied and productive. (Lazar and Huang, 2003). While all causes of user frustration are not as easily solvable, a large percentage of user frustrations ARE solvable. And there are many resources out there to help improve interface design, such as books, automated software tools, guidelines, and other resources (See or for more information).


Figure 2. An unclear error message





The iImplications for sDifferent Stakeholders might be separated out by:


Designers -

As designers, utilizing the results of studies such as this can yield the most benefits of all. B : can build more productive systems by learning what frustrates users in the workplace. designers can build more productive systems, that is systems with less frustration. Systems can be modified not only to have fewer errors but also to be more helpful. This may include better error messages, better and helpful descriptions of problems which can reduce the time needed to fix an issue, as well as designs based more closely on the way users work particularly with respect to how end users handle errors. This would improve efficiency overall as systems would be better equipped to handle problems faster and allow for the system to get back to operating normally (without problems) and in general make things more usable.


Managers -

Managers can benefits but by learning where frustrations occur within computing systems of their employees. This would let them see the bigger picture of how to improve processes as well as how to deal with frustrations in the workplace by understanding them. One such example might be allowing the manager to make better decisions in deploying systems and choosing the right IT support for employees. In other words less frustrating systems yield ahelp them to construct a more productive workplace, reduce workflow bottlenecks, and can make the environment more manageableproduce more satisfied employees in general. They can recommend training for employees and make more appropriate choices in software acquisitions.Managerial choices involving systems with varying frustration levels can only be made when understanding the causes and implications of IT related workplace frustrations.


Users -

End Users will ultimately benefit from the improvements made to systems based on this and similar studies.  This study also gives users a method for detailing and explaining errors, rather than simply working around them or ignoring them. Without such interaction and insight into where users get frustrated, improving things and reducing the frustration in the workplace is more difficult.of computers will appreciate learning that they are not alone in their frustrations. They can take steps to improve their training and increase their knowledge, but they can accelerate improvement by being consumer activists who report problems, complain to designers, and suggest improvements.


Policymakers -

 Policy makers, like managers, benefits from the larger picture and by seeing where inefficiencies occur.  It is highly possible to improve these frustrations through policy. Policies could be adapted to provide better solutions for handling incidents. Of particular importance would be policies and procedures for users interacting with help desk and IT support (which was a source of frustration in this study). By knowing where bottlenecks occur, policy can be modified to give quicker solutions and to make things easier not only for users, but for a company’s IT support and infrastructure as well.


ITInformation Technology Staff -

IT staff can be better prepared to handle frustrated users and learn which type of technical problems related to which levels ofproduce the largest frustration. This can help things move more smoothly and even help IT staff make better recommendations to managers and policymakers. IT Staff should also be better situated as the middleman and be able to get better information between users and vendors when understanding frustration in the workplace.


Policymakers, in industry and government, should recognize the severity of the productivity loss due to user frustration. Increased research funding, improved training, better data collection, and increased public awareness of the problems will help produce appropriate changes.


Essentially one ideal situation in which the implication might be turned into to practices is this. A frustrated user is dealing with a system problem. They contact the IT staff, explain the source of the frustration. The IT staff tries to help as best they can, then relays a summary of the incident to the manager, who then evaluates the frustration and talks to the policy makers. Together they improve the process of handling such incidents by making small adjustments. In addition they are better informed for future decisions that involve potentially frustrating systems. This results in filtering back down to the user, who now has an easier time when a frustrating incident occurs and can be more productive and recover quicker. IT Staff can then do their job more efficiently and give better information to the managers… This repeatedly refined process raises the bar with each iteration and makes a workplace better as a whole and is based on understanding the frustrating incidents themselves (and this understanding results from studies such as this).











The lead author of this article was partially supported by Training Grant No. T42/CCT310419 from the Centers for Disease Control and Prevention/National Institute for Occupational Safety and Health.  The contents are solely the responsibility of the author and do not necessarily represent the official views of the National Institute for Occupational Safety and Health.


We appreciate partial support from National Science Foundation grant for

Information Technology Research (#0086143) Understanding the Social Impact of

the Internet: A Multifaceted Multidisciplinary Approach and National Science Foundation grant for the Digital Government Initiative (EIA 0129978): Towards a Statistical Knowledge Network.


We acknowledge the assistance of Deborah Carstens and Robert Hammell, who both provided comments on an earlier draft of this paper.


Bandura, A. (1973). Aggression: A Social Learning Analysis. Englewood Cliffs, NJ: Prentice-Hall.

Bandura, A. (1986). Social Foundation of Thought and Action: A Social-Cognitive Theory. Englewood Cliffs, NJ: Prentice Hall.

Bandura, A. (1997a). Self-Efficacy. Harvard Mental Health Letter, 13(9), 4-6.

Bandura, A. (1997b). Self-Efficacy: The Exercise of Control. New York: Freeman.

Barker, R. (1938). The Effect of Frustration upon the Cognitive Ability. Character and Personality, 7, 145-150.

Barker, R., Dembo, T., & Lewin, K. (1965). Frustration and Regression: An Experiment with Young Children. In R. Lawson (Ed.), Frustration: The Development of a Scientific Concept. New York: MacMillan Publishing Co.

Baron, R. A. (1977). Human Aggression. New York, NY: Plenum.

Berkowitz, L. (1978). Whatever happened to the frustration aggression hypothesis? American Behavioral Scientist, 21(5), 691-708.

Bessiere, K. (2002). COMPUTER FRUSTRATION: A TEST OF A SOCIAL-PSYCHOLOGICAL MODEL. Unpublished Masters Thesis, University of Maryland, College Park, College Park, MD.

Bessiere, K., Lazar, J., Ceaparu, I., Robinson, J., & Shneiderman, B. (2002). Social Psychological Factors of End User Frustration and Their Implications. forthcoming.

Bias, R., & Mayhew, D. (Eds.). (1994). Cost-Justfying Usability. San Francisco: Morgan Kaufmann Publishers.

Britt, S. H., & Janus, S. Q. (1940). Criteria of Frustration. The Psychological Review, 47(6), 451-469.

Brosnan, M. J. (1998). The Impact of computer anxiety and self-efficacy upon performance. Journal of Computer Assisted Learning, 14, 223-234.

Cambre, M. A., & Cook, D. L. (1985). Computer Anxiety: Definition, Measurement, and Correlates. Journal of Educational Computing Research, 1(1), 37-54.

Campion, M., & Lord, R. (1982). A Control Systems Conceptualization of the Goal-Setting and Changing Process. Organizational Behavior and Human Performance, 30, 265-287.

Ceaparu, I., Lazar, J., Bessiere, K., Robinson, J., & Shneiderman, B. (20023, in press). Determining Causes and Severity of End-User Frustration. International Journal of Human-Computer Interactionforthcoming.

Ceaparu, I., and Shneiderman, S. (2004, in press). Finding Governmental Statistical Data on the Web: Three Empirical Studies of the FedStats Topics Page. Journal of the American Society for Information Science and Technology.

Clarke, J. (2001). Key factors in developing a positive user experience for children on the web: A case study. Proceedings of the Human Factors and the Web 2001, Available at:

Cohen, B. A., & Waugh, G. W. (1989). Assessing Computer Anxiety. Psychological Reports, 65(1), 735-738.

Compeau, D. R., & Higgins, C. A. (1995). Computer Self-Efficacy: Development of a Measure and Initial Test. MIS Quarterly, 19(2), 189-211.

Daniel, J., and Lazar, J. (2004). Improving the Usability of Governmental Statistics Presented on the Web. Working paper.

Dollard, J., Doob, L. W., Miller, N. E., Mowrer, O. H., & Sears, R. R. (1939). Frustration and Aggression. New Haven: Yale University Press.

Ferster, C. B. (1957). The Function of Aggression and the Regulation of Aggressive Drive. The Psychological Review, 71, 257-272.

Fowler, F. (1993). Survey Research Methods (2nd ed.). Newbury Park, California: Sage Publications.

Freud, S. (1921). Types of Onset and Neurosis. In J. Strachey (Ed.), The Standard Edition of the Complete Psychological Works of Sigmund Freud (Vol. 12, pp. 227-230). London: Hogarth Press.

Gilroy, F., & Desai, H. (1986). Computer Anxiety: Sex, Race, and Age. International Journal of Man-Machine Studies, 25(1), 711-719.

Glass, C. R., & Knight, L. A. (1988). Cognitive Factors in Computer Anxiety. Cognitive Therapy and Research, 12(4), 351-366.

Hu, J. (2002). AOL's Pop-Up Sacrifice. CNET News, pp. Available at:


Jex, S. M., & Gudanowski, D. M. (1992). Efficacy Beliefs and Work Stress: An Exploratory Study. Journal of Organizational Behavior, 13, 509-517.

Keenan, A., & Newton, T. J. (1984). Frustration in Organizations: Relationships to Role Stress, Climate, and Psychological Strain. Journal of Occupational Psychology, 57, 57-65.

Kemp, T. (2001, November 28, 2001). Macy's Doubles Conversion Rate., pp. Available at:

Lazar, J. and Huang, Y. (2003). "Improved Error Messages for Web Browsing: An Exploratory Experiment". In J. Ratner (ed.) Human Factors and Web Development, 2nd Edition, Mahwah, NJ: Lawrence Erlbaum Associates, 167-182.

Klein, H. J., & Kim, J. S. (1998). A Field Study of the Influence of Situational Constraints, Leader-Member Exchange, and Goal Commitment on Performance. Acadamy of Management Journal, 41(1), 88-95.

Lincecum, L. (2000). The Effects of Software Disruption on Goal Commitment, Task Self-Efficacy, Computer Self-Efficacy, and Test Performance in a Computer-Based Instructional Task. Unpublished Doctoral Dissertation, Texas Tech University.

Locke, E. A. (1996). Motivation Through Conscious Goal Setting. Applied Preventative Psychology, 5, 117-124.

Locke, E. A., & Latham, G. P. (1990). A Theory of Goal Setting and Task Performance. Englewood Cliffs, NJ: Prentice Hall.

Locke, E. A., & Latham, G. P. (2002). Building a Practically Useful Theory of Goal Setting and Task Motivation: A 35-Year Odyssey.Unpublished manuscript.

Loyd, B. H., & Gressard, C. (1984). The Effects of Sex, Age, and Computer Experience on Computer Attitudes. AEDS Journal, 18(2), 67-77.

Maurer, M. M. (1994). Computer Anxiety Correlates and What They Tell Us. Computers in Human Behavior, 10(3), 369-376.

McInerney, V., McInerney, D. M., & Sinclair, K. E. (1994). Student Teachers, Computer Anxiety, and Computer Experience. Journal of Educational Computing Research, 11(1), 177-189.

Meier, S. (1985). Computer Aversion. Computers in Human Behavior, 1(1), 171-179.

Mowrer, O. H. (1938a). Preparatory Set (Expectancy) -- a Determinant in Motivation and Learning. Psychological Review, 45, 62-91.

Mowrer, O. H. (1938b). Some Research Implications of the Frustration Concept as Related to Social and Educational Problems. Character and Personality, 7, 129-135.

Murphy, C., Coover, D., & Owen, S. (1989). Development and Validation of the Computer Self-Efficacy Scale. Educational and Psychological Measurement, 49, 893-899.

Murrell, A. J., & Sprinkle, J. (1993). The Impact of Negative Attitudes Toward Computers on Employees' Satisfaction and Commitment Within a Small Company. Computers in Human Behavior, 9, 57-63.

Nash, J. B., & Moroz, P. A. (1997). An Examination of the Factor Structures of the Computer Attitude Scale. Journal of Educational Computing Research, 17(4), 341-356.

Nielsen, J. (1994). Usability Engineering. Boston: Academic Press.

NTIA. (2001). A Nation Online: How Americans are Expanding Their Use of the Internet. Washington, DC: National Telecommunications and Information Administration.

Peters, L. H., Chassie, M. B., Lindholm, H. R., O'Connor, E. J., & Kline, C. R. (1982). The Joint Influence of Situational Constraints and Goal Setting on Performance and Affective Outcomes. Journal of Management, 8(2), 7-20.

Peters, L. H., & O'Connor, E. J. (1980). Situational Constraints and Work Outcomes: The Influences of a Frequently Overlooked Construct. Acadamy of Management Review, 5(3), 391-397.

Pew Internet and American Life Project. (2003). The Ever-Shifting Internet Population:
A new look at Internet access and the digital divide. Report available at: and downloaded on April 17, 2003

Peters, L. H., O'Connor, E. J., & Rudolf, C. J. (1980). The Behavioral and Affective Consequences of Performance-Relevant Situational Variables. Organizational Behavior and Human Performance, 25(79-96).

Phillips, J. S., & Freedman, S. M. (1984). Situational Performance Constraints and Task Characteristics: Their Relationship to Motivation and Satisfaction. Journal of Management, 10(3), 321-331.

Raub, A. (1981). Correlates of Computer Anxiety in College Students. Unpublished Doctoral Dissertation, University of Pennsylvania.

Riseberg, J., Klein, J., Fernandez, R., & Picard, R. (1998). Frustrating the user on purpose: Using biosignals in a pilot study to detect the user's emotional state. Proceedings of the CHI 1998: ACM Conference on Human Factors in Computing Systems, 227-228

Roberts-Witt, S. (2001, September 25, 2001). A Singular Focus. PC Magazine.

Shneiderman, B. (1982). System message design: Guidelines and experimental results, In Directions in Human-Computer Interaction, Badre, A. and Shneiderman, B. (Editors), Ablex Publishing Company: Norwood, NJ, 55-78.

Shorkey, C. T., & Crocker, S. B. (1981). Frustration theory: a source of unifying concepts for generalist practice. Social Work, 26(5), 374-379.

Spector, P. E. (1975). Relationships of Organizational Frustration with Reported Behavioral Reactions of Employees. Journal of Applied Psychology, 60(5), 635-637.

Spector, P. E. (1978). Organizational Frustration: A Model and Review of the Literature. Personnel Psychology, 31(815-828).

Storms, P. L., & Spector, P. E. (1987). Relationships of Organizational Frustration with Reported Behavioural Reactions: The Moderating Effect of Locus of Control. Journal of Occupational Psychology, 60, 227-234.

Tedeschi, B. (1999, August 30, 1999). Good Web Site Design Can Lead to Healthy Sales. The New York Times

Torkzadeh, G., & Angulo, I. E. (1992). The Concepts and Correlates of Computer Anxiety. Behavior and Information Technology, 11(1), 99-108.

Villanova, P., & Roman, M. A. (1993). A Meta-Analytic Review of Situational Constraints and Work-Related Outcomes: Alternative Approaches to Conceptualization. Human Resource Management Review, 3(2), 147-175.

Vroom, V. (1964). Work and Motivation. New York: Wiley.



Pre-Session Survey (Appendix A)

Email address: __________________________________


Section I: Demographic Information


1.   Age   __________

2.   Gender:  F    M

3.      Education: 

___ High School Graduate

___ Fresh/Soph in College

___ Jr./Sr. in College

___ College Graduate

___ Masters Degree

___ Doctoral-level


4.      In what field are you employed? ______________________________


5.      What is your job title?  ________________________________



Section II: Computer Experience and Attitudes


1.      How many years have you been using a desktop or laptop computer for home or work use?  _______

2.      How many hours per week do you use a desktop or laptop computer?  ______

3.      What type of Operating System is installed on the computer that you are currently using?

___ DOS

___ MacOS

___ Unix/Linux

___ Windows 95

___ Windows NT

___ Windows 98

___ Windows ME

___ Windows 2000

___ Windows XP





4.      What type of applications and programs do you typically use? (check all that apply)

___ Email

___ Other Internet Use

___ Graphic Design Programs

___ Chat/Instant Messaging

___ Word Processing

___ Programming Tools

___ Web Browsing

___ Spreadsheet Program (Excel)

___ Database management/                 Searching

___ Presentation Tools (powerpoint)

___ Other (please explain)


___ Multimedia (audio/video)


5.      How many years have you been using the internet? ___________

6.      How many hours per week do you spend online? Please indicate the amount of time that you are actually using the computer while online, not simply the amount of time you are connected to the internet.   _________

7.      At work, do you  ___ Have a permanent connection to the internet OR ___ dial in through a modem

8.      Which of the following do you do when encountering a problem on the computer or application that you are using?

___ try to fix it on my own

___ consult a manual/help tutorial

___ Ask help desk/ consultant for help

___ Ask a friend/relative for help

___ Give up or leave it unsolved



9.      How sufficient is your computer software and/or hardware for the work that you need to do?

Not at All   1   2   3   4   5   6   7   8   9   Very Sufficient


Section III: For the following questions, please choose the number that best corresponds to your feelings


1.  Computers make me feel:

Very Uncomfortable 1   2   3   4   5    6   7   8    9    Very Comfortable


2. When you run into a problem on the computer or an application you are using, do you feel:

                 Anxious   1   2   3   4      6   7    8   9   Relaxed/Indifferent


3. When you encounter a problem on the computer or an application you are using, how do you feel about your

ability to fix it?

               Helpless   1   2   3   4   5    6    7   8    9  Confident I can fix it


4. How experienced do you think you are when it comes to using a computer?

Very Inexperienced  1   2   3   4   5    6   7   8   9  Very Experienced


5. When there is a problem with a computer that I can't immediately solve, I would stick with it until

I have the answer.

  Strongly Disagree  1   2   3   4    5    6   7   8   9  Strongly Agree


6. If a problem is left unresolved on a computer, I would continue to think about it afterward.

  Strongly Disagree  1   2   3   4   5    6    7   8   9  Strongly Agree


7. Right now, how satisfied with your life are you?

    Very Unsatisfied  1   2   3   4    5   6   7    8    9  Very Satisfied


8. How often do you get upset over things?

       Not Very Often  1   2    3   4   5   6   7   8    9    Very Often


9. Right now, my mood is:

        Very Unhappy  1   2    3   4   5   6   7   8    9  Very Happy



Please fill out this form for each frustrating experience that you encounter while using your computer during the reporting session.  This should include both major problems such as computer or application crashes, and minor issues such as a program not responding the way that you need it to.  Anything that frustrates you should be recorded.


1.      What were you trying to do?



2.      On a scale of 1 (not very important) to 9 (very important), how important was this task to you?


Not very important  1     2     3     4     5     6     7     8     9   Very Important


3.      What software or program did the problem occur in? If the problem was the computer system, please check the program that you were using when it occurred (check all that apply).


___  ___  email

_______  file browsers

___  ____presentation software

(e.g. (e.g. powerpoint)

___  ___  chat and instant messaging

___  ___  spreadsheet programs

(e.g. (e.g. excel)

mult____media (audio/video software)

___  ___  web browsing

___  ___  graphic design

________other __________________

_______  other internet use

___  ___  programming tools


)))_____  word processing

_______  database management/

s               searching software



4.      Please write a brief description of the experience:




5.      How did you ultimately solve this problem? (please check only one)


___  I knew how to solve it because it has happened before

___  I ignored the problem or found an alternative solution

___  I figured out a way to fix it myself without help

___  I was unable to solve it

___  I asked someone for help.  Number of people asked ___

____I tried again

___  I consulted online help or the system/application tutorial

____I restarted the program

___  I consulted a manual or book


___  I rebooted




6.      Please provide a short step by step description of all the different things you tried in order to resolve this incident.





7.      How often does this problem happen? (please check only one)

            ___ more than once a day   ___ one time a day   ___ several times a week ___ once a week  

            ___ several times a month   ___ once a month   ___ several times a year    ___ first time it happened



8.      On a scale of 1 (not very frustrating) to 9 (very frustrating), how frustrating was this problem for you?


Not very frustrating  1     2     3     4     5     6     7     8     9   Very frustrating



9.      Of the following, did you feel: 

___ angry at the computer  ___ angry at yourself  ___ helpless/resigned

            ___ determined to fix it   ___neutral  ___  other: ___________



10.  How many minutes did it take you to fix this specific problem?  (if this has happened before, please account only for the current time spent) _____________________________



11.  Other than the amount of time it took you to fix the problem, how many minutes did you lose because of this problem?    (if this has happened before, please account only for the current time lost; e.g. time spent waiting or replacing lost work). ____________

Please explain:



12. Until this problem was solved, were you able to work on something else?

____Yes     ____No

Please explain:


Post-Session Survey (Appendix C)

Email address: __________________________________



For the following questions, please circle the number that best corresponds to your feelings. 


1. Right now, my mood is:


      Very Unhappy  1    2   3    4   5  6   7   8   9  Very Happy


2. We asked you to record your frustrating experiences. Overall, how frustrated are you after these experiences?


Not Frustrated at All  1   2   3   4    5   6   7   8   9  Very Frustrated


3. How will the frustrations that you experienced affect the rest of your day?


Not at All  1   2   3   4   5   6    7   8   9  Very Much


4. Are the incidents that occurred while you were recording your experiences typical of your everyday computer experience?


Yes                  No


5. In general, do you experience more or less frustrating incidents while using a computer on an average day?


Less   1   2   3   4   5   6    7   8  9  More


6. Did these frustrating experiences impact your ability to get your work done?

No impact  1   2   3   4   5   6    7   8  9  Severe impact


7.  Did these frustrating experiences impact your interaction with your co-workers?

No impact  1   2   3   4   5   6    7   8  9  Severe impact



 [I1] broken sentence

Web Accessibility