Complacency and Bias in Human Use of Automation: An Attentional Integration

Raja Parasuraman, George Mason University, Fairfax, Virginia, and Dietrich H. Manzey, Berlin Institute of Technology, Berlin, Germany

Objective: Our aim was to review empirical stud- INTRODUCTION ies of complacency and bias in human interaction with Human interaction with automated and deci- automated and decision support systems and provide an integrated theoretical model for their explanation. sion support systems constitutes an important Background: Automation-related complacency area of inquiry in human factors and ergonomics and automation bias have typically been considered (Bainbridge, 1983; Lee & Seppelt, 2009; Mosier, separately and independently. 2002; Parasuraman, 2000; R. Parasuraman, Methods: Studies on complacency and automation Sheridan, & Wickens, 2000; Rasmussen, 1986; bias were analyzed with respect to the cognitive pro- Sheridan, 2002; Wiener & Curry, 1980; Woods, cesses involved. 1996). Research has shown that automation Results: Automation complacency occurs under con- does not simply supplant human activity but ditions of multiple-task load, when manual tasks compete rather changes it, often in ways unintended and with the automated task for the operator’s attention. unanticipated by the designers of automation; Automation complacency is found in both naive and moreover, instances of misuse and disuse of expert participants and cannot be overcome with sim- ple practice. Automation bias results in making both omis- automation are common (R. Parasuraman & sion and commission errors when decision aids are Riley, 1997). Thus, the benefits anticipated by imperfect. Automation bias occurs in both naive and expert designers and policy makers when implement- participants, cannot be prevented by training or instruc- ing automation—increased efficiency, improved tions, and can affect decision making in individuals as well as safety, enhanced flexibility of operations, lower in teams. While automation bias has been conceived of as a operator workload, and so on—may not always special case of decision bias, our analysis suggests that it be realized and can be offset by human perfor- also depends on attentional processes similar to those mance costs associated with maladaptive use of involved in automation-related complacency. poorly designed or inadequately trained-for Conclusion: Complacency and automation bias repre- automation. sent different manifestations of overlapping automation- In this article, we review research on two induced phenomena, with attention playing a central role. An integrated model of complacency and automation bias such human performance costs: automation- shows that they result from the dynamic interaction of per- related complacency and automation bias. These sonal, situational, and automation-related characteristics. have typically (although not exclusively) been Application: The integrated model and attentional considered in the context of two different para- synthesis provides a heuristic framework for further digms of human-automation interaction, super- research on complacency and automation bias and design visory control (Sheridan & Verplank, 1978) and options for mitigating such effects in automated and deci- decision support, respectively. We discuss the sion support systems. cognitive processes associated with automation complacency and automation bias and provide a Keywords: attention, automation-related compla- synthesis that sees the two phenomena as repre- cency, automation bias, decision making, human-com- senting different manifestations of overlapping puter interaction, trust automation-induced phenomena, with attention playing a central role. Address correspondence to Raja Parasuraman, Arch Lab, MS 3F5, George Mason University, 4400 University Drive, AUTOMATION COMPLACENCY Fairfax, VA 22030; [email protected]. Definitions HUMAN FACTORS Vol. 52, No. 3, June 2010, pp. 381–410. The term complacency originated in references DOI: 10.1177/0018720810376055. in the aviation community to accidents or incidents Copyright © 2010, Human Factors and Ergonomics Society. in which pilots, air traffic controllers, or other 382 June 2010 - Human Factors operators purportedly did not conduct sufficient reported that in a survey of 100 highly experi- checks of system state and assumed “all was enced airline captains, more than half stated that well” when in fact a dangerous condition was complacency was a leading factor in accidents. developing that led to the accident. The National Initially, complacency was used to refer to inad- Aeronautics and Space Administration Aviation equate pilot monitoring in relation to any air- Safety Reporting System (ASRS) includes craft subsystem. With the advent of automation, complacency as a coding item for incident however, first in the aviation industry and later reports (Billings, Lauber, Funkhouser, Lyman, in many other domains, the possibility arose of & Huff, 1976). ASRS defines complacency as automation-related complacency. Consistent with “self-satisfaction that may result in non- this trend, in a more recent analysis of aviation vigilance based on an unjustified assumption of accidents involving automated aircraft, Funk satisfactory system state” (Billings et al., 1976, et al. (1999) also reported that complacency p. 23). Wiener (1981) discussed the ASRS defi- was among the top five contributing factors. nition as well as several others, including com- Complacency has also been cited as a con- placency defined as “a psychological state tributing factor in accidents in domains other characterized by a low index of suspicion” (p. than aviation. A widely cited example is the 119). Wiener proposed that empirical research grounding of the cruise ship Royal Majesty off was necessary to go beyond these somewhat the coast of Nantucket, Massachusetts (Degani, vague definitions so as to gain an understanding 2001; R. Parasuraman & Riley, 1997). This ship of the mechanisms of complacency and to make was fitted with an automatic radar plotting aid the concept useful in enhancing aviation safety. (ARPA) for navigation that was based on GPS Currently, there is no consensus on the defi- receiver output. The GPS receiver was connected nition of complacency. However, there is a core to an antenna mounted in an area where there set of features among many of the definitions was heavy foot traffic of the ship’s crew. As a that is common both to accident analyses and to result, the cable from the antenna frayed, lead- empirical human performance studies that could ing to a loss of the GPS signal. At this point, be used to derive a working definition. The first the ARPA system reverted to “dead reckoning” is that human operator monitoring of an auto- mode and did not correct for the prevailing tides mated system is involved. The second is that the and winds. Consequently, the ship was gradually frequency of such monitoring is lower than steered toward a sand bank in shallow waters. some standard or optimal value (see also Moray The National Transportation Safety Board (1997) & Inagaki, 2000). The third is that as a result of report on the incident cited crew overreliance substandard monitoring, there is some directly on the ARPA system and complacency associ- observable effect on system performance. The ated with insufficient monitoring of other sources performance consequence is usually that a sys- of navigational information, such as the Loran tem malfunction, anomalous condition, or out- C radar and visual lookout. right failure is missed (R. Parasuraman, Molloy, Automation complacency has been similarly & Singh, 1993). Technically, the performance cited in several other analyses of accidents and consequence could also involve not an omission incidents (Casey, 1998). The dangers of com- error but an extremely delayed reaction. However, placency have been described in many com- in many contexts in which there is strong time mentaries and editorial columns, including pressure to respond quickly, as in an air traffic those in leading scientific journals, such as Science control (ATC) conflict detection situation (Metzger (e.g., Koshland, 1989). Perhaps as a result of read- & Parasuraman, 2001), a delayed response would ing such anecdotal reports and opinion pieces be equivalent to a miss. (which have a tendency to overgeneralize and draw very broad conclusions), Dekker and col- Accident and Incident Reports leagues (Dekker & Hollnagel, 2004; Dekker & Operator complacency has long been impli- Woods, 2002) proposed that terms such as com- cated as a major contributing factor in aviation placency and situation awareness do not have accidents (Hurst & Hurst, 1982). Wiener (1981) scientific credibility but rather are simply “folk Automation Complacency and Bias 383 models.” R. Parasuraman, Sheridan, and conducting real ATC operations, did not have Wickens (2008) disputed this view, pointing to any competing tasks, only conflict detection. the growing scientific, empirical literature on R. Parasuraman et al. (1993) provided the the characteristics of both constructs. first empirical evidence for automation compla- Accident and incident reports clearly cannot cency and for the contributing role of high task be held to high scientific standards, but they pro- load. They had participants perform three con- vide a useful starting point for scientific under- current tasks from the Multiple Task Battery standing of any phenomenon. Describing the (MATB): a two-dimensional compensatory track- characteristics of complacency through empiri- ing and an engine fuel management task, both cal research would therefore appear to be an of which had to be carried out manually, and a important goal. In the three decades since the third task involving engine monitoring that early anecdotal reports and accident analyses required participants to detect abnormal read- mentioning complacency, a number of investiga- ings on one of four gauges; this task was sup- tors have followed Wiener’s (1981) call for such ported by an automated system that was not empirical research. perfectly reliable. In different conditions, the automation had either high (88%) or low Early Empirical Evidence for (52%) reliability in detecting engine malfunc- Complacency Effects in Monitoring tions. Complacency was operationally defined Automated Systems as the operator’s not detecting or being slow to Thackray and Touchstone (1989) conducted detect failures of the automation to detect engine an early yet ultimately unsuccessful attempt to malfunctions. obtain evidence of complacency. They had par- The variability of automation performance ticipants perform a simple ATC task requiring over time was also manipulated on the basis of detection of aircraft-to-aircraft conflicts (e.g., Langer’s (1989) concept of “premature cogni- those within 5 nautical miles of each other) with tive commitment,” defined as an attitude that or without a simulated automation aid that indi- develops when a person first encounters a cated that a conflict would occur. The automa- device in a particular context and that is then tion failed twice, once early and once late during reinforced when the device is reencountered in a 2-hr session. Although observers were some- the same way. Langer (1982) proposed that what slower to respond to the first failure when repeated exposure to the same experience using the automation, this was not the case for leads people to engage in “automated” or the later failure. Moreover, participants were as “mindless” behavior. R. Parasuraman and col- accurate at monitoring for conflicts with auto- leagues (1993) therefore reasoned that auto- mation as they were when performing the task mation that is unchanging in its reliability is manually, if not more so. more likely to induce complacency than is Thackray and Touchstone (1989) indicated automation that varies. In this case, partici- that their failure to obtain reliable evidence of pants will be more likely to develop a prema- complacency might be related to their use of a ture cognitive commitment regarding the relatively short test session, even though their nature of the automation and its efficiency. On testing period was 2 hr long and their ATC task the other hand, participants encountering was so simple and monotonous that many par- inconsistent automation reliability should have ticipants experienced considerable boredom a more open attitude concerning the efficiency (Thackray, 1981). However, subsequent studies of the automation and hence should be less in which complacency effects have been found likely to be complacent. Finally, to examine in shorter periods and with more complex tasks the effects of multiple-task load, Parasuraman indicate that the short test duration was unlikely et al. also conducted a second experiment in to have been the major reason for their failure. which participants had to perform only the A more likely factor is the extremely simple engine-monitoring task with automation sup- nature of the assignment given to the partici- port under either the constant-reliability or the pants in their study, who, unlike controllers variable-reliability condition. 384 June 2010 - Human Factors

1.0 away from the automation to other manual tasks in cases of high workload. 0.8 Factors Influencing Automation Constant Reliability 0.6 Complacency Variable Reliability There have been a number of additional studies 0.4 examining the characteristics of automation complacency following the original study by Automation Failure 0.2 Probability of Detecting R. Parasuraman et al. (1993). One question is whether the spatial location of the automated 0.0 1 2 3 4 5 6 7 8 9 10 11 12 task is an important factor in automation com- 10-Minute Blocks of Trials placency. In the Parasuraman et al. study, the automated task was always presented in the Figure 1. Detection of automation failures under periphery, away from the primary manual tasks constant-reliability and variable-reliability condi- that were centrally displayed. It is possible that tions. Adapted from “Performance Consequences the peripheral location led participants to of Automation-Induced ‘Complacency,’” by neglect the automated task. Singh, Molloy, and R. Parasuraman, R. Molloy, and I. L. Singh, 1993, Parasuraman (1997) accordingly examined whether International Journal of Aviation Psychology, 3, centrally locating the automated engine-moni- p. 10. Copyright 1993 by Taylor and Francis. Adapted toring task would boost performance and reduce with permission. or eliminate the complacency effect. They had participants perform the same three tasks and The results showed that complacency effects the same conditions as in Parasuraman et al., were linked both to the consistency of automa- with the single change that the engine-monitor- tion reliability and to task load. The mean detec- ing task was moved to the center of the display, tion rate of automation failures was markedly with the tracking and fuel management tasks higher for the variable-reliability condition (82%) located below it. Singh et al. found that monitor- than for the constant-reliability condition (33%) ing for automation failure was inefficient when (see Figure 1). The magnitude of the effect—a automation reliability was constant but not when 149% difference in detection rate—is dramatic, it varied over time, replicating Parasuraman considering that under single-task conditions, et al. Thus, the automation complacency effect detection of engine malfunctions was quite easy, was not prevented by centrally locating the auto- averaging about 97%. mated task. R. Parasuraman et al. (1993) also found that Automation reliability. In the automation detection of automation failures was significantly complacency studies described thus far, the poorer in the multitask condition than in the automation failure rate was relatively high, for single-task condition. When participants had example, 12% in the “high” reliability condi- simply to “back up” the automation routine with- tion of R. Parasuraman et al. (1993). Such high out other duties, monitoring was efficient and (and even higher) values of failure rate were near perfect in accuracy (~100%). needed for the study so that a sufficient number These results indicate that automation- of data points could be generated for estimating ind­uced complacency is more easily detect- the detection performance of individual partici- able in a multitask environment when operators pants on the engine-monitoring task. But an are responsible for many functions and their obvious drawback is that such high failure rates attention is focused on their manual tasks. are unrepresentative of any real automated sys- Impor­tantly, the findings suggest that compla- tem or at least any system that human operators cency is not a passive state that the operator would use. falls into (as the usage in everyday parlance To address this criticism, Molloy and would suggest). Rather, automation complacency Parasuraman (1996) conducted a study in which represents an active reallocation of attention the automation failed on only a single occasion Automation Complacency and Bias 385 during a test session. Participants performed the 100 same MATB simulation as in the R. Parasuraman et al. (1993) study. Two groups of participants 80 performed the MATB under either a multitask or a single-task condition, and a third group per- formed a simple line discrimination task; within 60 each group, participants performed two 30-min sessions separated by a rest break. For the two groups performing the MATB, the automation 40 failed only once in each session, either early Automation Failure Simple Task (first 10 min) or late (last 10 min); for the group 20 Single-Complex Task given the simple vigilance task, only one signal Proportion of Subjects Detecting Multi-Complex Task was presented, either early or late in the 30-min session. 0 First Last The authors found that in the single-task 10-Minute Block MATB condition, most participants detected the Figure 2. Automation complacency effect for a single automation failure, whether it occurred early or automation failure. Adapted from “Monitoring an late. Under multitask conditions, however, only Automated System for a Single Failure: Vigilance about half the participants detected the automa- and Task Complexity Effects,” by R. Molloy and tion failure, and an even smaller proportion R. Parasuraman, 1996, Human Factors, 38, p. 318. detected the failure if it occurred late than if it Copyright 1996 by the Human Factors and Ergonomics occurred early (see Figure 2). Bailey and Scerbo Society. Adapted with permission. (2007) replicated this finding using a somewhat different multitask flight simulation based on the MATB. various industrial monitoring and inspection jobs, De Waard, van der Hulst, Hoedemaeker, and Craig (1984) estimated that a critical signal might Brookhuis (1999) provided additional confir- occur about once every 2 weeks. However, given matory evidence in a study of automation in the impracticality of empirically testing partici- driving. They had participants in a simulator drive pants in a laboratory setting with such low signal a vehicle fitted for operation in an automatic rates, and given that detection of an unexpected highway system (AHS), in which steering and event typically decreases with reductions in sig- lateral control were automated but could be nal probability (Davies & Parasuraman, 1982), overridden by depressing the brake. Toward the these findings indicate that monitoring for auto- end of the scenario and on a single occasion, a mation failures is likely to be even poorer for vehicle merged suddenly into the same lane in very low failure rates representative of real but front of the participant’s AHS vehicle. The AHS imperfect automated systems. failed to detect the intrusion (automation fail- These considerations also suggest that ure). De Waard and colleagues found that half decreases in automation reliability should reduce of the drivers did not detect the failure, depress automation complacency, that is, increase the the brake, and retake manual control, and 14% det­ection rate of automation failures. In the did not respond quickly enough to avoid a R. Parasuraman et al. (1993) study, two auto- collision. mation reliability levels were compared, low These findings provide additional evidence and high, and although participants detected for the view that automation complacency more automation failures at the low reliability, occurs for highly reliable systems in which the difference was not statistically significant, automated control fails on only a single occa- possibly because of low power. In a replication sion. One could argue that even the single fail- study, however, Bagheri and Jamieson (2004) ure during a 30-min session is not representative did find that participants detected significantly of real systems, in which even lower failure more automation failures at low than at high rates might be seen. For example, in a review of automation reliability. Moreover, they confirmed 386 June 2010 - Human Factors

Parasuraman et al.’s finding that detection per- First-failure effect. In addition to the overall formance was better under variable-reliability automation failure rate, the temporal sequence than under constant-reliability automation. Finally, of failures and the time between failures may be May, Molloy, and Parasuraman (1993) varied important factors to consider as well. If compla- automation reliability across a range of values cency reflects an operator’s initial attitude toward under the same multitask condition used in high-reliability automation based on high trust, Parasuraman et al. They found that the detec- then a failure to monitor is perhaps to be tion rate of automation failures varied inversely expected the first time the automation fails. Lee with automation reliability, but no evidence was and Moray (1992, 1994) showed that the reduc- found for a lower limit of automation reliability tion in operator trust following an automation below which automation complacency did not failure was followed by a recovery in trust but occur. Rather, detection of automation failures at a slow rate. If so, then one would expect the was still worse than manual performance even complacency effect to be high for the first fail- at a low level of automation reliability. ure but to dissipate thereafter, a phenomenon Complacency represents a cost that can off- Merlo, Wickens, and Yeh (2000) referred to as set the benefits that automation can provide. the first-failure effect. Some evidence for the One would presume that when the reliability effect was reported in a recent study by Rovira, level of an automated system falls below some McGarry, and Parasuraman (2007) in which limit, that there would be neither benefits nor participants had to make simulated battlefield costs associated with automation. Wickens and engagement decisions under time pressure with Dixon (2007) proposed that this cutoff value is the aid of imperfect automation. Performance approximately 70% (with a standard error of declined the first time automation failed but ±14%), on the basis of studies examining the improved on subsequent failures. beneficial effects of automation support. If this These findings suggest that complacency is objective standard was followed, then observers associated with a cognitive orientation toward should not rely on imperfect automation whose very-high-reliability automation prior to the reliability falls below this value, but Wickens first time it has failed in the user’s experience and Dixon reported that some continue to do so. (R. Parasuraman & Wickens, 2008). Subsequent Congruent with this finding, May et al. (1993) exposure to automation failures may allow for also found that participants continued to show better calibration to the true reliability, so that complacency effects even at low automation detection performance improves. However, reliability. Moreover, other researchers have found whereas Rovira et al. (2007) did find some cor- that even automation with reliabilities lower roborative evidence, other studies using differ- than the 70% cutoff value can support human ent tasks have not found consistent evidence for operators who also have access to the “raw” the first-failure effect (Wickens, Gempler, & information sources, which they can combine Morphew, 2000). with the automation output to improve overall Expertise and automation realism. Although performance (de Visser & Parasuraman, 2007; the evidence for the first-failure effect is equiv- St. John, Smallman, Manes, Feher, & Morrison, ocal, the phenomenon raises the issue of whether 2005). complacency-like effects stem from insufficient Automation reliance appears to be strongly experience with the automation or from inade- context dependent, with the 70% threshold quate practice in performing the automated task. being important primarily under high workload With respect to the first issue, it is noteworthy (Wickens & Dixon, 2007). Multiple-task condi- that the complacency studies described thus far tions are also where the automation compla- all involved artificial types of automation not cency effect is strongest. Whether there is a fixed found in real systems. Furthermore, college lower bound of automation reliability, below students were used as participants. What of which neither benefits nor costs accrue, and the exp­erienced, skilled workers who are tested with influence of contextual factors on such a thresh- automation more closely resembling real auto- old are issues that need further investigation. mated systems? Do they exhibit automation Automation Complacency and Bias 387 complacency? Three studies have provided a Sharma, and Parasuraman (2001) found that the positive answer to this question. automation complacency effect obtained in the First, using the MATB simulation, Singh, standard paradigm described by R. Parasuraman Molloy, Mouloua, Deaton, and Parasuraman et al. (1993) was not reduced by up to 60 min of (1998) compared the performance of pilots with training. an average of approximately 440 hr of flight Although extended practice does not eliminate experience with that of nonpilots in the same automation complacency, other training proce- paradigm developed by R. Parasuraman et al. dures may provide some benefit. In particular, (1993). Clear complacency effects were obtained given that complacency is primarily found in for both groups, although the pilots detected more multitasking environments and represents atten- automation failures than did the nonpilots. tion allocation away from the automated task, Second, Galster and Parasuraman (2001) tested training in attention strategies might mitigate general aviation pilots with several hundred complacency. hours of flight experience on a multitask flight One such training procedure is the variable- simulation involving Instrument Flight Rated priority method proposed by Gopher (1996; flight using an actual cockpit automation sys- Gopher, Weil, & Siegel, 1989). For example, in tem, the Engine Indicator and Crew Alerting a dual-task setting, observers are trained to System (EICAS). Clear evidence of a compla- devote greater priority to one task (say, 80%) cency effect was found, with pilots detecting and less to the other (20%) in one block of train- fewer engine malfunctions when using the EICAS ing trials, followed by the reverse priority in a than when performing this task manually. subsequent block. Compared with fixed, equal- In a third study, by Metzger and Parasuraman priority training (50% and 50%), variable-priority (2005), experienced air traffic controllers were training results in faster acquisition of dual- tested on a high-fidelity simulation of a future task skills. Accordingly, Metzger, Duley, Abbas, ATC (Free Flight) scenario requiring detection and Parasuraman (2000) trained participants in of conflicts among “self-separating” aircraft. the three subtasks of the MATB using either Controllers were supported by a “conflict probe” the variable- or the fixed-priority method and automation that pointed to a potential conflict examined both overall performance and detec- several minutes before its occurrence. The auto- tion of failures in the automated task (compla- mation failed once toward the end of the sce- cency). Variable-priority training led to better nario. The authors found that significantly multitasking performance, and a trend for a fewer controllers detected the conflict when the reduction in the automation complacency effect conflict probe failed than when the same con- was observed. An additional training method flict was handled manually. This result was con- that might reduce complacency includes experi- sistent with an earlier finding from this group ence of automation failures (Bahner, Huper, & showing that conflict detection performance was Manzey, 2008). We consider this method in a better when controllers were actively involved later section of this article, where studies of in conflict monitoring and conflict resolution automation bias are discussed. (“active control”) than when they were asked to be passive monitors in a simulated Free Flight Automation Complacency, Attention, scenario involving pilot self-separation (Galster, and Trust Duley, Masalonis, & Parasuraman, 2001; Metzger The studies discussed thus far have shown & Parasuraman, 2001). that automation complacency—operationally Training. Notwithstanding these findings with defined as poorer detection of system malfunc- expert pilots and controllers, another aspect of tions under automation control compared with experience is familiarity with the simulation, manual control—is typically found under con- automation, and task setting per se. Could the ditions of multiple-task load, when manual tasks complacency effect simply reflect insufficient compete with the automated task for the opera- practice at performing the automated task in tor’s attention. The operator’s attention alloca- conjunction with other manual tasks? Singh, tion strategy appears to favor his or her manual 388 June 2010 - Human Factors tasks as opposed to the automated task. This strategy may itself stem from an initial orienta- Task A tion of trust in the automation, which is then Attention Task B Human reinforced when the automation performs at the Allocation System Operator Strategy same, constant level of reliability. Indicators The finding that variable-reliability automa- Task C tion, fluctuating between high and low and back Task D Trust again, is associated with the elimination of (Automated) complacency is certainly compatible with the notion that reduced operator monitoring of the Figure 3. Attention allocation and trust in manual and automation could be linked to trust. Thus, as automated tasks. Moray (2003; Moray & Inagaki, 2000) pointed out, an attention strategy devoted primarily to detected the conflict in both the automation and manual tasks and only occasionally to the auto- the manual conditions, there were no differences mated task can be considered rational (see also in the number of fixations of the primary radar Sheridan, 2002). Moray (2003; Moray & display where the conflicting aircraft were shown. Inagaki, 2000) also suggested that complacency Among controllers who missed the conflict, could be inferred only if the operator’s rate of however, there were significantly fewer fixa- sampling of the automated task was below that tions of the radar display under automation sup- of an optimal or normative observer. Moray’s port than under manual control. This finding views are considered further later in this article, provides strong evidence for a link between the but for now, we simply note that these consider- automation complacency effect and reduced ations reinforce a close link between automa- visual attention to the primary information sources tion complacency, attention, and trust. Figure 3 feeding automation which must be monitored to shows a schematic of these links. The operator detect an abnormal event (for related eye move- uses an attention allocation strategy to sample ment evidence on complacency, see Baghieri & his or her manual tasks, with attention to the Jamieson, 2004, and Wickens, Dixon, Goh, & automated task being driven in part by trust. Hammer, 2005). The original R. Parasuraman et al. (1993) study Although these studies point to a role for also discussed the finding of automated com- attention allocation in the automation compla- placency under constant-reliability conditions in cency effect, their interpretation depends on the terms of attentional and trust factors. However, assumption of a close link between eye move- that study did not obtain independent measures ments and attention. There is considerable evi- either of attention or of trust. Subsequent stud- dence for a link between the two (Corbetta, ies have used eye movement recordings to 1998; Shepherd, Findlay, & Hockey, 1986), and examine attention allocation under conditions studies of covert shifts of attention show that of manual and automation control. In particular, such attention shifts typically precede an eye Metzger and Parasuraman (2005) used different movement (Hoffman & Subramaniam, 1995). measures of eye movements to examine the However, attention and eye movements can attentional theory of automation complacency. also be dissociated, and the phenomena of “inat- In this study, experienced air traffic control- tentional blindness” or “change blindness” lers were required to detect aircraft-to-aircraft (Mack & Rock, 1998; Simons & Rensink, 2005) conflicts either manually or with the aid of auto- indicates that relatively salient items of infor- mation (a “look-ahead” conflict probe). Toward mation in the environment can be missed even the end of the scenario, the automation failed on if they are fixated (R. Parasuraman, Cosenzo, & a single occasion. A greater proportion of con- de Visser, 2009; Thomas & Wickens, 2006). trollers missed detecting the conflict with the Thus, although an eye fixation generally indi- automation than when, in a separate session, cates that a location was attended, it need not they handled the same conflict (rotated in sector always do so, because attention may have moved geometry to reduce familiarity effects) without to another location—the so-called looking-but- the help of automation. Among controllers who not-seeing phenomenon. Automation Complacency and Bias 389

Duley, Westerman, Molloy, and Parasuraman trust and automation studies, Lee and See (1997) reexamined the automation complacency (2004) did not find convincing links between effect to see whether it could be reduced or poor automation monitoring and high trust in eliminated by forcing eye fixations on the auto- automation, and currently, the Bailey and mated task. They did this by superimposing the Scerbo study provides the only, somewhat ten- (automated) engine-monitoring task on the tative, evidence. manual tracking task in the MATB simulation, One reason for the weak evidence for a rela- carefully interleaving the display elements of tionship between complacency and trust is the each task so that they did not mask each other. potential discrepancy between subjective and They chose the tracking task for superimposition objective measures of trust. Overt behavioral because it was a high-bandwidth task (cf. Moray indicators of trust, as revealed, for example, in & Inagaki, 2000) that needed to be frequently verification activities (St. John et al., 2005), visually sampled for successful performance. may provide stronger evidence of a closer link. Thus, operators would repeatedly have to fixate We consider this issue further later, when we the tracking task, and by doing so, the auto- discuss a study by Bahner et al. (2008). mated task would also fall within foveal vision at the same time. Surprisingly, Duley et al. Individual Differences found that the standard complacency effect of There are considerable individual differences R. Parasuraman et al. (1993) was again found: in the performance effects associated with auto- Participants were poorer in detecting engine mation complacency. R. Parasuraman et al. malfunctions under automation than when they (1993) found that in the experimental group that did the task manually. Tracking performance was exhibited this effect, the mean detection rate of equally good under both conditions. Thus, the automation failures was quite low, 33%. But automation complacency effect could not be there was considerable variability around this attributed to insufficient sampling of the auto- mean, and as many as half of the participants in mated task. These findings suggest that atten- the group did not detect any malfunctions (0% tion allocation away from the automated task detection) on one of the blocks of the test ses- associated with complacency may include not sion. Clearly, some participants exhibit the only fixation failures but attention failures as well. effect to a greater extent than others. What is the In addition to visual attention, automation source of these individual differences? complacency has been linked to an initial atti- In an initial examination of individual differ- tude of high trust toward the automation. ences, Singh, Molloy, and Parasuraman (1993a) Trust was not directly measured in the original distinguished between complacency potential R. Parasuraman et al. (1993) study. However, and behavior. Although an attitude of trust toward subjective measures of trust were obtained in automation can foster overreliance, this may two subsequent studies, by Baghieri and not in itself lead to complacent behavior but Jamieson (2004) and by Bailey and Scerbo may indicate only a potential for complacency. (2007). Consistent with Parasuraman et al., Complacent behavior—as reflected in poor Baghieri and Jamieson found that constant-reli- monitoring of automation—may occur only ability automation led to poorer detection of when complacency potential coexists with other automation failures than did variable-reliability conditions, such as high task load. Individuals automation, but this effect was not associated may differ both in the potential they “bring” to with higher subjective trust. However, in two a task setting as well as in their response to that experiments conducted by Bailey and Scerbo setting. using the single-failure paradigm of Molloy and Singh et al. (1993a) developed a 20-item Parasuraman (2006), an inverse relationship scale, the Complacency Potential Rating Scale was observed between subjective trust (mea- (CPRS), with items covering attitudes toward sured with a 12-item questionnaire) and moni- commonly encountered automated devices (such toring performance, although the relationship as automatic teller machines). Factor analysis emerged only when data from the two separate of questionnaire responses in a psychology experiments were combined. In a review of college population and a different sample of 390 June 2010 - Human Factors engineering students (R. Parasuraman, Singh, With respect to other sources of individual or Molloy, & Parasuraman, 1992) revealed factors group differences, there do not appear to be any with good internal consistency and test-retest gender differences in complacency. However, reliability, suggesting that the CPRS might be adult age differences in automation complacency useful in validation studies examining individ- have been reported, with older adults exhibiting ual differences in complacent behavior. greater automation-related complacency but only Singh, Molloy, and Parasuraman (1993b) rep­ under very high workload conditions (Hardy, orted one such study. They administered the CPRS Mouloua, Dwivedi, & Parasuraman, 1995; to two groups of participants who performed Vincenzi, Muldoon, Mouloua, Parasuraman, & the MATB simulation in the same conditions of Molloy, 1996). the R. Parasuraman et al. (1993) study. There Summary was some evidence that complacent behavior— poor detection of automation failures—was asso- The studies discussed thus far have shown ciated (r = –.42) with higher complacency that automation complacency—operationally potential as measured by the CPRS. However, defined as poorer detection of system malfunc- this correlation emerged only when the partici- tions under automation compared with under pants were subdivided by a median split of manual control—is typically found under con- CPRS scores into low- and high-compla- ditions of multiple-task load, when manual tasks cency-potential groups, with the association compete with the automated task for the opera- being found only for high-complacency-poten- tor’s attention. Several factors modulate this tial participants. Singh et al. suggested that effect: It is largely eliminated when automation there is perhaps a threshold of complacency reliability is varied over time as opposed to potential, with a link between general attitudes when reliability remains constant. Automation of reliance and trust toward automation being complacency is also reduced when the automa- reflected in complacent performance only in indi- tion failure rate is increased, but the issue of viduals who exhibit such attitudes to a strong whether there is a threshold reliability level degree. Very similar findings were reported in a below which automation complacency does not replication study by Prinzel, DeVries, Freeman, occur remains unresolved. Conversely, compla- and Mikulka (2001), who found that individuals cency occurs for highly reliable (yet imperfect) scoring high on the CPRS were particularly automation and even when the automation poor in monitoring automation under constant- fails on only a single occasion in the operator’s than under variable-reliability automation. experience. Singh et al. (1993a) proposed that compla- Finally, experience and practice do not app­ cency potential represents an attitude toward ear to mitigate automation complacency: Skilled automation rather than an enduring trait. Con­ pilots and controllers exhibit the effect, and sistent with this view, Singh et al. (1993b) found additional task practice in naive operators does no relationship between the automation com- not eliminate complacency. It is possible that placency effect and the personality trait of specific experience in automation failures may extraversion-introversion. Prinzel et al. (2001) reduce the extent of the effect. Automation also found no association with measures of complacency can be understood in terms of an boredom proneness or absentmindedness, attention allocation strategy whereby the opera- although in a subsequent study, they did find tor’s manual tasks are attended to at the expense some evidence for a negative correlation of the automated task, a strategy that may be between automation complacency and self- driven by initial high trust in the automation. efficacy (Prinzel, 2002). In general, however, AUTOMATION BIAS strong associations between personality or related indices of personal variability and auto- Automated Decision Aids mation complacency have not been found. Automated decision aids are devices that However, the currently available database is support human decision making in complex very small and therefore does not warrant any environments. A dichotomous alarm system decisive conclusions. that alerts human operators to a potential hazard Automation Complacency and Bias 391 represents the most basic example (Wiegmann, Consequently, Mosier and Skitka (1996) defined 2002). More complex examples include the automation bias as resulting from people’s Traffic Conflict and Alert System (TCAS) and using the outcome of the decision aid “as a heu- the Ground Proximity Warning System (GPWS), ristic replacement for vigilant information seek- which are installed in many commercial air- ing and processing” (p. 205). Such a definition craft. Other examples include navigation aids in treats automation bias as similar to other biases cars, fault diagnosis and management systems and heuristics in human decision making (e.g., in process control, and expert systems for phy- confirmation bias), with the qualification that sicians or computer-based aids for surgeons in the bias stems specifically from interaction with the medical domain. an automated system. Such systems are meant to support human Automation bias eventually can lead to deci- cognitive processes of information analysis sions that are not based on a thorough analysis and/or response selection by providing auto- of all available information but that are strongly matically generated cues to help the human user biased by the automatically generated advice. correctly assess a given situation or system state Whereas automation bias is inconsequential and to respond appropriately. Two different when the recommendations are correct, it can functions of decision aids can be distinguished: compromise performance considerably in case alerts and recommendations. The alert function, of automation failures, that is, if the aid does not which is the main feature of simple alarm sys- alert the user to become active or if the aid pro- tems and is embedded in more complex deci- vides a false recommendation or directive. An sion aids, makes the user aware of a situational error of omission, whereby the user does not change that might require action. In a car navi- respond to a critical situation, is related to the gation aid, for example, this function is acti- alert function. An example from everyday expe- vated whenever a turn must be made. The rience is a driver who misses the correct exit recommendation function involves advice on from a highway because the navigation aid choice and action. For example, navigation aids failed to notify the driver. The second type of provide specific recommendations of where to error is a commission error, which is related to drive; cockpit warning systems, such as TCAS the specific recommendations or directives pro- or GPWS, provide pilots with specific com- vided by an aid. In this case, users follow the mands (e.g., “Pull up! Pull up!”) to avoid colli- advice of the aid even though it is incorrect. An sion; and medical expert systems provide example is a driver who falsely enters a one-way recommendations about appropriate treatment street from the wrong side because the naviga- of patients and choice of drug doses. tion aid (which may not have had the one-way information in its database) tells the driver to do Definition and Characteristics of so. Another example is following the advice of Automation Bias an expert flight planner although its recom- Automated decision aids are meant to enhance mendations are wrong or less than optimal for human decision-making efficiency. This is of a particular situation (e.g., Layton, Smith, & particular value in areas where incorrect deci- McCoy, 1994). sions have high costs in terms of either economic Three main factors have been assumed to consequences (e.g., in the area of manufactur- contribute to the occurrence of automation ing) or safety outcomes (e.g., aviation, medi- bias (Dzindolet, Beck, Pierce, & Dawe, 2001; cine, process control). If the benefit of a decision Mosier & Skitka, 1996). One is the tendency of aid is to be realized, it needs to be used appro- humans to choose the road of least cognitive priately. Quite often, however, decision aids are effort in decision making, the so-called cogni- misused, for two main reasons. First, the auto- tive-miser hypothesis (Wickens & Hollands, matically generated cues are very salient and 2000). Instead of basing complex decisions on a draw the user’s attention. Second, users have a comprehensive analysis of available informa- tendency to ascribe greater power and authority tion, humans often use simpler heuristics and to automated aids than to other sources of advice. decision rules (Gigerenzer & Todd, 1999; 392 June 2010 - Human Factors

Kahneman, Slovic, & Tversky, 1982). Well- et al., 1994; McGuirl & Sarter, 2006; Mosier, known examples of this tendency include the Palmer & Degani, 1992; Mosier, Skitka, Heers, heuristics of representativeness and availability & Burdick, 1998; Sarter & Schroeder, 2001). (Tversky & Kahneman, 1974). It has been argued Layton et al. (1994) compared the impact of that recommendations and directives of automated electronic flight planning tools on the quality of aids might serve as a strong decision-making heu- the en route flight planning decisions of pilots. ristic used by the human user as a replacement for The aids differed in their level of automation more effortful processes of information analysis (LOA): The low LOA left most of the planning and evaluation (Mosier & Skitka, 1996). decisions to the pilots and provided only an A second factor is the perceived trust of evaluation of different plans, whereas the high humans in automated aids as powerful agents LOA specified and recommended a plan. Pilots with superior analysis capability (Lee & See, working with the high LOA aid spent less time 2004). As a consequence, users might tend to and effort in generating and evaluating alterna- overestimate the performance of automated aids. tive plans than the group working with the low- More specifically, they may ascribe to the aid LOA aid. This result is consistent with the greater performance and authority than to other cognitive-miser hypothesis of automation bias. humans or themselves. Some evidence for this In addition, a majority of these pilots also effect was provided by Dzindolet, Pierce, Beck, tended to accept the plan provided by the auto- and Dawe (2002). In this study, participants had mation in cases in which the plan for a number to predict the performance of an automated aid of reasons actually represented a suboptimal compared with the expected performance of solution. another human supporting them in a decision- Mosier et al. (1992) studied pilot decision making task. It turned out that the majority of making in simulated engine fire situations, in participants initially expected the automated aid particular when a decision aid provided wrong to perform considerably better than a human aid. advice. An automated electronic checklist was A third contributing factor to automation bias implemented that recommended that the pilot is the phenomenon of diffusion of responsibil- shut down the wrong engine (i.e., the one not ity. Sharing monitoring and decision-making tasks affected by fire). Mosier and colleagues found with an automated aid may lead to the same that 75% of pilots followed the wrong recom- psychological effects that occur when humans mendation and neglected to check relevant share tasks with other humans, whereby “social information available from other indicators. In loafing” can occur—reflected in the tendency contrast, only 25% of pilots using a traditional of humans to reduce their own effort when paper checklist committed the same type of working redundantly within a group than when commission error. they work individually on a given task (Karau In another study, Mosier et al. (1998) had & Williams, 1993). Similar effects occur when experienced commercial pilots perform a sim- two operators share the responsibility for a ulated flight in a trainer equipped with adv­ monitoring task with automation (Domeinski, anced cockpit automation systems (e.g., Flight Wagner, Schoebel, & Manzey, 2007). To the Management System, EICAS). During the sim- extent that human users perceive an automated ulated flight, various automation failures aid as another team member, they may perceive occurred; each provided opportunities to com- themselves as less responsible for the outcome mit either omission or commission errors. and, as a consequence, reduce their own effort Examples of the first kind included an altitude in monitoring and analyzing other available clearance that did not get loaded correctly, a information. heading change that was not executed properly by the flight system, and a frequency change Evidence for Automation Bias misload. In addition, one opportunity for a com- Aviation studies. Several studies examining mission error was presented in the form of a pilot interaction with expert systems or with falsely released engine fire warning that occurred advanced cockpit automation have provided without being supported by any other readings empirical evidence for automation bias (Layton of engine parameters or indications available Automation Complacency and Bias 393 from relevant cockpit displays, gauges, and A similar result emerged for commission indicators. Although all of these automation errors. In cases in which the automated aid pro- failures were, in principle, easily detectable by vided a wrong recommendation, approximately monitoring the relevant information available 65% of the participants committed a commis- from different cockpit displays (e.g., the pri- sion error by following this advice without mary flight display,), the omission error rate taking into account the clearly disconfirming was considerable and reached 55%. evidence directly available from the relevant The results for the commission error event gauges. All of these effects emerged although were even more dramatic: All the pilots decided the participants were informed that all readings to shut down the engine in response to the false from indicators and gauges were always per- EICAS alert. This action corresponds to a 100% fectly valid and provided a reliable basis for commission error rate. This result was in con- cross-checking the automation. trast to the indication by the same pilots in a Sarter and Schroeder (2001; see also McGuirl debriefing interview that an EICAS message & Sarter, 2006) examined the performance of alone would not be sufficient to shut down an pilots interacting with automated decision aids engine completely and that it would be safer that supported decision making in case of in- under these circumstances to just reduce the flight icing events. Such events represent a seri- power to idle. Clearly, the automatically gener- ous threat to flight safety and must be handled ated warning must have led the pilots to either promptly. Consequently, an in-flight icing event ignore (or discount) the contradicting informa- is characterized by time pressure and uncer- tion available from other displays and engine tainty: Pilots need to respond rapidly without indicators or to believe that it would be in line always being able to verify directly whether with the warning, as in a confirmation bias icing in fact has taken place. Two types of deci- effect. The debriefing interview with the par- sion aids were compared in a simulator study. ticipants provided confirmatory evidence for The first involved a status display that provided the latter explanation: 67% of the pilots reported information about the icing condition (i.e., wing that they had seen at least one more indication icing vs. tailplane icing) but left the selection of supporting the fire warning from the EICAS appropriate responses with the pilots. The sec- (which in fact was absent), an effect termed ond one involved a command display that in “phantom memory” by Mosier et al. (1998). addition to the information about the icing con- Additional evidence for automation bias was dition provided recommendations for proper provided in a laboratory experiment comparing actions (e.g., concerning appropriate pitch atti- the performance of nonpilots in a low-fidelity tude and settings of flaps and power). Performance flight simulation task with and without auto- data assessed included, among others, how long mation support, the Workload/Performance the pilots needed to respond to the ice accretion Simulation (W/PANES; Skitka, Mosier & and whether they responded in a way that effec- Burdick, 1999). Participants had to perform tively prevented serious consequences indicated three tasks simultaneously, including a compen- first by a buffet of the airframe and eventually satory tracking task, a waypoint task, and a by a stall. gauge-monitoring task. Both of these latter Compared to a baseline condition in which tasks had to be performed either manually or pilots had to manage in-flight icing encounters with the support of an automated aid. During without any automation support, that is, by just the experiment, automation failures occurred, relying on their own kinesthetic perception of six in which the automation failed to prompt the changed flight dynamics, the availability of the participants of a critical event and another six in aids increased the number of correct decisions which it gave a wrong directive. Only 59% of in response to different icing encounters consid- the former were correctly identified and responded erably. This finding was reflected in a signifi- to by the participants, which corresponds to a cantly lower frequency of early buffets (7.87% 41% omission error rate. In contrast, almost compared with 20.56% without support) as well 97% of participants working without automa- as fewer stalls (18.08% vs. 30.00%). However, tion detected them correctly. this performance benefit was observed only 394 June 2010 - Human Factors when the aid provided correct recommendations. any clear-cut interpretation in terms of automa- In case of inaccurate information, the availabil- tion bias difficult. ity of the aid resulted in clear performance dec- However, Alberdi et al. (2004, 2008) con- rements compared with the baseline condition, ducted more carefully designed studies on the with the frequency of early buffets increasing effects of automated aids on clinical decision up to 73.61% and those of stalls even to 88.89%. making. Experienced radiologists examined a This impairment of performance was mainly set of mammograms either with or without the related to the pilots’ inadvertently following the support of a detection aid that suggested areas aids’ recommendation even though the avail- containing lesions. Half of the mammograms able kinesthetic cues contradicted it, hence indi- contained signs of cancer, whereas the other cating a clear tendency toward automation bias. half was free of pathology. Four different kinds Moreover, a significant interaction effect was of cases were compared: (a) cases in which the found between the type and accuracy of the detection aid provided valid advice, either by decision aid. Whereas both aids led to worse correctly placing a prompt on a critical feature performance when the information provided or by leaving a mammogram without pathologi- was inaccurate compared with an accurate con- cal findings unmarked; (b) cases in which the dition, this effect was somewhat stronger for aid failed to prompt critical features, that is, left command than for status displays. However, the a mammogram unmarked although there were effect could not be replicated in a follow-up signs of cancer; (c) cases in which the aid incor- study to this research (McGuirl & Sarter, 2006), rectly placed a prompt in an area away from an thus raising some doubts as to its generality. actual sign of cancer; and (d) cases in which the Health care. Evidence of automation bias has aid incorrectly placed a prompt on a mammo- also been reported in the health care domain gram where in fact no signs of cancer were pres- (Alberdi, Povyakalo, Strigini, & Ayton, 2004; ent. Cases of Types b and c represented false Alberdi, Povyakalo, Strigini, Ayton, & Given- negatives that provoke omission errors if the Wilson, 2008; McKibbon & Fridsma, 2006; film reader’s decision making was biased by Tsai, Fridsma, & Gatti, 2003; Westbrook, Coiera, the aid’s suggestion. Cases of Type d provided & Gosling, 2005). McKibbon and Fridsma the opportunity for a commission error. (2006) and Westbrook et al. (2005) explored the The results provided clear evidence for auto- effects of electronic information systems on mation bias in terms of omission errors. The decisions made by primary care physicians. In detection rate for “unmarked” cancers (Case b) controlled field trials, they compared the cor- dropped from 46% in conditions without the aid rectness of answers to a set of standardized to 21% in the aided conditions, and incorrectly clinical questions provided by physicians before marked cases (c) reduced the detection rate of and after searching different electronic sources, cancer from 66% to 53%. Clearly, the film read- such as PubMed, Medline, and Google. They ers tended to take the absence of a computer found a small to medium (2% and 21%, respec- prompt as strong evidence for the absence of tively) increase in the rate of correct answers cancer. The authors interpreted this finding as attributable to the use of these sources com- evidence for complacency, which they further pared with the answers provided before its con- attributed to a lack of vigilance, following the sultation. Yet, in 11% (McKibbon & Fridsma, interpretation offered by Mosier and Skitka 2006) and 7% (Westbrook et al., 2005) of all (1996) for omission errors. cases, the search of electronic sources misled In contrast to the strong evidence for omission the physicians, who changed an initially correct errors, Alberdi et al. (2004) did not find evidence answer to an incorrect one after consulting the for commission errors in case of falsely placed electronic information source. prompts (Case d). However, this kind of bias was Although these results provide evidence that further investigated in a follow-up study (Alberdi physicians make commission errors when using et al., 2008). On the basis of a reanalysis of data electronic aids, the studies did not use a control available from a large-scale clinical trial in the group given nonelectronic resources, making United Kingdom focusing on the effectiveness of Automation Complacency and Bias 395 automated detection aids in breast screening commission errors, their results provide additional (Taylor, Champness, Given-Wilson, Potts, & indirect evidence for the existence of automation Johnston, 2004), the authors explored the possi- bias effects. For example, Rovira et al. (2007) ble consequences of falsely placed prompts. The investigated the effects of different automated results provided some evidence, albeit weak, for aids on military decisions under time pressure. automation bias in terms of commission errors. Specifically, they explored to what extent auto- Falsely placed prompts significantly raised the mated aids differing in LOA (information automa- probability by 12.3% that the prompted areas tion vs. three levels of decision automation) and actually were marked as malignancy compared overall reliability (60% vs. 80%) affect the speed with an unaided condition. and quality of command-and-control decisions Process control. Even stronger effects of involving the identification of most dangerous automation bias have been reported in a recent enemy targets and deciding which friendly unit series of studies investigating performance might be the best choice to combat it. consequences of an automated decision aid in a As expected, all of the automated aids imp­ simulated process control task (Bahner, Elepfandt, roved performance when they provided accu- & Manzey, 2008; Bahner, Huper, et al., 2008; rate advice. However, in case of inaccurate Manzey, Reichenbach, & Onnasch, 2008, 2009). recommendations, clear performance costs were In this research, the automated aid supported identified compared with an unsupported (man- fault identification and management tasks by ual) control condition. Decision accuracy declined providing operators with automatically gener- from 89% in the manual condition to 70% in ated diagnoses for system faults and recommen- supported conditions when the aid provided dations for how to deal with them. Evidence for incorrect recommendations, pointing to a sub- both omission and commission errors was stantial number of commission errors in the lat- found, with 20% to 50% of participants com- ter condition. Furthermore, some evidence was mitting a commission error in case the aid pro- found that these effects were moderated by the vided a wrong fault diagnosis for the first time LOA and the overall reliability of the aid. (first-failure effect). Because these studies Performance impairments in case of inaccurate also addressed possible links between compla- automation were most pronounced if the aid cency and automation bias, we describe them in provided a high level of support of decision- more detail in a subsequent section in which the making functions (i.e., provided a specific rec- two issues are discussed jointly. ommendation for an optimum decision) and Command and control. Finally, automation when the overall level of reliability was high. bias has also been recognized to represent an Whereas the former effect parallels the findings important issue with respect to intelligent deci- concerning the impact of status versus com- sion support systems for command-and-control mand displays on automation bias in aviation operations in the military domain (e.g., Crocoll (Crocoll & Coury, 1990; Sarter & Schroeder, & Coury, 1990; Cummings, 2003, 2004; Rovira 2001; see earlier discussion), the latter corre- et al., 2007). According to Cummings (2004), sponds to the impact of reliability on compla- automation bias effects in interaction with auto- cency effects in supervisory control (Bailey & mated decision aids have contributed to several Scerbo, 2007; R. Parasuraman et al., 1993). fatal military decisions, including inadvertent killing of friendly aircrews by U.S. missiles Factors Influencing Automation Bias during the Iraq War. Research in this domain Compared to research on automation-related has addressed issues of the appropriate LOA complacency, considerably less is known about that decision support systems should be set at relevant factors that modulate the degree of and its impact on decision making in case of automation bias. Factors to be taken into account inaccurate recommendations (Crocoll & Coury, include different aspects of system properties as 1990; Cummings, 2003; Rovira et al., 2007). well as the task context. Although these studies did not provide detailed System properties. As should be evident from information about the frequency of omission or the foregoing review of research, the strength of 396 June 2010 - Human Factors automation bias effects seems to depend at least an earlier observation that suggested that the to some extent on the LOA and the reliability of strength of automation bias in terms of omis- an automated aid. Specifically, there is evidence sion errors might be influenced by the degree to that automated aids that only support processes which pilots perceive themselves as account- of information integration and analysis may able for the automated tasks (Mosier et al., lead to lower automation bias effects (in terms 1998). The experiment involved 181 nonpilots of commission errors) than aids that provide who had to complete five trials with the same specific recommendations for certain actions W/PANES simulation described earlier. In the based on an assessment of the available infor- nonaccountable condition, participants were told mation (Crocoll & Coury, 1990; Rovira et al., that their performance would not be recorded 2007; Sarter & Schroeder, 2001). and that the main objective of the experiment Another factor that may have an impact on was to obtain their subjective evaluation of the automation bias effects includes the provision task environment. Four other groups were ins­ of system confidence information together with tructed to be particularly accountable so as to an automatically generated recommendation, as reach different performance objectives (e.g., suggested by another study addressing effects maximize overall performance, tracking perfor- of an automated aid on responses to in-flight mance, or speed or accuracy in the waypoint icing encounters (McGuirl & Sarter, 2006). and gauge-monitoring tasks) and that they would Using a similar paradigm as in the study of have to justify their performance in a debriefing Sarter and Schroeder (2001), they investigated interview after the experiment. effects of decision aids that not only provided The authors found that participants who were information about the icing condition (status instructed to be accountable for overall perfor- display) or specific recommendations for proper mance or the accuracy in the waypoint and actions (command display) but also confidence gauge-monitoring tasks committed significantly values for each. The confidence information less omission and commission errors than did was updated on a trial-by-trial basis and was the participants of the nonaccountable group or presented to the participants in a separate trend the groups who particularly were made account- display. Providing this additional information able for quick responses or their specific per- led to less performance decrements in case of formance in the tracking task. Furthermore, inaccurate aid recommendations compared with par­ticipants who felt responsible for overall per- a condition in which the additional confidence formance or accuracy also showed more atten- information was not available, that is, the pilots tive automation verification behavior than did received only information concerning the over- participants in all the other groups. The latter all reliability of the aid. Clearly, pilots sup- effect was shown by providing participants a ported by the advanced aid were better able to second monitor with which they could actively assess the validity of the aids’ single recommen- check the validity of any recommendations pro- dations and to make less biased decisions about vided by the aid in case of doubts. Participants whether to comply with the aid. in the nonaccountable group made significantly Task context. Most of the available knowl- less use of this device than participants feeling edge about the impact of task context factors on accountable for overall performance or the automation bias, thus far, has been obtained in a accuracy of the automated tasks. An even smaller series of studies conducted by Mosier, Skitka, tendency to cross-check the automation was and colleagues (Mosier, Skitka, Dunbar, & found in groups who felt particularly account- McDonnell, 2001; Skitka, Mosier, & Burdick, able for response speed or tracking performance, 2000; Skitka, Mosier, Burdick, & Rosenblatt, respectively. Although these results may be taken 2000). Specifically, three different factors were as evidence that accountability might be a rele- investigated, including social accountability, vant factor reducing automation bias, the results teams versus individuals, and instruction and are not fully conclusive in this respect. An alter- training interventions. native view is that the results might reflect a Skitka, Mosier, and Burdick (2000) exam- higher level of motivation and effort in the account- ined the impact of accountability on the basis of able groups induced by providing participants Automation Complacency and Bias 397 of these groups with some specific performance tasks, a finding that provides some support for the goals (Locke & Latham, 1990). cognitive-miser hypothesis of automation bias. Another set of two experiments explored to what extent the presence of a second crewmem- Summary ber in combination with several training inter- Human decision making can be biased when ventions or explicit prompts to cross-check the supported by imperfect automated decision aids. automation would moderate the degree of auto- Whereas this bias is relatively benign and actu- mation bias (Mosier et al., 2001; Skitka, Mosier, ally can be beneficial when a decision aid pro- Burdick, et al., 2000). The first of these studies vides correct recommendations (e.g., by speeding (Skitka, Mosier, Burdick, et al., 2000) again up decision making), it results in omission and involved nonpilots working on the W/PANES commission errors when the decision aid is task. In the two-person conditions, both crew- wrong. Evidence of both kinds of errors has been members had individual tasks but were instructed reported from several domains, including avia- to work redundantly on the waypoint or gauge- tion, medicine, process control, and command- monitoring task. Training interventions included and-control operations in military contexts. various briefings, ranging from just alerting The results show that automation bias repre- participants to verify the automation to a detailed sents a robust phenomenon that (a) can be found briefing about the nature of omission and com- in different settings, (b) occurs in both naive mission error as risk factors in human-automation and expert (e.g., pilots) participants, (c) seems interaction. In addition, conditions were com- to depend on the LOA and the overall reliability pared in which the participants did or did not of an aid, (d) cannot be prevented by training or receive explicit prompts to verify the automation, explicit instructions to verify the recommenda- which in the first case were presented along tions of an aid, (e) seems to be depend on how with the aid’s directive. accountable users of an aid perceive themselves The presence of a second crewmember did for overall performance, and (f) can affect deci- not affect the strength of automation bias for sion making in individuals as well as in teams. either omission or commission errors. This over- Interestingly, the first four of these results have all pattern of effects was replicated in a second also been found in studies of automation-related study involving 48 experienced glass cockpit complacency (the fifth, that is, impact of account- pilots (Mosier et al., 2001). The pilots were ability on complacency, and sixth, that is, com- required to perform missions in a part-task placency in teams, have not yet been systematically flight simulator similar to those in the previ- investigated). The common findings suggest ously described study of Mosier et al. (1998). that complacency and bias might be linked, a Neither detrimental nor beneficial effects of possibility that we discuss in more detail in the crew versus individual performance were found next section. with respect to omission and commission errors. Furthermore, neither of the training and instruc- TOWARD AN INTEGRATED MODEL OF tion interventions had any effect on the strength COMPLACENCY AND BIAS of automation bias. However, what was found is that the rate of omission errors varied between Theoretical Links individual participants and with the criticality Although the concepts of complacency and of events in terms of possible consequences if automation bias have been discussed separately they were missed. The latter effect confirms as if they were independent, they share several earlier results of Mosier et al. (1998) and may commonalities, suggesting they reflect different be taken as evidence that automation bias might aspects of the same kind of automation misuse not be related just to an automation-induced (cf. R. Parasuraman & Riley, 1997). Omission vigilance decrement but to a reallocation of errors with decision-aiding systems provide the attentional resources attributable to a delegation most obvious link between the two concepts, of responsibility to the automated aid. However, given that they occur when decision makers fail this reallocation of resources does not seem to to act because they were not informed of an be used to improve performance in concurrent imminent system problem by the automation. 398 June 2010 - Human Factors

Human operators thus rely on the alerting func- that at least some instances of commission tion of the aid at the expense of attentive moni- errors might be explained within the same atten- toring of important environmental cues. Such tional framework suggested for complacency inadequate monitoring clearly corresponds to and shown previously in Figure 3, substituting automation-induced complacency (Mosier & “Information Source A” for “Task A,” and so on. Skitka, 1996). To date, there is little empirical evidence to Similarly, complacency-like effects may also allow for an assessment to what extent commis- be responsible for the occurrence of commis- sion errors truly represent a bias in weighting sion errors. According to Skitka et al. (1999), information from different sources (automation “Commission errors can be the result of not vs. own information sampling) or a decision seeking out confirmatory or disconfirmatory bias reflected in neglecting to verify automa- information, or discounting other sources of tion. However, the previously described find- information in the presence of computer-generated ings of Skitka et al. (1999; Skitka, Mosier, & cues” (p. 993). The latter alternative, discount- Burdick, 2000) do support the view that neglect ing of contradictory information, reflects bias in of automation verification might constitute a decision making in a strict sense. Having con- major source of commission errors, as does a tradictory information from different sources, the study by Bahner, Huper, et al. (2008), which we operator, for some reason, cedes greater author- describe later. ity to the automated aid than to the other sources. This attitude could reflect greater trust in auto- Common Issues mation than in one’s own ability or difficulties The concepts of complacency and automation to comprehend the perceived information bias can be viewed as indications of automation correctly. misuse, that is, as a behavioral consequence However, the former alternative, following related to inappropriate overreliance on auto- the aid’s recommendation without verification, mation. This characterization was challenged by reflects a qualitatively different kind of decision Moray (2003; Moray & Inagaki, 2000), at least bias. In this case, the bias in favor of an auto- with respect to complacency. Moray proposed mated aid is expressed in operators’ relying so that what has been characterized as “compla- much on the proper function of an aid that they cent” behavior, that is, operators’ neglecting to neglect their own sampling of information and monitor automation, in fact may represent a become more selective in attending to different rational strategy. Moray argued that when oper- information sources. This kind of automation ators have several tasks to attend to, their moni- bias resembles what has been referred to as toring of automation should be in proportion to automation complacency in supervisory control its perceived reliability. Given that highly reli- tasks, given that the failure to verify (not attend- able automation will fail only very rarely, then ing to the raw data) indicates a reallocation of the rational strategy would be to monitor it also attention. only very infrequently. A natural consequence Furthermore, the cognitive-miser hypothesis of such a monitoring strategy is that when oper- of automation bias in high task load situations ators have other tasks to perform they will occa- could also reflect complacency in addition to a sionally miss automation failures because their strict decision bias. Similar to the operator who attention will be allocated elsewhere. tends to monitor an automated process inade- Up to this point, Moray’s (2003; Moray & quately if the workload is high, the user of a Inagaki, 2000) argument does not differ from decision aid may trust it to the extent that he or the framework for complacency we have pre- she directly follows the automatically generated sented that links it to a multitask attentional advice without making the attentionally demand- strategy (Figure 3). However, Moray went fur- ing effort of cross-checking its validity against ther in arguing that complacency should be other available and accessible information. This inferred only if operators monitor automation view points to an overlap between the concepts of less frequently than the optimal value for a par- automation complacency and bias and suggests ticular system. If, on the other hand, they monitor Automation Complacency and Bias 399 more often than the optimal value, they could include such cost and value parameters and has be called “skeptical.” Moray suggested that an been found to explain well observers’ visual operator who monitored at the optimal fre- scanning patterns (or lack thereof) in a number quency should be characterized as “eutactic” or of different tasks. “well calibrated”—similar to the concept of an We agree with Moray and Inagaki’s (2000) operator whose trust is calibrated to the actual view that complacency should ideally be evalu- reliability of automation (Lee & See, 2004). ated independently of the outcome of insuffi- What is the optimal (or normative) monitoring cient monitoring—for example, not detecting rate? Moray suggested that the Shannon- an automation failure. Most previous studies, Weaver-Nyquist sampling theorem provides a including the first study by Parasuraman et al. basis for determining the optimum. This theo- (1993), inferred complacency from detection rem states that to perfectly reproduce a continu- performance alone and not from an independent ously varying process (e.g., an analog signal) measure of monitoring. Studies measuring eye from intermittent samples (e.g., digital values), movements provide an exception (Bagheri & the frequency of sampling should be at least Jamieson, 2004; Metzger & Parasuraman, 2005), twice the frequency of the highest frequency in but eye fixation rates still need to be compared the continuous or analog signal. with optimal monitoring rates. Hence, Moray Does human sampling behavior also follow and Inagaki suggested that studies on compla- the sampling theorem? An early study by Senders cency to date had not provided convincing evi- (1964) provides some supporting evidence. He dence for the phenomenon of automation-related had participants monitor up to six gauges with complacency. continuously varying values to detect out-of- Moray and Inagaki’s (2000) critique can also limit targets. The frequency (or bandwidth) of be applied to studies on automation bias. As the continuously varying signals ranged from described previously, although operators com- slow to fast. Senders found that participants’ mit omission or commission errors in using fixations on a particular gauge were directly such aids, such errors may not necessarily indi- proportional to the bandwidth of the signal dis- cate that the operators were complacent. Such a played on that gauge (see also Moray, 1984). conclusion would be warranted only if it could Moray and Inagaki’s (2000) suggestion con- be shown that these performance consequences cerning the necessity of comparing actual and indeed were related to operators’ monitoring or optimal monitoring rates in automated systems verifying automation behavior less frequently is important. However, their proposal that the than the rate indicated by a normative model. sampling theorem provides a basis for deter- mining the optimal rate for monitoring is difficult Evidence for an Integrative Concept of to test outside of simple laboratory experiments Complacency and Automation Bias (as in Senders, 1964), because the frequency A recent series of experiments has taken this content of many real-world information sources caveat into account and provided evidence for is difficult or impossible to compute. Nevertheless, the proposed link between complacency and there may be other ways in which a normative automation bias (Bahner, Elepfandt, et al., 2008; value for monitoring automation can be calcu- Bahner, Huper, et al., 2008; Manzey et al., 2009; lated, as described in a later section. Moreover, Manzey, Reichenbach, et al., 2008). These stud- the sampling models posited by Moray (2003; ies involved the use of a process control micro- Moray & Inagaki, 2000) and Senders (1964) do world (AutoCAMS; Lorenz, Di Nocera, Rottger, not consider the potential cost of sampling (e.g., & Parasuraman, 2002; Manzey, Bleil, et al., extensive eye or head movements or search 2008) that simulates an autonomous life support time) or the value provided by sampled infor- system for a space station consisting of five mation, which can pose severe limits on the rate subsystems (e.g., O2, CO2, pressure) critical to of sampling (Sheridan, 1970, 2002). The salience, maintaining cabin atmosphere. The primary effort, expectancy, and value (SEEV) model of task of the operator involved supervisory control Wickens et al. (2007), on the other hand, does of the different subsystems, including diagnosis 400 June 2010 - Human Factors and management of system faults, which regarded as complacent to different degrees, occurred occasionally and unpredictably. Operators dependent on how much they deviated from the also had two secondary tasks: a prospective optimal sampling strategy. This operational def- memory task and a simple reaction time task. inition of complacency made it possible to eval- The primary task was supported by an auto- uate the level of complacency in interaction mated aid that, in case of failures of single sub- with the aid independent of its possible perfor- systems, provided an automatically generated mance consequences. fault diagnosis as well as recommendations for These results provide direct empirical evi- fault management. dence for the proposed relationship between Bahner, Huper, et al. (2008) had two groups perceived reliability of automation, complacency, of well-trained engineering students perform and automation bias. In both experiments, par- the AutoCAMs simulation. One group was told ticipants of all experimental groups exhibited a that the aid would work highly reliably, although complacency effect at least to some extent, that not perfectly, and that they should carefully is, did not fully verify the automatically gener- cross-check each diagnosis before accepting it ated diagnoses for system faults completely (information group). The second group received before accepting it. Figure 4 from Bahner, the same information but was additionally exposed Huper, et al. (2008) shows that operators in the to some rare automation failures during training information group were more complacent (sam- (experience group). Automation failures could pled fewer parameters) than were those in the include a failure of the diagnostic function, that experience group. This figure also shows that is, the aid provided wrong diagnoses for 2 out of the degree of operator verification was less than 10 system faults (Bahner, Huper, et al., 2008), that required by the normative model, irrespec- or a failure of the alarm function, that is, the aid tive of training. Yet, as expected, the degree of failed to indicate 2 out of 10 system faults the complacency effect was moderated by the (Bahner, Elepfandt, et al., 2008). On the basis of experiences the participants had with the aid earlier research (e.g., Lee & Moray, 1992), it during training. Participants who were exposed was assumed that the practical experience of to failures of the diagnostic function of the aid, automation failures should reduce overall trust but not those exposed to failures of the alarm in the system. function (Bahner, Elepfandt, et al., 2008), Bahner, Huper, et al. (2008) assessed com- showed a significantly lower level of compla- placency by measuring the extent to which par- cency than participants who were informed just ticipants verified the automation’s diagnosis that the aid may fail. before accepting it. Participants were provided In addition, a clear link between the level of independent access (via mouse click) to all rel- complacency and automation bias was also found: evant system information (e.g., tank levels, flow 21% of participants committed commission errors, rates at different valves, history graphs display- which occurred despite the fact that the automa- ing the time course of system parameters), tion error could easily be recognized (Bahner, needed to verify the aid’s diagnoses. This pro- Huper, et al., 2008). Analyses of information- cedure allowed the investigators to assess the sampling behavior revealed that these errors level of operator complacency in interaction with indeed were related to a generally higher level the aid by contrasting the actual information of complacency in this subgroup of participants sampling behavior of operators with a “norma- compared with those who did not commit this tive model” of optimal behavior. According to this kind of error (see Figure 5). Bahner, Elepfandt, logic, any participant who accessed just the infor- et al. (2008) found even more participants (18 out mation appropriate to verify a given diagnosis of 24) committed a commission error when before accepting it was regarded as showing the aid provided a wrong diagnosis for the first noncomplacent (eutactic) behavior in interac- time. Inspection of the verification behavior just tion with the aid. However, participants sam- before committing the error again revealed pling less information than that necessary to that 80% of the participants followed the completely verify the aid’s recommendation were false recommendation because of insufficient Automation Complacency and Bias 401

necessary to verify that the automated advice was wrong. The underlying determinants of this latter effect were further investigated in a follow-up experiment (Manzey, Reichenbach, et al., 2008) in which participants worked with three differ- ent decision aids that differed in their amount of support (LOA). The first aid provided only an automatically generated diagnosis for a given system fault but left it to the operator to plan and implement all necessary actions; the second one provided additional recommendations for necessary actions, which, however, had then to Figure 4. Proportion of parameters sampled prior to act- be implemented manually by the operator; and ing on the recommendation of an automated decision the third performed fault management autono- aid, for different faults and for the information and expe- mously if the operator confirmed the proposed rience groups. From “Misuse of Automated Decision diagnosis and plan of interventions. Aids: Complacency, Automation Bias and the Impact Independent of the LOA of the decision aid, of Training Experience,” by E. Bahner, A.-D. Huper, approximately 43% of operators did not detect a and D. Manzey, 2008, International Journal of Human- wrong diagnosis of the aid when it occurred for Computer Studies, 66, p. 694. Copyright 2008 by the first time. Analyses of information-sampling Elsevier Science. Reprinted with permission. behavior revealed that half of the participants made this commission error because of a clear complacency effect, as reflected in an incomplete sampling of information needed for automation verification. The other half of participants again committed this kind of error despite the fact that they had sampled all system parameters neces- sary to verify the diagnosis provided by the aid. However, more detailed analyses of this effect revealed an interesting difference between these participants and participants who detected the wrong diagnosis. Participants who correctly recognized that the aid’s advice was wrong showed a sharp increase in the average time needed to process a sampled system parameter Figure 5. Proportion of relevant parameters sampled in case that its information did not fit the diag- among participants who did and did not commit a com- nosis of the aid, compared with trials in which mission error. From “Misuse of Automated Decision the aid’s diagnoses were correct. Clearly, these Aids: Complacency, Automation Bias and the Impact participants became aware of the contradictory of Training Experience,” by E. Bahner, A.-D. Huper, information and needed time to evaluate it. and D. Manzey, 2008, International Journal of However, this effect did not emerge for partici- Human-Computer Studies, 66, p. 695. Copyright pants who sampled all relevant information but 2008 by Elsevier Science. Adapted with permission. nevertheless committed a commission error. These participants did not show any difference in processing time when the aid’s diagnosis was cross-checking of the relevant system informa- correct and when it was wrong. tion needed to verify the aid’s recommendation. Given this finding, the commission error Yet 20% of the participants followed the recom- committed by these participants cannot be exp­ mendation despite seeking out all parameters lained by a decision bias view associated with a 402 June 2010 - Human Factors discounting of contradictory information. In seems to be more related to a hindsight justifi- addition, an explanation in terms of confirma- cation of the final decision than to a misreading tion bias in processing the sampled information of the parameters. or difficulties of comprehension is also unlikely, This finding is particularly interesting because given that the participants were very well it provides evidence that commission errors trained and that the system parameter always associated with complete (optimal) information contradicted the aid’s diagnosis in an unequiv- sampling may result not necessarily from a mis- ocal way. Thus, the two types of decision bias weighting of contradictory information but from discussed previously, confirmation bias and dis- a complacency-like, covert attention effect counting bias, can be ruled out as explanations, (Duley et al., 1997), which, in this case, was at least for this group. Rather, the commission reflected in a withdrawal of attentional resources errors in this group seem to be attributable to a from processing the available (and looked-at) sort of looking-but-not-seeing effect, analogous information. to what in other contexts has been referred to as In summary, two main conclusions can be “inattentional blindness” (Mack & Rock, 1998) drawn from these results. The first is that it is and which also has been suggested to underlie necessary to decompose the underlying determi- complacency effects in supervisory control nants of commission errors that operators make tasks (Duley et al., 1997). while interacting with automated decision aids. Direct evidence for this latter interpretation Three different sources can be distinguished: (a) was provided by another AutoCAMS experi- an actual, overt redirection of visual attention in ment that investigated this hypothesis in more terms of reduced proactive sampling of relevant detail (Reichenbach, Onnasch, & Manzey, in information needed to verify an automated aid; press). In this study, the simulation was stopped (b) a more subtle effect reflected in less attentive immediately after the participants had commit- processing of this information, perhaps because ted a commission error, and the participants covert attention is allocated elsewhere (an effect were required to report (a) what system param- analogous to inattentional blindness); and (c) an eters they had accessed to verify the aid’s rec- active discounting of information that contra- ommendation and (b) what the values of these dicts the recommendation of the aid because of parameters were. Out of the 11 participants either difficulties in comprehension or an over- committing a commission error in this study, 6 reliance on the automated source. were found to have sampled only part of the The first two of these sources relate the system information that would have been nec- occurrence of automation bias to issues of selec- essary to check to verify the aid’s advice. The tive or less attentive processing of information remaining 5 participants committed the com- and seem to constitute a major source of bias in mission error despite checking all the parame- human interaction with decision aids. Such an ters that were necessary to realize that the effect provides strong evidence that not only automatically generated diagnosis was wrong. omission errors but also commission errors can However, only 1 of them actually was aware of be related to essentially the same attentional all the contradictory information but failed to processes that have been shown to underlie give an explanation for the final decision to complacency effects in supervisory control tasks. nevertheless follow the aid. This parallelism further suggests that what has The other 4 participants were not able to recall been referred to as complacency in supervisory correctly the information they had accessed. control studies (R. Parasuraman et al., 1993) Instead, they tended to report system values that and automation bias in studies examining use of would have to be expected, given that the advice decision aids (Mosier & Skitka, 1996), at least of the aid would have been correct. This effect to a large extent, might represent different man- replicates the phantom memory effect reported ifestations of overlapping automation-induced by Mosier et al. (1998). However, given that the phenomena. actual system values always differed consider- Second, given the operational definition of ably from the ones to be expected, this effect complacency and automation bias in the set of Automation Complacency and Bias 403 studies we have described (e.g., Bahner, Huper, of complacency and automation bias, referred et al., 2008; Manzey et al., 2009), the view that to as “complacency potential” and “attentional the observed effects reflect an inevitable side bias” in information processing; (b) the differ- effect of a rational strategy used by operators in entiation between automation-induced atten- interacting with automated systems can be ruled tional phenomena and its possible performance out, as confirmed by a contrast of the actual consequences in terms of omission and com- automation verification behavior of operators mission errors; and (c) the dynamic and adap- with a normative (rational) model of information tive nature of complacency and automation sampling, as suggested by Moray and Inagaki bias, reflected in the two feedback loops. (2000). The results instead support the view that The first distinction capitalizes on a similar complacency and automation bias indeed repre- conceptual differentiation between two differ- sent a human performance cost of certain automa- ent aspects of complacency first proposed by tion system designs, that is, systems characterized Singh et al. (1993a, 1993b) and later adopted by high reliability and high levels of automation, by Manzey and Bahner (2005). Complacency particularly in high task load situations. Such costs potential is conceived of as a behavioral ten- need to be considered as potentially serious risk dency to react in a less attentive manner in factors when evaluating the overall efficiency and interacting with a specific automated system. safety of human-automation systems. It is assumed to be influenced by the (per- ceived) reliability and consistency of the sys- An Integrated Model of Complacency tem, the history of experiences of the operator and Automation Bias with this system (see later discussion), and Our analysis suggests that automation-induced individual characteristics of the human opera- complacency and automation bias represent tor. The assumption that this tendency to overrely closely linked theoretical concepts that show on a certain system is system specific contrasts considerable overlap with respect to the under- with earlier conceptions that have conceived lying processes. We propose an integration such of complacency potential as a general ten- that they represent different manifestations of dency toward automation in general (Singh similar automation-induced phenomena, with et al., 1993a). However, previous findings attention playing a central role. Furthermore, (Prinzel, 2002; Prinzel et al., 2001) indicate complacency and automation bias, although that both perceived system characteristics as affected by individual differences, cannot be well as individual differences determine the considered as just another type of human error final level of complacency in supervisory con- but constitute phenomena that result from a com- trol of an automated system. plex interaction of personal, situational, and Relatively less is known about the nature of automation-related characteristics. individual characteristics contributing to differ- An integrated model of complacency and auto- ences in complacency potential. As has been mation bias is shown in Figure 6. Note that this discussed in some detail previously, differences model is not meant to cover all kinds of automa- in attitudes toward technology may play a role. tion bias but only those that are related to a selec- Yet other findings suggest that personality traits tive or less attentive processing of information. might also contribute to individual differences Furthermore, the model does not address instances in complacency, for example, self-efficacy (Prinzel, of complacency and automation bias that solely 2002), as well as trust in automation in general reflect performance consequences stemming from (Merritt & Ilgen, 2008). Clearly, more research operators’ lacking appropriate system knowledge, is needed in this area. that is, instances in which the automated system has However, even high complacency potential to be relied on because the human operator does does not guarantee that individuals will neces- not possess the necessary competency or knowl- sarily exhibit selective or less attentive informa- edge to verify its proper function. tion processing in interaction with an automated The three main critical features of this model system. As shown earlier, complacency and include (a) the distinction between two aspects automation bias effects emerge primarily when 404 June 2010 - Human Factors

Negative Feedback Loop

System Properties Task Context Level of Automation Concurrent tasks, workload, Reliability, constancy of function Consistency allocation, accountability re Performance ilu fa Consequences: to Error of Omission Attentional Bias in Au Information Processing Error of Commission – “Complacency Inappropriate reallocation Loss of SA + Potential” of attentional resources A Selective Information uto n No Performance Processing o rm Consequences al

Person Individual State Technology-related Operator state attitudes, Self-Efficacy, Motivation Personality Traits Positive Feedback Loop “Learned Carelessness” Figure 6. An integrated model of complacency and automation bias. task load is high (e.g., multitask demands). This described as a “heuristic use of automation” that fits with the general understanding of complacency may or may not lead to overt performance conse- and automation bias as representing issues quences, depending on whether the automation related to task prioritization and allocation of fails. More specifically, the model assumes that the attentional resources. Furthermore, operator immediate performance consequence of a selec- state variables may also influence to what extent tive or less attentive processing of information is high complacency potential leads to an overt a loss of situation awareness. This has no conse- attentional bias in interaction with automation. quences as long as the automation works prop- For example, a recent study found that fatigued erly but directly leads to errors of omission or operators interacted in a more careful and less commission in case of automation failure. complacent manner with a decision aid than did This aspect of the model has particular impli- alert operators (Manzey et al., 2009). cations for an appropriate operational definition A second important feature of the integrated of the concepts of complacency and automation model represents the distinction between com- bias in future research. Specifically it suggests placency and automation bias as an attentional that operational definitions of complacency or effect and its performance consequences in terms automation bias as behavior need to be based of omission or commission errors. This distinc- on direct or indirect behavioral indicators of atten- tion directly relates to the analysis of the link tion allocation. Direct indicators may be derived between complacency and automation bias pre- from eye-tracking analyses or other indicators sented previously. It emphasizes that complacency of monitoring or information-sampling behav- and automation bias in interaction with auto- ior. Indirect indicators of attention may include mated systems is reflected in an automation- assessments of a reallocation of attentional induced withdrawal or reallocation of attentional resources by means of secondary-task methods. resources. These attentional effects constitute a However, as proposed by Moray (2003; Moray conscious or unconscious response of the human & Inagaki, 2000), the mere fact that automated operator induced by overtrust in the proper systems influence allocation of attentional function of an automated system. In this sense, resources cannot be regarded as an indication the effects reflect what Mosier and Skitka (1996) of complacency or automation bias. Such a Automation Complacency and Bias 405 conclusion is justified only if the observed multitasking environments where operators effects are compared with some normative have to perform manual tasks as well as super- model of “optimal attention allocation” in inter- vise automation. Automation complacency can action with a given system. Defining appropri- thus be understood in terms of an attention allo- ate normative models for interaction with given cation strategy whereby the operator’s manual automated systems represents an important chal- tasks are attended to at the expense of the auto- lenge for future research. mated task. Automation bias is reflected in omis- Finally, following an earlier proposal by sion and commission errors made by operators Manzey and Bahner (2005), the model con- interacting with imperfect decision aids. Auto­ ceives of complacency and automation bias as mation bias can be conceived of as a special case resulting from an adaptive process that dyn­ of human decision biases, such as confirmation amically develops over time on the basis of bias and discounting bias. However, recent evi- experiences an operator has in interaction with dence suggests that at least some forms of auto- an automated system. As shown in Figure 6, mation bias result from attentional processes there is a positive feedback loop that leads to a similar to those involved in automation-related rise in complacency potential over time. Given complacency. Thus, complacency and automa- the usually high reliability of automated sys- tion bias represent different manifestations of tems, even highly complacent and biased overlapping automation-induced phenomenon, behavior of operators rarely leads to obvious with attention at the center. performance consequences. Over time, the lack Our integrated attentional model provides a of performance consequences might affect com- heuristic framework for further research on placency potential by inducing a cognitive pro- com­placency and automation bias. The model cess that resembles what has been referred to as proposes that attentional factors contribute to “learned carelessness” in the domain of work some but not all forms of automation bias. safety (Frey & Schulz-Hardt, 1997) and has This suggests that future research would be already been included in cognitive models of desirable on the relative importance of atten- pilot performance (Luedtke & Moebus, 2005). tional effects versus discounting of contradic- On the other hand, experiences of automa- tory information as determinants of automation tion failures can be assumed to initiate a nega- bias. This goal could be achieved through tive feedback loop, reducing the complacency additional studies that not only investigate the potential in interaction with a given system. performance consequences of automation bias This pattern is suggested by empirical results but also conduct microanalyses of information- showing that even single automation failures sampling strategies, using either eye-tracking may considerably reduce the trust in an auto- or explicit verification procedures as devel- mated system (Lee & Moray, 1992; Madhavan, oped by Bahner, Huper et al. (2008) or both. In Wiegmann, & Lacson, 2006) and also affect the particular, the extent to which contradictory strength of complacency and automation bias information is available, its saliency, and the effects (Bahner, Huper, et al., 2008). In this cost of obtaining it may determine the degree respect, our model of complacency and automa- of influence of attentional factors. tion bias resembles the more general model of In addition, further research is required on trust and reliance in automation proposed by the relative importance and interplay of the two Lee and See (2004) that includes similar feed- proposed feedback loops in our integrated model back loops. (see Figure 6). We propose a positive feedback loop that increases complacency potential while CONCLUSION negative feedback reduces it. Studies that var- Automation-related complacency and auto- ied the relative effectiveness of these feedback mation bias have previously been viewed as two loops would allow for a better understanding of separate and independent potential human per- the dynamics of development of complacency formance costs of certain automation designs. and automation bias in human interaction with Automation complacency is generally found in automation. 406 June 2010 - Human Factors

Finally, individual differences were consid- • Although both concepts have been described sep- ered only briefly in this article, but there is pre- arately, they share several commonalities with liminary evidence that personal, experiential, respect to the underlying attentional processes. and situational factors influence automation • Empirical evidence suggests that attentional fac- complacency and bias. There is currently grow- tors contribute to many but not all forms of auto- ing interest in using knowledge of individual mation bias. differences to improve design and training efforts • An integrated model is presented that views com- in human factors and ergonomics generally placency and automation bias as resulting from a (Szalma, 2009). Similar research in the area of complex interplay of personal, situational, and automation complacency and bias would appear automation-related factors. to be worthwhile. • The integrated model can provide design guid- Our integrated model of automation compla- ance and serve as a heuristic framework for fur- cency and bias also provides a framework within ther research. which practical applications, particularly those aimed at mitigating complacency and bias in REFERENCES automated systems, can be better examined. Alberdi, E., Povyakalo, A. A., Strigini, L., & Ayton, P. (2004). Several studies have shown that the LOA and the Effects of incorrect computer-aided detection (CAD) output on type of processing supported by automation can human decision-making in mammography. Academic Radiology, affect its benefits and costs, including compla- 11, 909–918. Alberdi, E., Povyakalo, A. A., Strigini, L., Ayton, P., & Given- cency and situation awareness (R. Parasuraman Wilson, R. (2008). CAD in mammography: Lesion-level ver- et al., 2000). There is also preliminary evidence sus case-level analysis of the effects of prompts on human that automation bias can be mitigated with aiding decisions. International Journal of Computer-Assisted Radiology and Surgery, 3, 115–122. of information analysis as opposed to decision Bagheri, N., & Jamieson, G. A. (2004). Considering subjective support, although additional research is needed trust and monitoring behavior in assessing automation- to confirm the efficacy of this design option. induced ‘‘complacency.” In D. A. Vicenzi, M. Mouloua, & Additional promising points of attack include O. A. Hancock (Eds.), Human performance, situation awareness, and automation: Current research and trends a change of situational conditions (including (pp. 54–59). Mahwah, NJ: Erlbaum. function allocation; see Manzey, Reichenbach, Bahner, J. E., Elepfandt, M., & Manzey, D. (2008). Misuse of diag- et al., 2008; Parasuraman, Mouloua, & Molloy, nostic aids in process control: The effects of automation misses 1996) or providing practical experience with a on complacency and automation bias. In Proceedings of the 52nd Meeting of the Human Factors and Ergonomics Society given automated system. Changes of situational (pp. 1330–1334). Santa Monica, CA: Human Factors and conditions may include taking actions to raise the Ergonomics Society. perceived accountability of operators or using Bahner, E., Huper, A.-D., & Manzey, D. (2008). Misuse of auto- flexible strategies of function allocation, which mated decision aids: Complacency, automation bias and the impact of training experience. International Journal of Human- both have been shown to reduce issues of com- Computer Studies, 66, 688–699. placency and automation bias in interaction with Bailey, N., & Scerbo, M. S. (2007). Automation-induced compla- different automated systems (R. Parasuraman cency for monitoring highly reliable systems: The role of task et al., 1996; Skitka, Mosier, & Burdick, 2000). complexity, system experience, and operator trust. Theoretical Issues in Ergonomics Science, 8, 321–348. Our integrated model also suggests the impor- Bainbridge, L. (1983). Ironies of automation. Automatica, 19, tance of a negative feedback loop in minimizing 775–779. complacency and bias effects (see Figure 6), Billings, C. E., Lauber, J. K., Funkhouser, H., Lyman, G., & Huff, E. M. (1976). Aviation Safety Reporting System which could be used to make users of automated (Technical Report TM-X-3445). Moffett Field, CA: National systems more resilient to complacency and auto- Aeronautics and Space Administration Ames Research Center. mation bias effects. Casey, S. M. (1998). Set phasers on stun: And other true tales of design, technology, and human error. Santa Barbara, CA: Aegean. KEY POINTS Corbetta, M. (1998). Frontoparietal cortical networks for directing attention and the eye to visual locations: Identical, independent, • Complacency and automation bias are phenomena or overlapping neural systems? Proceedings of the National that describe a conscious or unconscious response Academy of Sciences (USA), 95, 831–838. Craig, A. (1984). Human engineering: The control of vigilance. In of the human operator induced by overtrust in the J. S. Warm (Ed.), Sustained attention in human performance proper function of an automated system. (pp. 247–291). New York, NY: Wiley. Automation Complacency and Bias 407

Crocoll, W. M., & Coury, B. G. (1990). Status or recommendation: complacency in IFR-rated general aviation pilots. In selecting the type of information for decision aiding. In Proceedings of the International Symposium on Aviation Proceedings of the Human Factors Society 34th Annual Psychology (pp. 245–249). Columbus, OH: Association of Meeting (pp. 1525–1528). Santa Monica, CA: Human Factors Aviation Psychology. and Ergonomics Society. Gigerenzer, G., & Todd, P. A. (1999). Simple heuristics that make Cummings, M. L. (2003). Designing decision support systems for us smart. London, England: Oxford University Press. revolutionary command and control domains (PhD disserta- Gopher, D. (1996). Attention control: Explorations of the work of tion). University of Virginia, Charlottesville. an executive controller. Cognitive Brain Research, 5, 23–38. Cummings, M. L. (2004, September). Automation bias in intel- Gopher, D., Weil, M., & Siegel, D. (1989). Practice under varying ligent time critical decision support systems. Paper pre- priorities: An approach to the training of complex skills. Acta sented to the American Institute for Aeronautics and Psychologica, 71, 147–177. Astronautics First Intelligent Systems Technical Conference, Hardy, D., Mouloua, M., Dwivedi, C., & Parasuraman, R. (1995). Reston, VA. Retrieved from http://citeseerx.ist.psu.edu/ Monitoring of automation failures by young and older adults. viewdoc/summary?doi=10.1.1.91.2634 In Proceedings of the International Symposium on Aviation Davies, D. R., & Parasuraman, R. (1982). The psychology of vigi- Psychology (pp. 1382–1386). Columbus, OH: Association of lance. London, England: Academic Press. Aviation Psychology. Degani, A. (2001). Taming Hal: Designing interfaces beyond 2001. Hoffman, J. E., & Subramaniam, B. (1995). The role of visual atten- New York, NY: Macmillan. tion in saccadic eye movements. Perception and Psychophysics, Dekker, S. W. A., & Hollnagel, E. (2004). Human factors and folk 57, 787–795. models. Cognition, Technology, and Work, 6, 79–86. Hurst, K., & Hurst, L. (1982). Pilot error: The human factors. Dekker, S. W. A., & Woods, D. D. (2002). MABA-MABA or abraca- New York, NY: Aronson. dabra? Progress on human automation coordination. Cognition, Kahnemann, D., Slovic, P., & Tversky, A. (1982). Judgement under Technology, and Work, 4, 240–244. uncertainty: Heuristics and biases. New York, NY: Cambridge de Visser, E., & Parasuraman, R. (2007). Effects of imperfect University Press. automation and task load on human supervision of multiple Karau, S. J., & Williams, K. D. (1993). Social-loafing: A meta- uninhabited vehicles. In Proceedings of the 51st Annual analytic review and theoretical integration. Journal of Meeting of the Human Factors and Ergonomics Society Personality and Social Psychology, 65, 681–706. (pp. 1081–1085). Santa Monica, CA: Human Factors and Koshland, D. E. (1989). Low probability-high consequence acci- Ergonomics Society. dents. Science, 244, 405. De Waard, D., van der Hulst, M., Hoedemaeker, M., & Langer, E. J. (1982, April). Automated lives. Psychology Today, Brookhuis, K. A. (1999). Driver behavior in an emergency situ- pp. 60–71. ation in the automated highway system. Transportation Human Langer, E. J. (1989). Mindfulness. Reading, MA: Addison-Wesley. Factors, 1, 67–82. Layton, C., Smith, P. J., & McCoy, C. E. (1994). Design of coop- Domeinski, J., Wagner, R., Schoebel, M., & Manzey, D. (2007). erative problem-solving system for en-route flight planning: an Human redundancy in automation monitoring: Effects of social empirical evaluation. Human Factors, 36, 94–119. loafing and social compensation. In Proceedings of the Human Lee, J. D., & Moray, N. (1992). Trust, control strategies and alloca- Factors and Ergonomics Society 51st Annual Meeting tion of function in human-machine systems. Ergonomics, 35, (pp. 587–591). Santa Monica, CA: Human Factors and 1243–1270. Ergonomics Society. Lee, J. D., & Moray, N. (1994). Trust, self-confidence, and opera- Duley, J. A., Westerman, S., Molloy, R., & Parasuraman, R. (1997). tors’ adaptation to automation. International Journal of Effects of display superimposition on monitoring of automation. Human-Computer Studies, 40, 153–184. In Proceedings of the 9th International Symposium on Aviation Lee, J. D., & See, J. (2004). Trust in automation and technology: Psychology (pp. 322–326). Columbus, OH: Association of Designing for appropriate reliance. Human Factors, 46, 50–80. Aviation Psychology. Lee, J. D., & Seppelt, B. D. (2009). Human factors in automation Dzindolet, M. T., Beck, H. P., Pierce, L. G., & Dawe, L. A. (2001). design. In S. Nof (Ed.), Springer handbook of automation (pp. A framework of automation use (Report No. ARL-TR-2412). 417–436). New York, NY: Springer. Aberdeen Proving Ground, MD: Army Research Laboratory. Locke, E. A., & Latham, G. P. (1990). A theory of goal setting and Dzindolet, M. T., Pierce, L. G., Beck, H. P., & Dawe, L. A. (2002). task performance. Englewood Cliffs, NJ: Prentice Hall. The perceived utility of human and automated aids in a visual Lorenz, B., Di Nocera, F., Rottger, S., & Parasuraman, R. (2002). detection task. Human Factors, 44, 79–94. Automated fault-management in a simulated spaceflight Frey, D., & Schulz-Hardt, S. (1997). Eine Theorie der gelernten micro-world. Aviation, Space, and Environmental Medicine, Sorglosigkeit [A theory of learned carelessness]. In H. Mandl 73, 886–897. (Ed.), Bericht über den 40. Kongress der Deutschen Gesellschaft Luedtke, A., & Moebus, C. (2005). A case study for using a cogni- für Psychologie (pp. 604–611). Goettingen, Germany: Hogrefe. tive model of learned carelessness in cognitive engineering. In Funk, K., Lyall, B., Wilson, J., Vint, R., Niemczyk, M., Suroteguh, C., G. Salvendy (Ed.), Proceedings of the 11th International & Owen, G. (1999). Flight deck automation issues. International Conference of Human-Computer Interaction. Mahwah, NJ: Journal of Aviation Psychology, 9, 109–123. Erlbaum. Retrieved from http://www.lks.uni-oldenburg .de/down- Galster, S., Duley, J. A., Masalonis, A., & Parasuraman, R. (2001). load/abteilung/Luedtke_Moebus_crv.pdf Air traffic controller performance and workload under mature Mack, A., & Rock, I. (1998). Inattentional blindness. Cambridge, Free Flight: Conflict detection and resolution of aircraft self- MA: MIT Press. separation. International Journal of Aviation Psychology, 11, 71–93. Madhavan, P., Wiegmann, D. A., & Lacson, F. C. (2006). Automation Galster, S., & Parasuraman, R. (2001). Evaluation of countermea- failures on tasks easily performed by operators undermine trust sures for performance decrements due to automated-related in automated aids. Human Factors, 48, 241–256. 408 June 2010 - Human Factors

Manzey, D., & Bahner, J. E. (2005). Vertrauen in Automation als Mosier, K. L. (2002). Automation and cognition: Maintaining coher- Aspekt der Verlaesslichkeit von Mensch-Maschine-Systemen ence in the electronic cockpit. In E. Salas (Ed.), Advances in [Trust in automation as an aspect of dependability of human- human performance and cognitive engineering research (Vol. 2, machine-systems]. In K. Karrer, B. Gauss, & C. Steffens pp. 93–121). Amsterdam, Netherlands: Elsevier Science. (Eds.), Mensch-Maschine-Systemtechnik aus Forschung und Mosier, K. L., Palmer, E. A., & Degani, A. (1992). Electronic Praxis (pp. 93–109). Duesseldorf, Germany: Symposion. checklists: Implications for decision making. In Proceedings of Manzey, D., Bleil, M., Bahner-Heyne, J. E., Klostermann, A., the Human Factors Society 36th Annual Meeting (pp. 7–11). Onnasch, L., Reichenbach, J., & Röttger, S. (2008). AutoCAMS Santa Monica, CA: Human Factors and Ergonomics Society. 2.0 manual. Berlin, Germany: Berlin Institute of Technology, Mosier, K. L., & Skitka, L. J. (1996). Human decision makers and Chair of Work, Engineering and Organisational Psychology. automated decision aids: Made for each other? In R. Parasuraman Retrieved from http://www.aio.tu-berlin.de/?id=30492 & M. Mouloua (Eds.), Automation and human performance: Manzey, D., Reichenbach, J., & Onnasch, L. (2008). Performance- Theory and application (pp. 201–220). Mahwah, NJ: Erlbaum. consequences of automated aids in supervisory control: The Mosier, K. L., Skitka, L. J., Dunbar, M., & McDonnell, L. (2001). impact of function allocation. In Proceedings of the 52nd Meeting Aircrews and automation bias: The advantages of teamwork? of the Human Factors and Ergonomics Society (pp. 297–301). International Journal of Aviation Psychology, 11, 1–14. Santa Monica, CA: Human Factors and Ergonomics Society. Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998). Manzey, D., Reichenbach, J., & Onnasch, L. (2009). Human per- Automation bias: Decision-making and performance in high- formance consequences of automated decision aids in states of tech cockpits. International Journal of Aviation Psychology, 8, fatigue. In Proceedings of the 53rd Meeting of the Human 47–63. Factors and Ergonomics Society (pp. 329–333). Santa Monica, National Transportation Safety Board. (1997). Grounding of the CA: Human Factors and Ergonomics Society. Panamanian passenger ship Royal Majesty on Rose and May, P., Molloy, R., & Parasuraman, R. (1983, October). Effects of Crown shoal near Nantucket, Massachusetts, June 10, 1995 automation reliability and failure rate on monitoring perfor- (Report NTSB/MAR-97-01). Washington, DC: Author. mance in a multi-task environment. Paper presented at the annual Parasuraman, R. (2000). Designing automation for human use: meeting of the Human Factors Society, Santa Monica, CA. Empirical studies and quantitative models. Ergonomics, 43, McGuirl, J. M., & Sarter, N. B. (2006). Supporting trust calibration 931–951. and the effective use of decision aids by presenting dynamic Parasuraman, R., Cosenzo, K., & de Visser, E. (2009). Adaptive system confidence information. Human Factors, 48, 656–665. automation for human supervision of multiple uninhabited McKibbon, K. A., & Fridsma, D. B. (2006). Effectiveness of clinical- vehicles: Effects on change detection, situation awareness, and selected electronic information resources for answering primary mental workload. Military Psychology, 21, 270–297. care physician’s information needs. Journal of the American Parasuraman, R., Molloy, R., & Singh, I. L. (1993). Performance con- Medical Informatics Association, 13, 653–659. sequences of automation-induced “complacency.” International Merlo, J. L., Wickens, C., & Yeh, M. (2000). Effect of reliability on Journal of Aviation Psychology, 3, 1–23. cue effectiveness and display signaling. In Proceedings of the Parasuraman, R., Mouloua, M., & Molloy, R. (1996). Effects of 4th Annual Army Federated Laboratory Symposium (pp. 27–31). adaptive task allocation on monitoring of automated systems. College Park, MD: Army Research Laboratory. Human Factors, 38, 665–679. Merritt, S. M., & Ilgen, D. R. (2008). Not all trust is created equal: Parasuraman, R., & Riley, V. (1997). Humans and automation: Dispositional and history-based trust in human-automation Use, misuse, disuse, abuse. Human Factors, 39, 230–253. interactions. Human Factors, 50, 194–210. Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A Metzger, U., Duley, J., Abbas, R., & Parasuraman, R. (2000). Effects model for types and levels of human interaction with automa- of variable-priority training on automation-related complacency: tion. IEEE Transactions on Systems, Man, and Cybernetics, Performance and eye movements. In Proceedings of the IEA Part A: Systems and Humans, 30, 286–297. 2000/HFES 2000 Congress (pp. 2-346–2-349). Santa Monica, Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2008). CA: Human Factors and Ergonomics Society. Situation awareness, mental workload, and trust in automation: Metzger, U., & Parasuraman, R. (2001). The role of the air traffic Viable, empirically supported cognitive engineering con- controller in future air traffic management: An empirical study structs. Journal of Cognitive Engineering and Decision of active control versus passive monitoring. Human Factors, Making, 2, 141–161. 43, 519–528. Parasuraman, R., & Wickens, C. D. (2008). Humans: Still vital after Metzger, U., & Parasuraman, R. (2005). Automation in future air traf- all these years of automation. Human Factors, 50, 511–520. fic management: Effects of decision aid reliability on controller Parasuraman, S., Singh, I. L., Molloy, R., & Parasuraman, R. performance and mental workload. Human Factors, 47, 35–49. (1992). Automation-related complacency: A source of vulner- Molloy, R., & Parasuraman, R. (1996). Monitoring an automated ability in contemporary organizations. In R. Aiken (Ed.), system for a single failure: Vigilance and task complexity Education and society: Information processing 92 (Vol. 2, effects. Human Factors, 38, 311–322. pp. 426–432). Amsterdam, Netherlands: Elsevier Science. Moray, N. (1984). Attention to dynamic visual displays in man- Prinzel, L. J. (2002). The relationship of self-efficacy and compla- machine systems. In R. Parasuraman & D. R. Davies (Eds.), cency in pilot-automation interaction (Technical Memorandum Varieties of attention (pp. 485–513). San Diego, CA: Academic No. TM-2002-211925). Hampton, VA: National Aeronautics Press. and Space Administration Langley Research Center. Moray, N. (2003). Monitoring, complacency, scepticism and eutac- Prinzel, L. J., De Vries, H., Freeman, F. G., & Mikulka, P. (2001). tic behaviour. International Journal of Industrial Ergonomics, Examination of automation-induced complacency and indi- 31, 175–178. vidual difference variates (Technical Memorandum No. Moray, N., & Inagaki, T. (2000). Attention and complacency. TM-2001-211413). Hampton, VA: National Aeronautics and Theoretical Issues in Ergonomics Science, 1, 354–365. Space Administration Langley Research Center. Automation Complacency and Bias 409

Rasmussen, J. (1986). Information processing and human-com- Szalma, J. (2009). Incorporating human variation into human factors/ puter interaction. Amsterdam, Netherlands: North Holland. ergonomics research and practice [Special issue]. Theoretical Reichenbach, J., Onnasch, L., & Manzey, D. (in press). Misuse of Issues in Ergonomics, 10, 377–488. automation: The impact of system experience on complacency Taylor, P. M., Champness, J., Given-Wilson, R. M., Potts, H. W. E., and automation bias in interaction with automated aids. In & Johnston, K. (2004). An evaluation of the impact of Proceedings of the 54th Meeting of the Human Factors and computer-based prompts on screen readers’ interpretation of Ergonomics Society. Santa Monica, CA: Human Factors and mammograms. British Journal of Radiology, 77, 21–27. Ergonomics Society. Thackray, R. I. (1981). The stress of boredom and monotony: A Rovira, E., McGarry, K., & Parasuraman, R. (2007). Effects of consideration of the evidence. Psychosomatic Medicine, 43, imperfect automation on decision making in a simulated com- 165–176. mand and control task. Human Factors, 49, 76–87. Thackray, R. I., & Touchstone, R. M. (1989). Detection efficiency Sarter, N. B., & Schroeder, B. (2001). Supporting decision making on an air traffic control monitoring task with and without com- and action selection under time pressure and uncertainty: The puter aiding. Aviation, Space, and Environmental Medicine, case of in-flight icing. Human Factors, 43, 573–583. 60, 744–748. Senders, J. W. (1964). The human operator as a monitor and con- Thomas, L., & Wickens, C. D. (2006). Effects of battlefield display trollers of multidegree of freedom systems. IEEE Transactions frames of reference on navigation tasks, spatial judgments, and on Human Factors in Electronics, HFE-5, 1–6. change detection. Ergonomics, 49, 1154–1173. Shepherd, M., Findlay, J. M., & Hockey, R. J. (1986). The relation- Tsai, T. L., Fridsma, D. B., & Gatti, G. (2003). Computer decision ship between eye movements and spatial attention. Quarterly support as a source of interpretation error: the case of electro- Journal of Experimental Psychology A, 38, 475–491. cardiograms. Journal of the American Medical Informatics Sheridan, T. B. (1970). On how often the supervisor should sample. Association, 10, 478–483. IEEE Transactions on Systems Science and Cybernetics, SSC- Tversky, A., & Kahneman, D. (1974). Judgement under uncer- 6(2), 140–145. tainty: Heuristics and biases. Science, 185, 1124–1131. Sheridan, T. B. (2002). Humans and automation: Systems design Vincenzi, D. A., Muldoon, R., Mouloua, M., Parasuraman, R., & and research issues. Santa Monica, CA: Human Factors and Molloy, R. (1996). Effects of aging and workload on monitor- Ergonomics Society. ing of automation failures. In Proceedings of the 40th Annual Sheridan, T. B., & Verplank, W. L. (1978). Human and computer Meeting of the Human Factors and Ergonomics Society control of undersea teleoperators (Technical Report, Man- (pp. 1556–1561). Santa Monica, CA: Human Factors and Machine Systems Laboratory, Department of Mechanical Ergonomics Society. Engineering). Cambridge, MA: MIT. Westbrook, J. I., Coiera, W. E., & Gosling, A. S. (2005). Do online Simons, D. J., & Rensink, R. A. (2005). Change blindness: Past, information retrieval systems help experienced clinicians present, and future. Trends in Cognitive Sciences, 9, 16–20. answer clinical questions? Journal of the American Medical Singh, I. L., Molloy, R., Mouloua, M., Deaton, J., & Parasuraman, R. Informatics Association, 12, 315–321. (1998). Cognitive ergonomics of cockpit automation. In Wickens, C. D., & Dixon, S. (2007). The benefits of imperfect I. L. Singh & R. Parasuraman (Eds.), Human cognition: A diagnostic automation: A synthesis of the literature. Theoretical multidisciplinary perspective (pp. 242–253). New Delhi, Issues in Ergonomics Science, 8, 201–212. India: Sage. Wickens, C. D., Dixon, S., Goh, J., & Hammer, B. (2005). Pilot Singh, I. L., Molloy, R., & Parasuraman, R. (1993a). Automation- dependence on imperfect diagnostic automation in simulated induced “complacency”: Development of the complacency- UAV flights: An attentional visual scanning analysis. In potential rating scale. International Journal of Aviation Proceedings of the 13th International Symposium on Aviation Psychology, 3, 111–121. Psychology (pp. 919–923). Columbus, OH: Association of Singh, I. L., Molloy, R., & Parasuraman, R. (1993b). Individual Aviation Psychology. differences in monitoring failures of automation. Journal of Wickens, C. D., Gempler, K., & Morphew, M. E. (2000). Workload General Psychology, 120, 357–373. and reliability of traffic displays in aircraft traffic avoidance. Singh, I. L., Molloy, R., & Parasuraman, R. (1997). Automation- Transportation Human Factors Journal, 2, 99–126. related monitoring inefficiency: The role of display location. Wickens, C. D., & Hollands, J. G. (2000). Engineering psychology International Journal of Human-Computer Studies, 46, 17–30. and human performance. New York, NY: Prentice Hall. Singh, I. L., Sharma, H. O., & Parasuraman, R. (2001). Effects of Wickens, C. D., McCarley, J. S., Alexander, A., Thomas, L., manual training and automation reliability on automation Ambinder, M., & Zheng, S. (2007). Attention-situation aware- induced complacency in a flight simulation task. Psychological ness (A-SA) model of pilot error. In D. Foyle & B. Hoeey Studies, 46, 21–27. (Eds.), Pilot performance models (pp. 213–240). Mahwah, NJ: Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automa- Erlbaum. tion bias decision-making? International Journal of Human- Wiegmann, D. A. (2002). Agreeing with automated diagnostic aids: A Computer Studies, 51, 991–1006. study of users’ concurrence strategies. Human Factors, 44, 44–50. Skitka, L. J., Mosier, K. L., & Burdick, M. (2000). Accountability Wiener, E. L. (1981). Complacency: Is the term useful for air safety? and automation bias. International Journal of Human-Computer In Proceedings of the 26th Corporate Aviation Safety Seminar (pp. Studies, 52, 701–717. 116–125). Denver, CO: Flight Safety Foundation Skitka, L. J., Mosier, K. L., Burdick, M., & Rosenblatt, B. (2000). Wiener, E. L., & Curry, R. E. (1980). Flight-deck automation: Automation bias and errors: Are crews better than individuals? Promises and problems. Ergonomics, 23, 995–1011. International Journal of Aviation Psychology, 10, 85–97. Woods, D. D. (1996). Decomposing automation: Apparent simplic- St. John, M., Smallman, H. S., Manes, D. I., Feher, B. A., & ity, real complexity. In R. Parasuraman & M. Mouloua (Eds.), Morrison, J. G. (2005). Heuristic automation for decluttering Automation and human performance (pp. 3–18). Mahwah, NJ: tactical displays. Human Factors, 47, 509–525. Erlbaum. 410 June 2010 - Human Factors

Raja Parasuraman is a university professor of psy- Technology, Germany. He received his PhD in ex- chology at George Mason University. He obtained perimental psychology at the University of Kiel, his PhD in applied psychology from Aston University Germany, in 1988 and his habilitation in psychology in 1976. at the University of Marburg, Germany, in 1999. Dietrich H. Manzey is a professor of work, engineer- Date received: December 1, 2009 ing, and organizational psychology at the Institute of Psychology and Ergonomics, Berlin Institute of Date accepted: May 21, 2010