International Journal of Industrial Ergonomics 26 (2000) 301}325

Situation awareness in aircraft maintenance teams Mica R. Endsley! *, Michelle M. Robertson"

!SA Technologies, 4731 East Forest Peak, Marietta, GA 30066, USA "Liberty Mutual Research Center, Hopkinton, MA 01748, USA Accepted 15 June 1998

Abstract

Research was conducted at a major airline to investigate factors related to situation awareness in aviation maintenance teams. Situation awareness has been found to be critical to performance and error prevention in many environments. Its role in the maintenance domain for the performance of both individuals and teams is discussed. Situation awareness requirements for aviation maintenance were determined as well as the technologies and personnel resources used to achieve situation awareness. Barriers and problems for situation awareness both across and within teams involved in aviation maintenance were revealed. Based on this analysis, recommendations for the development of a training program to improve situation awareness in aircraft maintenance at the individual and team level are presented. Relevance to industry The importance of situation awareness for preventing errors in maintenance is discussed as well as factors that contribute to problems with situation awareness across multiple teams. Speci"c recommendations for improving situation awareness through organization and system design and through training are made that are applicable to a wide variety of industrial settings. ( 2000 Elsevier Science B.V. All rights reserved.

Keywords: Situation awareness; Teams; Maintenance; Training; Error

1. Introduction (1994), for instance, report that 12% of accidents are due to maintenance and inspection faults, and Insu$cient attention has been paid to human around one-third of all malfunctions can be at- error in aircraft maintenance. While the number of tributed to maintenance de"ciencies. In addition to incidents due to mechanical failures that can be its impact on safety of #ight, the e$ciency of main- traced to maintenance problems are relatively tenance activities can also be linked to #ight delays, few when compared to other causal factors (e.g. ground damage and other factors that directly in#ight human error), they do exist and can be impact airline costs and business viability. systematically addressed. Marx and Graeber In examining human error that may occur within the maintenance arena, several key issues can be identi"ed. * Corresponding author. Tel.: #1-770-565-9859; fax: #1- 770-579-1132. (1) The "rst involves shortcomings in the detec- E-mail address: [email protected] (M.R. Endsley). tion of critical cues regarding the state of the

0169-8141/00/$ - see front matter ( 2000 Elsevier Science B.V. All rights reserved. PII: S 0 1 6 9 - 8 1 4 1 ( 9 9 ) 0 0 0 7 3 - 6 302 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325

aircraft or subsystem. Several accidents have a maintenance error in servicing the engines in been traced to metal fatigue or loose and miss- which critical o-rings were left o! (National ing bolts that should have been visible to main- Transportation Safety Board, 1984). This error tenance crews. Incidents exist of aircraft being has been directly linked to a problem with returned to service with missing parts or in- coordination of information across shifts and complete repairs. Frequent errors include loose between maintenance departments (along with objects left in aircraft, fuel and oil caps missing other contributing factors). In addition, con- or loose, panels and other parts not secured, siderable energy is often directed at coordina- and pins not removed (Marx and Graeber, tion across maintenance sites to accommodate 1994). While several factors may contribute to maintenance tasks within the #ight schedule this type of error, in all of these cases the state and part availability constraints. These factors of the system (i.e. the defect, or the loose or add a level of complexity to the problem that missing item) was not detected prior to return- increases the probability of tasks not being ing the aircraft to service. completed, tasks completed improperly, im- (2) Often, even when important information is portant information not being communicated, perceived, there may be di$culties in properly and problems going undetected as responsibil- interpreting the meaning or signi"cance of that ity for tasks becomes di!used. information. For instance, Ru!ner (1990) found that in more than 60% of avionics re- 1.1. Situation awareness pairs, the incorrect avionics system is replaced in an aircraft. While the symptoms may be All of these di$culties point to a lack of situation observed correctly, a signi"cant task remains awareness (SA). Situation awareness has been in properly diagnosing the true cause of the found to be important in a wide variety of systems failure. While not much data exists regarding operations, including piloting, air tra$c control the impact of misdiagnoses of this type, there is and maintenance operations. Maintenance crews a signi"cant increase in the probability of an need support and training in ascertaining the cur- incident occurring when the aircraft under- rent state of the aircraft system in addition to takes the next #ight with the faulty system still current training programs that concentrate on aboard. technical skills. Formally de"ned, `situation aware- (3) Problems in properly detecting the state of the ness is the detection of the elements in the environ- system and diagnosing or interpreting cues ment within a volume of space and time, the that are perceived are compounded by the fact comprehension of their meaning, and the projec- that many di!erent individuals may be in- tion of their status in the near futurea (Endsley, volved in working on the same aircraft. In this 1988). In the context of aircraft maintenance, this situation, it is very easy for information and means being aware of the state of the aircraft sys- tasks to fall through the cracks. The presence of tem (and the subsystem one is working on). Termed multiple individuals heightens the need for Level 1 SA, this would include perception of factors a clear understanding of responsibilities and such as metal fatigue, loose or missing items, pins, good communications between individuals to or screws, oil or #uid leaks, tread wear, or systems support the performance of shared tasks. not functioning properly. Level 2 SA would involve (4) In addition to the need for intra-team coord- the technicians' understanding or comprehension ination, a signi"cant task for maintenance of the signi"cance of observed system states. Spe- crews is the coordination of activities and pro- ci"cally this would include their diagnosis of the vision of information across teams to those on causal factors associated with observed symptoms. di!erent shifts or in di!erent geographical lo- An aviation maintenance technician (AMT) cations. For example, an Eastern Airlines air- with Level 1 SA might be aware that a particular craft nearly crashed when oil pressure was lost subsystem is not working properly. An AMT with to all engines almost simultaneously due to Level 2 SA also understands what is speci"cally M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 303 wrong with that subsystem. Level 3 SA, the ability are shared goals, the interdependence of their ac- to project the state of the system in the near future, tions, and the division of labor in terms of estab- is considered the highest level of SA in dynamic lished responsibilities for meeting those goals. systems. An AMT with Level 3 SA would be able to Based on this framework, teams will have some project what e!ect a particular defect might have division of labor, and thus some of their SA require- on the performance of the aircraft in the future. ments will be independent. However, they will also While SA has generally been discussed in terms have shared goals and their actions will be interde- of the operation of a dynamic system, such as an pendent, and thus they will also have shared SA aircraft, the concept is also applicable to the main- requirements. tenance domain. The complexity of aircraft systems Team situation awareness can be de"ned as `the and the distributed nature of equipment and system degree to which every team member possesses the components posses a signi"cant challenge to the situation awareness required for his or her respon- AMT's ability to determine the state of the system sibilitiesa (Endsley, 1989). In this context, the weak (Level 1 SA) during diagnosis and repair activities. link in the chain occurs when the person who needs Putting together observed cues to form a proper a given piece of information (per his or her job understanding of the underlying nature of malfunc- requirements) does not have it. Therefore, the con- tions (Level 2 SA) is a signi"cant problem in diag- cept of team SA embodies the need for each person nostic activities. In the maintenance domain, on a team to have the information needed for AMTs may need to be able to project what will his/her speci"c job. As shown in Fig. 1, if one happen to an aircraft's performance with (or with- considers the SA requirements of di!erent team out) certain actions being taken or with given members, there will be some area of overlap be- equipment modi"cations/repairs/adjustments oc- tween them. That is, there will be some SA require- curring (Level 3 SA). This task may be even more ments (at each of the three levels) that are pertinent di$cult for AMTs. As they often receive little or no only to a given individual, but also some SA re- feedback on the e!ects of their actions, they may quirements that are the same across one or more have di$culty developing an adequate mental team members. Successful team performance re- model for making accurate predictions. The ability quires that not only does each team member have to project system status forward (to determine pos- good SA on his or her individual requirements, but sible future occurrences) also may be highly related to the ability to project system status backward, to determine what events may have led to an observed system state. This ability is particularly critical to e!ective diagnostic behavior.

1.2. Team situation awareness

In aircraft maintenance, as in many other do- mains, the requirement for situation awareness be- comes compounded by the presence of multiple team members and multiple teams. Teams di!er from a collection of individuals in that they are a distinguishable set of two or more people who interact dynamically, interdependently, and adap- tively toward a common and valued goal/object/ mission, who have each been assigned speci"c roles or functions to perform, and who have a limited life span of membershipa (Salas et al., 1992). Thus, the major factors contributing to the concept of a team Fig. 1. Team situation awareness (from Endsley, 1989a). 304 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 also the same SA across shared requirements. This factors need to be considered to optimize team SA. has been termed `shared SAa. For example, skills are needed for identifying criti- In maintenance teams, as in teams in many other cal information and ensuring that it is passed domains, errors frequently occur because informa- across teams (and team members) and then inter- tion is not successfully transmitted from one indi- preted based on a common framework across team vidual to the next within a team, or from one team members. The present research therefore sought to to the next. The failures may occur at Level 1 (in- identify SA requirements for various aircraft main- formation not transmitted successfully to a team or tenance teams, analyze how SA needs are currently team member), Level 2 (information not compre- being met in a typical maintenance environment, hended properly or in the same way across teams and establish concepts and requirements for train- or team members), or Level 3 (the implications of ing to improve SA in maintenance teams. transmitted information for future events not Why examine the maintenance domain from the understood or the same across teams or team mem- perspective of SA? First, based on a naturalistic bers). The ways in which these failures might occur model of decision making, we "nd that peoples' and the factors that contribute to such failures have classi"cation of the situation largely drives their not previously been examined from the perspective decision making and performance (Endsley, 1995b; of situation awareness in the maintenance arena. Klein, 1989). Studies of human error in the cockpit The objective of the present study was to address have found that the vast majority of errors are due situation awareness related di$culties in aircraft not to poor decision making, but to problems with maintenance. This information would be useful for SA (Endsley, 1995a; Hartel et al., 1991). The present supporting the development of cohesive mainten- study speci"cally sought to determine whether SA ance teams and the promotion of team situation was relevant in the maintenance domain and the awareness. This is critical for the development of an way in which it serves decision making and perfor- environment in which people have a shared under- mance in this, a very di!erent, environment. standing of who does what when, reducing the Secondly, examining SA may provide a unique probability of information or tasks going unat- perspective on the challenges and problems within tended. Individuals need to not only understand the maintenance arena. The reasons for many er- the status of the system they are working on, but rors and shortcomings in aircraft maintenance may also what other individuals or teams are (and are not be apparent at "rst glance. By looking at SA, not) doing as well, as both factors contribute to and the many challenges to SA in this domain, a their ultimate decision making and performance. better understanding of the causes of human error Within a team it is important to provide the may be achieved. It is important to note that while members with the skills to function e!ectively. Spe- many problems may occur due to challenges in ci"cally, training can be provided to assist teams in information acquisition (e.g. noise, poor lighting, achieving situation awareness as this is the critical out-dated technology), the way in which informa- factor that will allow team members to carry out tion is used and understood by individuals within the maintenance tasks they have been trained for. the maintenance organization, across its various Several related training programs have been suc- teams, shifts and locales, may also underlie many cessful within this domain. Drury (1993) has shown problems in the system. success in training maintenance personnel in visual inspection. Taylor et al. (1993) were successful in improving high-level performance objectives (e.g. 2. Methodology dependability and safety) after instituting a pro- gram to train aircraft maintenance personnel in Two major research activities were conducted positive, assertive communication and teamwork towards the accomplishment of the research coordination skills. While both of these training goal: (1) A determination of the requirements and examples may help SA, neither directly addresses resources for SA in aircraft maintenance teams, and SA as an over-riding objective. In particular, other (2) an assessment of training needs for improving M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 305 team SA. The objective of the requirements analysis was to determine exactly what SA means within the maintenance environment: What is it that needs to be perceived by each person in his or her job, what is it that needs to be comprehended and what projections need to be made? Based upon an under- standing of the actual requirements for SA in vari- ous parts of the organization, recommendations can be made for improving the technologies or organizational factors available for supporting SA and for creating a training program to enhance SA in aviation maintenance teams. While SA is critical for decision making and performance in many areas, developing appropriate methods for improv- Fig. 2. Team SA context analysis. ing SA in any given domain is contingent on such an analysis. Since it was not possible to review practices at for training team SA were developed based on the all airlines or all locations, this research was con- results of the analysis. A team SA context analysis ducted at an aircraft maintenance facility for one methodology was developed for conducting the major US airline which served as a representative analysis. This method consists of two parts: An SA maintenance environment. Speci"cally, B-check requirements analysis and an SA resource analysis, maintenance operations at a major airport were as shown in Fig. 2. analyzed. The B-check is conducted on a periodic basis for all aircraft and involves thorough checks 2.1. SA requirements analysis of avionics systems and power units, various interior and exterior components, and lubrication The "rst step was to determine the speci"c situ- and checks of engines, wings and gears. ation awareness requirements of individuals in the The subject airline is a large US air carrier. It was aircraft maintenance arena. This was addressed believed that the selected airport B-check operation through a goal-directed task analysis which as- was fairly representative of the type of activities sessed (1) the goals and sub-goals associated with performed in aircraft maintenance, although obvi- maintenance crews, (2) the decision requirements ously variations may exist due to di!erences be- associated with these goals, and (3) the situation tween airlines or management practices at various awareness requirements necessary for addressing maintenance stations. This airline, like many in the the decisions at all three levels } detection, compre- industry, had undergone a number of reorganiza- hension, and projection. This type of analysis has tions in the previous "ve years which involved been successfully conducted for pilots in several consolidating maintenance bases and out-sourcing classes of aircraft (Endsley, 1989,1993), air tra$c some of its maintenance functions. While a signi"- control (Endsley and Jones, 1995; Endsley and cant impact to personnel within the organization Rodgers, 1994) and airway facilities maintenance and to the way of doing business, it was not felt that (Endsley, 1994). these factors would signi"cantly a!ect the major Analyses were conducted through expert elici- goals of the study } determining the applicability of tation with experienced maintenance personnel, SA and team SA to aircraft maintenance, and de- observation of aircraft maintenance activities, and signing intervention strategies for improving it. review of all available maintenance documentation. The research was conducted by "rst identifying The analysis concentrated on B-Check mainten- SA requirements and the resources used to support ance activities. Interviews were conducted with those requirements in the selected representative three maintenance supervisors, four lead techni- maintenance environment and operations. Concepts cians and four AMTs. In addition, personnel in 306 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 planning and stores, maintenance control, mainten- 3. Results ance operations control and aircraft-on-ground in the company's technical operations were interviewed. A hierarchy of goals in the maintenance environ- ment was developed for several categories of job 2.2. SA resource analysis function within the maintenance team (supervisors, leads and AMTs), and for several organizations The second part of the team SA context analysis or teams that work closely with the maintenance concentrated on identifying the SA resources used team to achieve its goals: Material Services (stores), in the maintenance environment to achieve the SA Planners, and Maintenance Control, including requirements identi"ed in the goal-directed task Maintenance Operations Control and Aircraft-on- analysis. Two major categories of resources were Ground (an organization that deals with acceler- considered: other personnel as a source of informa- ated maintenance for aircraft that do not meet tion and technologies used as sources of informa- airworthiness requirements). These goals were used tion. to develop a list of SA requirements for each group. To provide an assessment of the personnel SA Next, the personnel resources used for meeting resources, an analysis of communications between these SA requirements were determined. The organizations and individuals was conducted using technology resources used for meeting the SA a contextual inquiry approach. The contextual requirements were also surveyed and barriers and inquiry approach (O'Neill and Robertson, 1996; problems for achieving job goals were identi"ed Robertson and O'Neill, 1994) focuses on under- during the interviews. As these analyses were quite standing and describing the communication pat- extensive, the results for only the AMT are shown terns within and between teams as related to their here as an example. The analysis of SA issues across performance and job goals. The contextual in- maintenance teams will also be discussed. quiries were conducted simultaneously with the interviews for determining the SA requirements. 3.1. Goals and functions The contextual inquires involved semi-structured interviews in which each individual was asked to Speci"c tasks and functions included within the describe his/her major functions and goals and the B-check operations were identi"ed to include: organizations or individuals that served as re- (1) avionics in cockpit: radar, radios, #aps, gauges, sources in meeting those goals. A mapping was (2) interior: lavatories, emergency equipment, sea- determined showing the communications and in- ts, overhead bins, lap belts, emergency lighting teractions among and between team members. (3) exterior: tires, brakes, fuselage, leading edges Each individual was asked to make an estimate of and #aps, cargo bays, (4) right and left engines and the overall frequency of communication with each wings, (5) right/left gear lubrication, nose gear lu- identi"ed unit and the importance of the commun- brication and tail lubrication, (6) auxiliary power ication for achieving their goals. Finally, they were unit, and (7) addressing placards (notices placed in asked to identify barriers to e!ective communica- maintenance logs for non-functioning items, allow- tion and performance in the work setting. ing the plane to be #own for a speci"ed number of In addition, the technologies for obtaining each hours using redundant systems while repairs are requirement within the current system were doc- completed). umented. Based on this analysis, an assessment was An analysis of the AMTs' goals in performing made of the degree to which the current system sup- their tasks revealed that they appear to be oriented ports SA in maintenance teams and the skills and towards the dual objectives of ensuring aircraft abilities that are required for achieving good SA safety and delivering aircraft for service on time. within this environment. This assessment was used A breakout of AMT goals is shown in Table 1. In to identify system design recommendations and general, the top level goals of supervisors and leads training concepts for improving SA in maintenance are very similar to those of AMTs shown in Table 1. teams. They assume many of the same subgoals as the M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 307

Table 1 AMT goals

1.0 Aircraft safety 1.1 Deliver aircraft in airworthy, safe condition 1.1.1 Find potential problems 1.1.2 Solve problems 1.1.3 Make repairs 1.1.3.1 Determine part availability 1.1.3.2 Placard problem 1.1.4 Service aircraft 1.1.5 Provide quality workmanship 1.2 Keep area clean 2.0 Deliver aircraft on time 2.1 Prioritize tasks

AMTs in identifying and solving maintenance that are important for meeting goals in this do- problems as needed to support AMTs when they main. In reviewing Table 2, a few issues should be run into di$culties. Supervisors and leads, how- noted. ever, also have additional subgoals associated with (1) At any given time more than one goal or sub- managing the maintenance teams (assigning and goal may be operating, although these goals prioritizing tasks, assessing aircraft status and pro- will not always have the same priority. The SA viding coordination). requirements in Table 2 do not assume any Reviewing the goals of each of the other organ- prioritization among them, or that each sub- izations involved in aviation maintenance revealed goal within a goal will always be present. signi"cant interdependencies between teams. For (2) These are goals or objectives not tasks. The instance, AMTs are dependent on material services analysis is as technology free as possible. How to have parts, tools, and materials ready when the information is acquired is not addressed in needed. Material services is in turn dependent on the SA requirements analysis. (This matter is the planners to provide relevant task information addressed in the SA resource analysis). and on the aircraft-on-ground organization to de- (3) The analysis sought to de"ne what AMTs liver needed parts as soon as possible. These inter- would ideally like to know to meet each goal. It dependencies, while not surprising, highlight the is recognized that they often must operate on need for good information transfer across teams. the basis of incomplete information and that By examining in detail the SA requirements of each some desired information may not be easily team, it should be possible to gain an understand- available with today's system. ing of the types of information required by each (4) Static knowledge, such as procedures or rules team and the ways in which the transfer of this for performing tasks, is also outside the bounds information can be improved. of this analysis. The analysis primarily identi"- es dynamic situational information (informa- 3.2. SA requirements tion that changes from situation to situation) that e!ects how AMTs perform their tasks. A breakout of SA requirements for each goal and subgoal identi"ed for each maintenance team was A similar analysis was conducted for each of the derived from the interviews and observations. The other maintenance teams (e.g. stores, inspectors, goals, subgoals, major decisions and SA require- etc.) This analysis shows the factors that are impor- ments for AMTs are shown in Table 2. The analysis tant for SA in maintenance crews, not only in terms identi"ed a number of SA requirements at all three of the basic information that is needed, but also levels (perception, comprehension and projection) how that information is used, interpreted and 308 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325

Table 2 AMT SA requirements

1.0 Aircraft safety 1.1 Deliver aircraft in airworthy, safe condition 1.1.1 Find potential problems E Item within or beyond serviceable limits? E Item near limits needing preventive maintenance? E reported problems E pilot reports E placards E new problems E worn tires/brakes E miswiring E dents/damage E loose items E fuel/oil leaks E items out of ordinary E functioning of convenience items 1.1.2 Solve problems E Fix problem or defer? E potential impact of problem on #ight safety E time required to solve problem E time required to get part E length of time item can be deferred without repair E location(s) aircraft is going to E facility maintenance capabilities E today's load E problem deferability category (placardable, groundable) E minimum equipment list (MEL) status E How to solve problem? E impact of potential approaches on time E impact of potential approaches on #ight safety E impact of potential approaches on other tasks/jobs E possible methods E possible sources of problem E maintenance/failure history of item E part availability (see 1.1.3.1) E proposed repair authorized E Engineering Change Request Authorization number 1.1.3 Make repairs 1.1.3.1 Determine part availability E Correct part supplied? E manufacturer's part number E aircraft type, model, tail number E maintenance and equipment list number E e!ectivity number E How long to get part here? E in-stock status E manfacturer's part number E aircraft type, model, tail number E maintenance and equipment list number E e!ectivity number M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 309

Table 2 (Contd.)

E part & tooling availability E where E when it will be here E delivered or pick-up E arrival #ight number E arrival gate number 1.1.3.2 Placard problem E Can problem be placarded? E type of problem E Minimum Equipment List (MEL) status E Deferred information placard (DIP) E Open item list (OIL) E redundant systems available E control number E log page number E #ight number E employee number 1.1.4 Service aircraft E Service activities needed? E tasks to be done E fuel status E lavatory status E Are we meeting schedule? E time aircraft due at gate E delays to aircraft E estimated time of arrival at gate E aircraft repair status E Where do we need to go? E gate assignments E permission to taxi E permission to do high power run-up E taxi/runway clearances E Current status of job? E status of other tasks impacting own task E other tasks own task will impact E who can help E who needs help E tasks started E tasks completed E tasks/activities being done next E who is doing each task E activity currently being performed by others E major problems encountered 1.1.5 Provide quality workmanship E Activities performed correctly? E tasks performed correctly E steps to be done E steps completed E location of designated components on system E system type E paperwork completed E parts installed correctly E inspection approved 310 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325

Table 2 (Contd.)

1.2 Keep area clean E Area free of foreign objects? E loose objects (screws, parts, 2), tools, trash 2.0 Deliver aircraft on time 2.1 Prioritize tasks E Best order for tasks? E task time requirements E interdependence/sequencing requirements of tasks E part availability (see 1.1.3.1) E problem deferability category (placardable, groundable) E minimum equipment list (MEL) status E availability of kits, tools, equipment, vehicles E availability of personnel, skill level

Table 3 or units within maintenance technical operations, AMT personnel SA resources that are needed to achieve the AMTs' SA require- SA resources: personnel Mean Mean ments and job goals. Lines and arcs show commun- importance frequency (%) ication patterns between organizations. Estimates were made by each individual who was Supervisor 3.2 2.25 interviewed concerning the overall frequency of Maintenance control 2.5 3.75 communication with each of the other maintenance }Maintenance operations Control }Aircraft on ground (AOG) organizations and other team personnel. Each indi- Quality assurance 2.5 (1.00 vidual assigned a number to each organization Aircraft inspectors 2.0 3.25 which re#ected the overall frequency of commun- Planning 2.0 (1.00 ication of each interchange (out of a total of 100%). Lead technician 1.5 26.75 For each of these SA resources, the importance of Stores 1.5 8.25 Other AMTs 1.0 54.50 the communication also was rated on a four-point Airport operations 1.0 (1.00 scale, where 1 represents a very important resource Company operations 1.0 (1.00 for achieving the team's performance goals and 4 represents a relatively low importance resource. Mean estimates of communication frequency and importance were determined for each of the main- analyzed to make decisions and carry out required tenance organizations. tasks. Any attempts to develop technologies or Table 3 displays the personnel SA resources, other changes to support the maintenance techni- mean importance and mean frequencies for the cian must directly impact on the accuracy or ease of communication interfaces from the perspective of achieving each of these requirements. the AMT. Several personnel and work units were indicated as very important SA resources necessary 3.3. SA resources: personnel to accomplish the AMT's job. These were the other AMTs, airport operations (tower) and company The personnel resources and technology re- operations (ramp personnel), closely followed by sources used within the organization to meet the leads and stores (material services). The highest SA requirements identi"ed in Table 2 were ascer- reported frequency of communication was with the tained through the contextual inquiry methodo- other AMTs (54.50%), followed by lead technicians logy. Results of the contextual inquiries for the (26.75%). AMT are presented in Fig. 3. The "gure depicts the Overall, a great deal of interdependency was personnel SA resources, in terms of the individuals found between the organizations and personnel M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 311

Fig. 3. AMT personnel SA resources.

included in this study. Each job type involved inter- tenance activities, several non-integrated databases acting with between 3 and 14 di!erent organiza- used across several organizations, and technical tions to attain (or supply) the information needed documentation which could be found in various for good SA and job performance. In general, two hard copy manuals and micro-"che. (Issues and or three of these interactions were viewed as very recommendations involving these technologies are important and constituted the majority of each addressed later.) organization or team's interactions, however, for many of the teams there were also many organiza- 3.5. Barriers and problems tions that were interacted with only occasionally. Speci"c issues regarding these interactions that can During the interview process, factors that created have a signi"cant impact on SA were uncovered barriers to e!ective communication and perfor- during the analysis and presented in the discussion mance were elicited. Barriers are issues that slow section. down or hinder performance. These are problems that maintenance teams must routinely overcome 3.4. SA resources: technologies in order to meet their goals. They encompass or- ganizational, technical and personnel issues. In In addition to ascertaining the personnel re- general, most people felt that the system worked sources used for attaining situation awareness, quite well, however, almost all could name a few the technologies used within the maintenance areas where improvement was possible. These are organization were also examined. The primary listed in Table 4. technologies used for passing information included The most frequently mentioned barrier was a commercial information system for logging main- a lack of proper tooling for completing jobs, a 312 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325

Table 4 track the quality and reliability of these parts which Barriers to performance the current system does not support. The second most commonly listed problem was Barrier Frequency an unavailability of parts and di$culty in deter- Lack of tooling; out-sourcing of parts 13 mining when the proper parts would be available to Parts availability; determining status of parts 10 the AMT. Often the parts supplied would not be No backlog of critical parts 10 correct for the speci"c model and type of aircraft. Non-integrated databases; redundancy of tasks; 10 This is a particular problem as the company has hard to "nd needed information Tracking the parts; getting the parts to the aircraft 8 aircraft which were purchased from many di!erent Computerized database 8 airlines, each with subtle di!erences between them. Lack of support and feedback from management 8 Parts supplied by stores often are not the correct Other organizations don't understand what we do, 8 ones due to these slight di!erences. This serves as problems we face a frequent sources of frustration, necessitating sche- Lack of teamwork; information being passed 7 among & between team members dule delays or issuing a placard for repair at a later Personality con#icts 7 date. Related to this problem is the lack of a back- Instability of organization 7 log of critical parts. Critical parts are frequently not Downsizing of organization 6 available when needed, leading to having an air- Shiftwork; fatigue 6 craft down for an extended period or necessitating Computer system in stores 5 Workcards; changing of procedures with aircraft 5 expensive and time consuming rush procurements Need for better information and communication 5 through the aircraft-on-ground organization. to solve problems locally Tracking parts for a speci"c aircraft was fre- Streamline engineering authorizations 4 quently mentioned as a signi"cant di$culty. Deter- Poor housekeeping and maintenance of tools 4 mining where parts are in the system (speci"cally in Computer system for customer service 3 Need more training on using computer system 3 relation to items being obtained from outside the Need for more proactive procedures 3 system, or in transit from somewhere) and getting Need to develop consistent procedures for 3 the parts to the aircraft were described as common obtaining/borrowing parts problems. In general, maintenance personnel ex- Need more explicit requirements for contract 3 perienced signi"cant uncertainty regarding when, suppliers Low experience levels of some personnel due 2 where and how parts would be delivered and spent to lay-o!s and job changes extra time trying to get this information and to ascertain its reliability. Signi"cant problems in switching between the various information databases (such as the stores database and the customer service database among problem which is exacerbated by the fact that much others) were noted. Maintenance personnel cur- of the repair work has been out-sourced to outside rently need to retrieve information from multiple contractors, making rapid access to correct tooling sources, however, the ability to readily access and and parts di$cult. In addition to a certain lack of gain needed information from multiple systems at trust of these external organizations (probably the same time is quite limited. For example, book- stemming from lack of information), there was an ing, monitoring bills, baggage handling, and track- expressed frustration with not being able to interact ing items and parts are all activities that need to be with personnel where the repair work had been conducted by aircraft-on-ground. These activities out-sourced so that questions pertaining to speci"c require accessing and integrating information items could be addressed. A decreased reliability of across several databases on an almost continual parts and quality control problems with parts basis in order to keep up-to-date with the current coming from out-sourced vendors was also men- situation. This situation also leads to redundant tioned. Maintenance personnel expressed the need tasks between paperwork, manuals and the com- to be able to track parts by vendor names and to puter systems. Maintenance personnel expressed M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 313 a need for an integrated computerized database organization was also a signi"cant concern. Just system, allowing for more e$cient monitoring of prior to the time period of this study, there were activities and parts thus facilitating getting the part many reorganizations, changes in management, to the aircraft in a more expeditious fashion. Other layo!s, and reassignments/relocation of many of needs expressed included a ready list of `hota parts the personnel. and items, a means of tracking minimum equip- Other problems mentioned included fatigue and ment list (MEL) items better and a database on problems associated with shift work (particularly parts reliability. The commercial computer in- among graveyard shift workers), concerns over or- formation database in particular was considered ganizational down sizing, lack of up-dating of the a signi"cant barrier or problem. Personnel ex- stores computer system to re#ect the nuances of pressed considerable frustration with the system as particular aircraft, a need for more training on the it made data entry very di$cult and the user inter- computer system, a lag in updating work cards to face was very clumsy. re#ect changes in work procedures, and poor Organizational issues were also mentioned. In housekeeping and maintenance of tools. particular, the feeling was expressed that manage- People expressed the desire to be able to solve ment was not providing visible and active support, problems locally if only they had the information particularly in regard to feedback on how person- they needed. For example, a particular problem nel were doing, improvements that could be made, may be placarded and passed on to another station, and guidance on which direction personnel should when it could be "xed locally if information on go in and why. Maintenance personnel expressed scheduling and parts availability was shared better. a desire for better feedback or rewards when they People at the local level wanted to be more in- make progress in the right direction. volved in the decision-making process in order to Maintenance personnel also expressed a certain help meet the organizational goal of having the degree of frustration regarding other organizations. aircraft back in service as soon as possible. Related Many felt that other groups did not really under- to this issue, personnel also expressed a desire for stand what they did. For example, the AMTs did more proactive problem solving instead of waiting not like having maintenance operations control until a crisis situation develops. They felt they (MOC) tell them what to do, when `they aren't out needed to get information sooner and to obtain here working in the cold and the darka. On the earlier involvement of the respective parties in the other hand, MOC personnel felt they were misun- problem solving process. derstood as they all had worked in the AMT job A need was also expressed for streamlining the before. They also felt they had the best information process used for obtaining engineering authoriz- to be able to ascertain the impact of a given prob- ations and for developing consistent procedures lem on changes in the system (e.g. scheduling). They that everyone could follow for borrowing and ob- did not feel the AMTs had this `big picturea. The taining parts. Due to a lack of consistent proced- end result of these types of di!erences is misunder- ures, a lot of time and e!ort may go into one standings between organizations, and ine$ciencies particular strategy for obtaining a part and then in problem solving as neither group has the full when that method falls through, they have to try picture and the same information possessed by the a new strategy having lost valuable time. Proced- other group. ures that incorporate alternate parallel tracks and Several interpersonal issues were mentioned. action plans when parts are needed are desired. While most personnel were considered to be `team Di$culty with contract suppliers was expressed. playersa, others were considered to be de"cient by The feeling was that contract suppliers need to be not pitching in to help complete tasks. Problems given clearer expectations regarding what they are with information not being transferred between to deliver and quality requirements. Clearer pro- team members both during a shift and between cedures and processes need to be conveyed to them, shifts were cited. Related to this, personnel con#icts particularly in light of signi"cant culture and time were listed as a problem. The instability of the zone di!erences. 314 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325

Finally, the low experience levels of some person- time consuming and may be prone to miscommuni- nel was described as a problem. Due to a number of cation errors, leading to SA problems. layo!s, people with high seniority, but perhaps low The process of placarding also poses a signi"cant experience level in a particular job type are more problem. AMTs may spend a considerable amount common. This has a signi"cant impact on schedul- of time disassembling a system and trouble-shoot- ing the AMTs on particular tasks and teams. With ing to arrive at a diagnosis, only to "nd they cannot the perceived pressure to save money and do more "x the problem due to an unavailability of parts or with less, this issue was felt to have an impact on schedule constraints. This is a process which is performance as more highly trained individuals fairly ine$cient and which they "nd very frustrat- might not be able to work with fellow team mem- ing due to lack of closure in addressing the very bers as much as might be needed. problems they are trained to "x. Completing re- pairs is a factor from which they derive their major job satisfaction. AMTs get very discouraged when 4. Discussion they are not allowed to "x things that clearly need "xing. It is also a waste of time and human re- Based on the analysis of the SA requirements and sources to have to reassemble a subsystem and resources for each organization and the barriers placard it so that it can be unassembled again and and problems expressed, several observations can "xed later on at another maintenance station. Al- be made pertinent to team SA in the aircraft main- though sometimes placarding is unavoidable, it is tenance domain. The largest problem for team SA generally best if problems can be "xed immediately. exists when gaps are present between organizations The system does not appear to be currently opti- or individuals. These gaps may be the result of mized to avoid placarding, however. A review of mismatched goals, lack of needed information on the goals of supporting organizations, such as the part of one or both parties, lack of understand- maintenance control and its sub-organizations, re- ing of the exact information that another group veals that they place far more emphasis on rem- needs, or di!erent interpretations of information edying existing placards than avoiding new ones. that is passed from another group. This goal mis-match may be at the heart of con- AMTs face several challenges in meeting their SA siderable misunderstandings between groups. requirements that can be linked to team SA. First, While AMTs report a need to ascertain job AMTs spend a great deal of their time and re- status and schedule progress, they currently get sources in ascertaining whether they have the cor- only limited information concerning these issues. rect parts or when and how they will get the correct While they supply information regarding progress parts. A considerable gap exists between the AMTs on their own tasks up the line on an ongoing basis, and the stores organization which often may supply leads and supervisors frequently provide little in- the incorrect part (due to di$culties with e!ectivity formation back down the line over the course of number di!erences between di!erent aircraft mod- a shift. Leads reportedly did not feel that AMTs els, for instance) or may not have the correct part really needed information on how other team mem- due to stocking limitations. These situations in- bers were doing in terms of progress on their re- crease both the probability of error (incorrectly spective tasks. Without this knowledge, however, installing the wrong but very similar part) and may the AMTs have no way to engage in compensatory lead to considerable ine$ciencies, waste and de- activities (e.g. pitching in to help each other), and lays. When parts are not available, the AMTs fre- may not be aware of ongoing activities of other quently must involve their leads and supervisors, team members that may have an e!ect on their own maintenance control and aircraft-on-ground. This tasks (or vice versa). In some cases, tasks must be necessitates the involvement of several organiza- done in a certain order. In other cases, certain tasks tions and personnel, all of whom need to be can a!ect the activities of other AMTs in a way that brought up to speed on pertinent situational in- creates a safety hazard unless both parties take formation to make good decisions. This process is precautionary measures. Thus, a lack of up-to-date M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 315 knowledge on within team progress contributes to situation. If supervisors do not have a complete SA gaps within the maintenance team. understanding of all pertinent information, for in- Although regulations are very speci"c regarding stance, they may not pass information on that will the criteria specifying when an item must be re- allow maintenance control to make the best deci- paired or replaced, discretion is available in allow- sion. In addition, leads and supervisors are fre- ing AMTs to repair or replace items that might be quently responsible for passing information back to nearing the acceptable limit. It may be both safer the AMT. If they only pass information regarding and more time and cost e!ective to promote this what the AMT should do (the decision) but not type of action in some circumstances (e.g. if a given regarding why the decision was made, this may lead subsystem is already disassembled for other work to both a lack of understanding by the AMT and and the part is available). Discretion is also avail- may deny the AMT the opportunity to volunteer able on items that are placardable: they can be "xed information he or she may have that would be immediately or placarded and sent to another sta- pertinent to the decision made. Leads and supervi- tion for later repair. Better sharing of information is sors form a critical link in the SA chain between the needed in regard to these issues so that AMTs, various organizations and need to have a full un- leads and supervisors can make decisions that are derstanding of what information other people in line with de"ned organizational priorities and really need and of how to get all the information realities. For example, they may need the informa- they need themselves. tion that a given aircraft will only be going to other Stores (material services) appears to work prim- stations that are not well equipped to "x the par- arily based on planned demands in order to obtain ticular type of problem or that are overloaded. This needed parts in advance. Some stores personnel, would indicate that they should "x the problem however, reportedly do not understand the unique instead of placarding it, even it means a slight di!erences between aircraft models and tail num- schedule delay. Personnel need to understand what bers that allow them to procure parts with the current organizational priorities are and why. proper e!ectivity number. (This problem appears A shared understanding of the cost and bene"t to be at least partially due to problems in documen- tradeo!sin"xing things on the spot versus delay- tation and the databases provided to them.) This ing repairs would allow them to form a better situation leads to considerable problems with understanding of situations they encounter and AMTs who complain of not having the correct make better decisions. parts. The lack of availability of needed parts has In reviewing the SA requirements and resources been identi"ed as one of the most critical factors in of the maintenance leads and supervisors, it is determining whether an aircraft will be repaired or apparent that they serve largely in the role of not. There also exist problems in keeping up with coordinators and can become information the status of the inventory when there are numer- `middle-mena. In addition to administrative duties, ous people who have access to parts and may not they become involved when problems arise and keep databases up-to-date. The greatest SA need assistance is needed, providing support themselves for this group is in determining methods to insure or interacting with other organizations (e.g. main- that they have correct information on needed parts tenance control or aircraft-on-ground) to get and to provide them with a better ability to project needed support. This role is very critical in the parts requirements (Level 3 SA). While they do process of achieving good SA at the team level. work with projections from planning and with typi- When they become involved, supervisors and leads cal part usage requirements, their ability to project need to get a considerable amount of situational requirements for parts could probably be enhanced information from the AMTs or from others who through better system feedback and advanced may be in geographically distributed locations. planning. This process can be highly prone to information Maintenance control (MC) and its sub-groups falling through the cracks or can result in in- appear to function largely in a trouble shooting, dividuals not forming a full understanding of a reactive mode. They become involved when help is 316 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 needed and primarily focus on expediting problem impact of decisions or to understand why other solutions by bringing resources (parts, expertise, organizations are making certain statements. These routing) to bear on identi"ed problems. They face issues are very important to e!ective decision mak- several challenges in this role. Maintenance control ing, particularly when organizations are highly geo- has a great deal of general system knowledge at its graphically distributed and such di!erences may disposal, both in terms of technical skill and docu- not be evident. Each maintenance site also needs to mentation. AMTs in the "eld have the best situ- be able to share relevant information (e.g. problems ation knowledge, however, as they are on-site with detected with certain aircraft or parts) across sites the aircraft and have the most contact with perti- in order to allow the whole organization to achieve nent aircraft data. The challenge is to combine the highest knowledge level possible. Across the these two sources of information most e!ectively to organization, members of each group need to de- arrive at proper diagnoses and solutions. This gap velop an understanding of what information is between those with situation information and those needed by other groups and how to clearly pass on with the best technical knowledge may be reduced needed information about their own situations. with improved understanding between the two groups or from technologies that assist in the shar- ing of information between the two groups. Although the stated goal of maintenance opera- 5. System and organizational recommendations tions control is to minimize placards, it should be noted that they primarily focus on making The issues unearthed by this analysis are myriad. sure existing placards do not exceed prescribed Clearly, many di!erent strategies can be proposed time limits. The process seems to be to "rst approve that would be helpful for improving information placards (if allowable) and then to work to remove #ow and SA within the organization. Table 5 shows them. Neither maintenance operations control nor several changes and improvements to both the maintenance control appear to focus on proactive technologies and organizational system that can be tasks to avoid placards in the "rst place. This state implemented to improve the distribution of in- of a!airs also appears to form a gap between these formation. While these factors are speci"c to this organizations and the AMTs in the "eld. It may be airline, many of these proposed solutions may also that some organizational streamlining between be applicable to the maintenance organizations of maintenance operations control and maintenance other airlines. control may also be of bene"t, reducing the need to Improving situation awareness within a widely have distributed decision making in meeting their distributed organization such as aviation mainten- shared goal. ance, however, will also require that personnel are In reviewing the SA resources used by each able to properly utilize this information to form group, several general comments can also be made. good higher level assessments of the situation (com- It appears that the AMTs interact mostly with prehension and projection) and are better able other team members on site. Moving up in the to communicate information within and across organization, leads and supervisors are far more maintenance teams. For this reason, a program for likely to interact with other groups (such as plan- training to improve team SA within aviation ning, stores, and maintenance control) and with maintenance was also developed. maintenance units at other stations, as do the sup- port organizations. These groups have an increased need to understand the other groups they interact with. For example, understanding the di!erences 6. Team SA training recommendations between maintenance sites (manpower availability and skill levels, load levels, parts and equipment Several training concepts were identi"ed for im- availability) may be very important in allowing proving Team SA within the maintenance setting personnel to develop a good understanding of the based on this analysis and discussion. M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 317

Table 5 Technological and organizational interventions

Information system improvements

E Support for information exchange with out-sourced maintenance organizations and vendors E Support to insure proper parts are available and supplied for jobs E Support for tracking parts on order E Integration of multiple databases used within the maintenance organization E Support for tracking MEL items, parts reliability information, and &&hot'' items E Redesign of user interface/usability of computer support system Organizational and personnel improvements E Better feedback and guidance from management E Better understanding between maintenance organizations E Better teamwork E Improved organizational stability E Improved use of consistent procedures for engineering authorizations and obtaining parts E More training and use of experienced personnel E Organizational streamlining E Increased involvement by personnel at all levels in problem solving

6.1. Shared mental models factors they take into account in their decision processes, a better shared model can be developed. From the analysis it was determined that di!er- This should greatly enhance not only interpersonal ent teams (organizations) do not have a good men- interactions among teams, but also the quality of tal model of what other teams know, do not know the decision processes. or need to know. Good situation awareness at the team level depends on having a clear understanding 6.2. Verbalization of decisions of what information means when it is conveyed across team members. Thus, teams need to share There also exists a need for teams to do a better not only data, but also higher levels of SA, includ- job of passing information to other teams regarding ing the signi"cance of data for team goals and why they decide to (or not to) take a particular projection information. This process is greatly en- course of action (e.g. deferments, schedules, etc2). hanced by the creation of a shared mental model Unless the rationale and reasons are passed along, which provides a common frame of reference for considerable misunderstandings may occur. In ad- team member actions and allows team members to dition, this will deny the possibility of getting better predict each other's behaviors. A shared mental information from the other team, who may have model may provide more e$cient communications access to other pertinent information that would by providing a common means of interpreting and make for a more optimal solution. Conveying why predicting actions based on limited information, a particular decision was made provides a much and therefore may form a crucial foundation for greater level of SA (particularly at the comprehen- e!ective teamwork. When shared mental models sion level). It allows other teams to either under- are not present, one team may not fully understand stand and accept the decision or to o!er other the implications of information transmitted from solutions that may be better in achieving organiza- another team and misunderstandings, errors and tional goals. More information also needs to get ine$ciencies are likely to occur. By providing each conveyed on what diagnostic activities have been team with better information on the goals of other performed when passing aircraft to another station, teams, how they perform their tasks, and what and a need exists for better communications 318 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 between stations and teams in general. Training feedback. Not only do managers and leads need to that focuses on teaching people to verbalize the take an active role in providing this feedback, but rationale behind decisions and provide greater de- AMTs (and others) can also be trained to provide tail regarding diagnostic activities should help im- more feedback (either over the phone or through prove team SA considerably. the computer system) on what worked and what did not. 6.3. Better shift meetings and teamwork 6.5. SA training Team leads need to receive explicit training on how to (1) run a shift meeting to convey common Common problems can be linked to SA failures goals for the team, (2) provide a common group in a number of systems, including (1) forgetting understanding of who is doing what, (3) set-up an information or steps, frequently in association with understanding of the inter-relationship between task interruptions, (2) not passing information be- tasks and personnel activities and (4) provide ex- tween shifts or team members, (3) missing critical pectations regarding teamwork. Shift meetings pro- information due to other task related distractions, vide an excellent opportunity to provide this shared and (4) misinterpreting information due to expec- understanding among the members of a team. This tations. Training can be used to provide heightened information is crucial for allowing team members awareness of these problems and ways of combat- to have a good mental model regarding what every- ing them. For instance, task interruptions are one is doing and how tasks inter-relate so that they a common problem leading to SA errors. Fre- can have good SA regarding the impact of their quently, such interruptions lead to skipping steps actions and tasks on other personnel and on the or missing activities. Personnel can be trained to overall goal. Team leads also need to receive speci- take particular measures following a task interrup- "c training on the importance of passing informa- tion (double check previous work performed, tion on job status within teams over the course of double check area for loose tools, etc.). This type of the shift. Without this type of feedback, people can training may be useful for helping maintenance easily lose sight of how they are progressing in personnel to insure that they are not missing criti- relation to the other tasks being performed. This cal information in the performance of their tasks. feedback is important for individual performance and SA, and also for fostering a team spirit in 6.6. Team SA training program carrying out activities. The team SA training course (Endsley and 6.4. Feedback Robertson, 1997; Robertson and Endsley, 1997) was developed to address these "ve SA training Currently, personnel receive little feedback on goals and objectives. In addition, the course also how well a particular solution worked. A tricky provides a review of maintenance resource manage- diagnosis and repair may have been totally success- ment (MRM) principles which are considered to be ful, or may have failed again a few days later at prior knowledge requirements for the trainees. The another station. At present, it is very di$cult to team SA training course was designed to be pre- track the performance of a particular action or part sented as an 8 h classroom delivery to personnel (partially due to the cumbersome nature of the from across all maintenance operations depart- computer system). Such feedback is crucial to the ments. The course is best taught to a class development of better mental models of the tech- composed of a mixed cross section from di!erent nical systems AMTs work on. Without such feed- maintenance operations organizations (e.g. stores, back, it is very di$cult to improve one's diagnostic AMTs, inspectors, maintenance operations control, skills. While system enhancements are recom- etc.). This is because the course focuses on helping mended to help with this problem, it is also re- to reduce the gaps and miscommunications that commended that people be trained to provide such can occur between these di!erent groups and it was M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 319 anticipated that much of the course's bene"t would of line mechanic (AMT), followed by leads and come from the interaction that occurs when supervisors. A good cross sections of other organ- trainees share di!erent viewpoints and information izations within the maintenance organization in going through the exercises. were also represented, including inspection, An extensive set of Powerpoint௡ slides covering planning, and documentation support personnel. the "ve team SA training principles, group exer- Attendees were very experienced at their jobs cises, maintenance examples and case studies are (mean"10.41 yr) and within the organization included as part of the course to encourage active (mean"12.16 yr). learning. The instructional strategy used for the The evaluation process consisted of three levels: course features adult inquiry and discovery learn- Value and usefulness of the training, pre/post train- ing. This allows a high level of interaction and ing measures, and changes in behavior on the job. participation amongst the trainees creating an ex- A questionnaire was administered immediately fol- periential learning process. The team SA training lowing the course to get participant's subjective course strongly encourages participation in prob- opinions on the value and usefulness of the course. lem solving, discussion groups and responding to In addition, the amount of learning on attitudes open ended questions, thus promoting the acquisi- and behaviors related to SA was also measured. tion and processing of information. An evaluation form was provided immediately prior to the training to assess knowledge and behaviors of the trainees related to SA. It was administered 7. Evaluation of team SA training course again immediately following the course to measure changes in attitudes and self-reported intentions to In order to assess the e!ectiveness of the course, change behavior as a result of the training. The an initial evaluation of it was conducted at the form was administered again 1 month later to as- same subject airline. sess changes in behavior on the job.

7.1. Evaluation method 7.2. Results

The team SA training course was delivered by 7.2.1. Value and usefulness the airline at four of its large maintenance bases. The post-training course evaluation was used to Most of the maintenance organization personnel in measure the level of usefulness and perceived value this airline had already received MRM training of the course. Course participants scored each sub- which is considered to be a precursor to the team section of the course on a "ve-point scale which SA training course. The course was delivered over ranged from 1-waste of time to 5-extremely useful. a two-day period by this airline. (It was expanded On average, they rated each of the major topics as from the original 8 h course design by this airline to `very usefula (mean scores between 3.5 and 4.7). In allow for more group exercises, interaction and addition to rating topics in the course, participants case studies.) also answered several questions related to the Seventy-two people from nine di!erent mainten- course as a whole, shown in Fig. 4. The mean rating ance locations attended the training sessions at for the course overall was 4.3, corresponding to which the present evaluation took place. Participa- better than `very usefula. A whopping 89% of the tion in the course was voluntary and participation participants viewed the course as either `very use- in the course evaluation was also voluntary and fula or &&extremely useful'', representing a high level con"dential. Participants were present from a full of enthusiasm for the course. There were no low cross-section of shifts. The majority of the partici- ratings of the course as a whole. Over 94% of the pants were male (86%), however, 14% of the par- participants felt the course was either `very usefula ticipants were female. The participants came from or `extremely usefula for increasing aviation safety a wide range of technical operations departments and teamwork e!ectiveness (mean rating of 4.4). and job titles. The most frequent job title was that Over 89% felt the course would be either very or 320 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325

Fig. 4. Overall course evaluation. extremely useful to others (mean rating of 4.3). The Wilcoxon non-parametric statistical analy- When asked to what degree the course would a!ect sis revealed that attitudes and self-reported behav- their behavior on the job, 83% felt they would iors changed signi"cantly on seven of the 33 items make a `moderate changea or a `large changea,as (p(0.05). Participants reported after the training shown in Fig. 5. they would be more likely to keep others up-to- date with their status as they go along in doing 7.2.2. Changes in behavior and attitudes their jobs (an increase of 15%). They also were The mean change in the post-test compared to slightly more likely to report that they would try to the pre-test on each behavior described in the keep up with which activities others were working pre/post self-reported SA behavioral measure form on over the course of the shift (an increase of 10%). was also assessed. A factor analysis on the ques- Both of these items relate to improved situation tionnaire revealed a moderate degree of homogen- awareness across the team. eity. That is, responses on the items were somewhat Participants reported they would be more likely interrelated, however, no large groupings of related to try to understand others' viewpoints when en- factors were revealed to explain a large portion of gaged in a disagreement with other departments the variance. (Only one factor accounted for more (an increase of 15%). This relates to an e!ort to than 10% of the variance, with most accounting for develop better shared mental models regarding less than 5%). The questionnaire was therefore other departments. In addition, participants re- treated as a set of independent items. Changes ported changes in several behaviors related to im- on each item were compared for each subject proved communications and teamwork. They were using a paired comparison analysis (pre-test to more likely to report improved written commun- post-test). ication when sending an aircraft with a MEL to M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 321

Fig. 5. Perceived a!ect of course on behavior. another station (an increase of 21%). Participants responses on each item between the post-test ques- were more likely to report that they would tionnaire and the one-month questionnaire was make sure to pass on information about an aircraft made using the Wilcoxon test. This analysis re- and work status to the next station (an increase vealed no changes on any of the test items at a 0.05 of 13%). level of signi"cance. Therefore, it would appear that They were also more likely to report making sure the behaviors participants reported they would en- all problems and activities are discussed during gage in following the training were carried out in shift meetings (an increase of 11%), and encourag- practice, at least for this small sample. ing others to speak up during shift meetings to voice concerns or problems (an increase of 12%). 7.3. Discussion of training evaluation These di!erences between the pre-test and post- test measures on SA related behaviors and attitudes Overall the SA team training course was highly indicates that in addition to participants respond- successful. The course content associated with all of ing positively to the course, they reported actual the major training objectives was rated very highly changes in behaviors they would make on the job with the vast majority of participants rating each as a result of the course, thus improving SA on the area as `very usefula or `extremely usefula. The job both between and within maintenance teams. course was viewed as between `very usefula and `extremely usefula overall, for increasing aviation 7.2.3. Changes on the job safety and in terms of usefulness to others. The In order to assess whether participants actually course training methods and media, including the made the intended changes in their job behaviors case studies, videos and group exercises, were following the course, the same form was again viewed as particularly successful and supportive in administered one month following the course. At acquiring the learning objectives. In fact, the only the time of this analysis, the participants of only suggestion many participants had for improvement one course had been on the job for a full month was to use even more of these materials. Clearly, an after the training session. Of these participants (17), instructional strategy that emphasized experiential six responses were available for this analysis (rep- learning and participation was e!ective for achiev- resenting a return rate of 35% which is typical of ing the training objectives and facilitating the learn- mail-in questionnaires). A paired comparison of ing process. 322 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325

The course was administered to a fairly experi- maintenance. Fig. 6 illustrates the enthusiast sup- enced aviation maintenance group who represented port for crew resource management (CRM) and a wide range of departments and skill areas within MRM courses by #ight operations and main- maintenance department of the airline. The fact tenance participants, respectively, as measured that the course included such a mix of participants immediately following training (Taylor and also was viewed as a key ingredient in its success. Robertson, 1995). This is compared to the response The mixed group allowed people from di!erent measured in this study to the team SA training areas to better understand each other's viewpoints, course. Nearly two-thirds of the #ight operations contributing to the development of shared mental groups reported that the CRM training was `very models and open communications for future deci- usefula or `extremely usefula (Helmreich et al., sion making. 1987). Even though this response is very strong, the The majority of participants felt that the course response of maintenance personnel to the MRM would result in making changes in their behaviors training was even stronger. Ninety percent of the on the job. The results of the follow-up question- maintenance personnel sampled at two di!erent naire, administered one month after the training airlines felt the course was `very usefula or `ex- course, supported these intentions. The self-re- tremely usefula (Robertson et al., 1995; Taggart, ported behavior follow-up questionnaire showed 1990). The team SA training course, evaluated in that participants were making the changes they had this study, drew a response that was comparable to intended to make following the training. that found for the highly successful MRM Training These results are very similar to those achieved program that was conducted at the same airline in previous evaluations of MRM training programs (Taylor and Robertson, 1995). Based on this result, which have been shown to be highly successful in it can be concluded that the team SA training improving safety and performance in aviation course is viewed as highly useful at a rate that is

Fig. 6. Comparison of team SA training course usefulness to CRM/MRM courses. M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 323 favorably compared to previous courses in the being performed by other team members), their MRM/CRM area. ability to make correct assessments (e.g. whether These "ndings are based on the responses of an a detected problem should be "xed now or plac- initial group of course participants. To further vali- arded for later), and their ability to correctly project date these "ndings, this evaluation should be con- into the future to make good decisions (e.g. time tinued with succeeding groups of trainees in the required to perform task, availability of parts, etc.). course. In addition, more follow-up research is Results from the SA requirements analysis con- needed to validate the results of the on-the-job ducted here provide a "rm foundation for identify- behavior changes. At the time of this analysis, very ing the knowledge, skills, and abilities that few course participants had been back on the job maintenance personnel need to attain a high level for one full month. Therefore, the sample size for of SA. The SA requirements identi"ed provide in- this analysis was very small; Probably, too small for formation on the speci"c knowledge that mainten- much con"dence in the results. By following up ance personnel need to achieve a high level of SA with the remaining participants at the one month for completing their tasks. Providing personnel point (and again at longer durations), more reliable with knowledge is not enough, however. Mainten- results can be obtained regarding the degree to ance personnel must also have the skills and abil- which the training e!ected job behaviors related ities required to e!ectively communicate that to SA. knowledge, and need the ability to recognize which Finally, it would be highly desirable to ascertain information needs to be exchanged among and the degree to which the training impacts critical between team members. Several gaps between maintenance performance measures at the airline. teams were discussed. The bottom-line objective is to reduce maintenance In addition to specifying the role of SA in an errors, improve aviation safety and improve perfor- aircraft maintenance environment, an assessment mance. Since the course had been administered to was made of systems and technologies used to so few participants (scattered over 9 cities), making support SA in this organization, and potential any meaningful assessment of the e!ect of the train- areas for improvement identi"ed. The training con- ing on performance outcomes was not feasible in cepts proposed here also show promise as a means this study. In the future, however, the e!ect of the of enhancing the skills and abilities needed for training implementation on several key safety and achieving a high level of SA in a team environment. performance measures should be assessed. These concepts were used to create a deliverable training program that was prototyped by a major airline. By enhancing the SA skills of the various 8. Conclusion maintenance teams, speci"cally focusing on areas where SA breakdowns are likely to occur, improve- Overall, the applicability of the concept and im- ments in the e$ciency and safety of the aircraft portance of situation awareness in maintenance maintenance operations can be achieved. teams has been supported in this analysis. Teams of The prototype implementation of the course was AMTs are supported by many other personnel and used to obtain an indication of the value of training organizational units to achieve their goals, each of of this nature. The idea of using training to improve which has a major impact on the attainment of the SA is very new and no courses for training team SA these goals. In this context it is necessary to exam- have previously been developed. The evaluation ine how information #ows between and among presented here represents an initial evaluation of team members in order to identify system and per- the team SA training course in its prototype imple- sonnel factors that will impact on the degree to mentation phase. It was the "rst time the course which team members are able to maintain an accu- had been o!ered to a group of maintenance person- rate picture of an aircraft's status. This information nel. The fact that it was viewed so positively as appears to be crucial to their ability to perform useful to maintenance operations is highly indica- tasks (as each task is interdependent on other tasks tive of its success. It is strongly recommended that 324 M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 the airline continue to implement the course and Endsley, M.R., Robertson, M.M., 1997. Team situation aware- that additional airlines consider adopting the ness training program. In: Human Factors Issues in Aircraft course to improve SA within maintenance organ- Maintenance and Inspection '97 CD-ROM. Galaxy Scienti"c Corporation, Atlanta, GA. izations. Overall, the value of the team SA training Endsley, M.R., Rodgers, M.D., 1994. Situation awareness in- program has been supported by this analysis and formation requirements for en route air tra$c control. In: further implementation and evaluation of the Proceedings of the Human Factors and Ergonomics Society course are recommended. 38th Annual Meeting, Santa Monica, CA, pp. 71}75. Hartel, C.E., Smith, K., Prince, C., 1991, De"ning aircrew coord- ination: searching mishaps for meaning. Paper presented at the Sixth International Symposium on Aviation Psychology, Acknowledgements April. Helmreich, R.L., Foushee, H.C., Benson, R., Russini, W., 1987. This work was performed under a sub-contract Cockpit management attitudes: exploring the attitude be- to Galaxy Scienti"c Corporation, Dr. William John- havior linkage. Aviation, Space and Environmental Medi- cine 57, 1198}1200. son, Project Managers, for the Federal Aviation Klein, G.A., 1989. Recognition-primed decisions. In: Rouse, Administration O$ce of Aviation Medicine, Dr. W.B. (Ed.), Advances in Man}machine Systems Research. William Sheppard and Ms. Jean Watson, Program JAI Press, Inc, Greenwich, CN, pp. 47}92. Manger. We express our sincere appreciation to Marx, D.A., Graeber, R.C., 1994. Human error in aircraft main- Ms. Karin Porter, Mr. John Stelly and Mr. Tom tenance. In: Johnston, N., McDonald, N., Fuller, R. (Eds.), Aviation Psychology in Practice. Avebury, Aldershot, UK, Kirwan at Continental Airlines for their support pp. 87}104. of this research and their innovative e!orts to National Transportation Safety Board, 1984. Aircraft Accidents improve technical operations in aviation. Report, Eastern Air Lines, Inc., L-1011, Miami, Florida, May 5, 1983, Author, Washington, DC. O'Neill, M.J., Robertson, M.M., 1996. A participatory design process for developing ergonomic o$ce design criteria. In: References Brown, O., Hendrick, H.W. (Eds.), Human Factors in Or- ganizational Design and Management } V. North-Holland, Drury, C., 1993. In: Corporation, G.S. (Ed.), Training for visual Amsterdam, pp. 209}215. inspection of aircraft structures. Human Factors in Aviation Robertson, M.M., Endsley, M.R., 1997. Development of a situ- Maintenance } Phase Three, Progress Report (DOT/FAA/ ation awareness training program for aviation maintenance. AM-93/15), O$ce of Aviation Medicine. Federal Aviation In: Proceedings of the Human Factors and Ergonomics Administration, Washington, DC, pp. 133}154 (Chapter Society 41st Annual Meeting, Santa Monica, CA, pp. eight). 1163}1167. Endsley, M.R., 1988. Design and evaluation for situation aware- Robertson, M.M., O'Neill, M.J., 1994. Development of contex- ness enhancement. In: Proceedings of the Human Factors tual inquiries for examining knowledge workers in technolo- Society 32nd Annual Meeting, Santa Monica, CA, pp. 97}101. gical o$ce environments. Herman Miller, Inc., Zeeland, MI. Endsley, M.R., 1989. Final report: situation awareness in an Robertson, M.M., Taylor, J.C., Stelly, J.W., Wagner, R., 1995. advanced strategic mission. NOR DOC 89-32, Northrop A systematic training evaluation model applied to measure Corporation, Hawthorne, CA. the e!ectiveness of an aviation maintenance team training Endsley, M.R., 1993. A survey of situation awareness require- program. In: Proceedings of the Eighth International Sym- ments in air-to-air combat "ghters. International Journal of posium on Aviation Psychology, Columbus, OH, pp. Aviation Psychology 3 (2), 157}168. 631}636. Endsley, M.R., 1994. Situation awareness in FAA Airway Facili- Ru!ner, J.W., 1990. A survey of human factors methodologies ties Maintenance Control Centers (MCC). Final Report, and models for improving the maintainability of emerging Texas Tech University, Lubbock, TX. army aviation systems. US Army Research Institute for the Endsley, M.R., 1995a. A taxonomy of situation awareness errors. Behavioral and Social Sciences, Alexandria, VA. In: Fuller, R., Johnston, N., McDonald, N. (Eds.), Human Salas, E., Dickinson, T.L., Converse, S., Tannenbaum, S.I., 1992. Factors in Aviation Operations, Avebury Aviation. Ashgate Toward an understanding of team performance and training. Publishing Ltd, Aldershot, England, pp. 287}292. In: Swezey, R.W., Salas, E. (Eds.), Teams: their training and Endsley, M.R., 1995b. Toward a theory of situation awareness. performance. Ablex, Norwood, NJ, pp. 3}29. Human Factors 37 (1), 32}64. Taggart, W., 1990. Introducing CRM into maintenance training. Endsley, M.R., Jones, D.G., 1995. Situation awareness require- In: Proceedings of the Third International Symposium on ments analysis for TRACON air tra$c control. TTU-IE- Human Factors in Aircraft Maintenance and Inspection, 95-01, Texas Tech University, Lubbock, TX. Washington, DC, pp. M.R. Endsley, M.M. Robertson / International Journal of Industrial Ergonomics 26 (2000) 301}325 325

Taylor, J.C., Robertson, M.M., 1995. The e!ects of crew resource Taylor, J.C., Robertson, M.M., Peck, R., Stelly, J.W., 1993. management (CRM training in airline maintenance: results Validating the impact of maintenance CRM training. In: following three year's experience. NASA Ames Research Proceedings of the Seventh International Symposium on Center O$ce of Life and Microgravity Sciences and Aviation Psychology, Columbus, OH, pp. 538}542. Applications, Mo!ett Field, CA.