NASA/CR--2002-211406
Activity Tracking for Pilot Error Detection from Flight Data
Todd J. CalIantine San Jose State University, San Jose, California
June 2002 The NASA STI Program Office ... in Profile
Since its founding, NASA has been dedicated to CONFERENCE PUBLICATION. the advancement of aeronautics and space Collected papers from scientific and science. The NASA Scientific and Technical technical conferences, symposia, Information (STI) Program Office plays a key seminars, or other meetings sponsored part in helping NASA maintain this important or co-sponsored by NASA. role. SPECIAL PUBLICATION. Scientific, The NASA STI Program Office is operated by technical, or historical information from Langley Research Center, the lead center for NASA programs, projects, and missions, NASA's scientific and technical information. often concerned with subjects having The NASA STI Program Office provides access substantial public interest. to the NASA STI Database, the largest collection of aeronautical and space science STI TECHNICAL TRANSLATION. in the world. The Program Office is also English-language translations of foreign NASA's institutional mechanism for scientific and technical material disseminating the results of its research and pertinent to NASA's mission. development activities. These results are published by NASA in the NASA STI Report Specialized services that complement the STI Series, which includes the following report Program Office's diverse offerings include types: creating custom thesauri, building customized databases, organizing and publishing research TECHNICAL PUBLICATION. Reports of results ... even providing videos. completed research or a major significant phase of research that present the results For more information about the NASA STI of NASA programs and include extensive Program Office, see the following: data or theoretical analysis. Includes compilations of significant scientific and • Access the NASA STI Program Home technical data and information deemed to Page at http://www.sti.nasa, gov be of continuing reference value. NASA counterpart of peer-reviewed formal • E-mail your question via the Internet to professional papers, but having less help@sti.nasa.gov stringent limitations on manuscript length and extent of graphic presentations. • Fax your question to the NASA STI Help Desk at (301) 621-0134 TECHNICAL MEMORANDUM. Scientific and technical findings that are • Telephone the NASA STI Help Desk at preliminary or of specialized interest, e.g., (301) 621-0390 quick release reports, working papers, and bibliographies that contain minimal Write to: annotation. Does not contain extensive NASA STI Help Desk analysis. NASA Center for AeroSpace Information CONTRACTOR REPORT. Scientific and 7121 Standard Drive technical findings by NASA-sponsored Hanover, MD 21076-1320 contractors and grantees. NASA/CR--2002-211406
Activity Tracking for Pilot Error Detection from Flight Data
Todd J. Callantine San Jose State University, San Jose, California
National Aeronautics and Space Administration
Ames Research Center Moffett Field, California 94035
June 2002 Acknowledgments
This work was funded under the System Wide Accident Prevention element of the FAAJNASA Aviation Safety Program. Thanks to the NASA Langley B757 Flight Test team for their assistance with data collection.
Available from:
NASA Center for AeroSpace Information National Technical Information Service 7121 Standard Drive 5285 Port Royal Road Hanover, MD 21076-1320 Springfield, VA 22161 301-621-0390 703-605-6000 (e.g., Byrne and Bovair, 1997), that can elucidate Introduction the cognitive bases of human error. Problems associated with human error have long This report describes an application of the Crew been recognized (e.g., Babbage, 1961). More Activity Tracking System (CATS) that could recently, Perrow (1984) characterized how high contribute to future efforts to reduce flight crew system complexity contributes to accidents and, errors. It demonstrates how CATS tracks crew together with the introduction of 'glass cockpit' activities to detect errors, given flight data and air aircraft (e.g., Wiener and Curry, 1980), traffic control (ATC) clearances (already invigorated interest in human error. Reason's provided, in some cases, by digital data link (1990) theoretical treatment marks the beginning communication technology, e.g., Smith, Brown, of human error research on several fronts. One Poison, and Moses, 2001). CATS implements a sort is devoted to the collection and analysis of so-called 'intent inference' technology, called data on human error and its effects. For example, activity tracking, in which it uses a computational Johnson (1998) investigates methods for 'engineering' model of the operator's task, analyzing temporal features of incidents, as well together with a representation of the current as new ways to report incidents (Johnson, 2000). operational context, to predict nominally Accessing information in a large database of preferred operator activities and interpret actual incident reports has in turn led to research on operator actions. advanced search tools (McGreevy, 2001). CATS, too, has its roots in glass cockpit aircraft Another research area focuses on formal methods automation research. It was originally that can help reveal potential error-related implemented to track the activities of Boeing 757 problems during the design process. For example, pilots, with a focus on automation mode errors Degani and Heymann (2000) use formal (Callantine and Mitchell, 1994). The CATS specifications of system behavior to identify activity tracking methodology was validated as a unsafe interface abstractions. Sherry et al. (2001) source of real-time knowledge to support a pilot use a formal system model to explain how training/aiding system (Callantine, Mitchell, and operators misunderstand a system, and how it Palmer, 1999). CATS is useful as an analysis tool might be redesigned. Formal task representations for assessing how operators use procedures also enable scrutiny of human-error tolerance developed to support new operational concepts (Wright, Fields, and Harrison, 1994) and temporal (Callantine, 2000a, 2000b). It also serves as a aspects of operator, system, and environmental framework for developing agents to represent behavior (Fields, Wright, and Harrison, 1996). A human operators in incident analyses and methodology for analyzing the potential for distributed simulations of new operational human error during the design process also concepts (Callantine, 2001a). incorporates some of these ideas (Fields, Harrison, and Wright, 1997). Johnson (2001) examines The research described here draws in large part how error reporting can be used to support from these earlier efforts. In particular, the CATS system refinement in the initial stages of model of B757 flight crew activities has been implementation when the design is still in flux. expanded and refined. The representation of operational context used to reference the model Models of human operators anchor two to predict nominally preferred activities has additional areas of research. One uses similarly undergone progressive refinement. And, engineering-oriented computational models as while the idea of using CATS to detect flight crew the basis for preventing error and improving errors from flight data is not new, this report error recovery by training and later aiding the presents an example of CATS detecting a operator (e.g., Mitchell, 2000). Another research genuine, in-flight crew error from actual aircraft area seeks to develop models, either theoretical flight data. (e.g., Busse and Johnson, 1998) or computational UsingCATSto detect errors from flight data has Activity Tracking several potential benefits (Callantine, 2001b). Activity tracking is not merely the detection of First, CATS provides information about operational 'deviations'. The activity tracking procedural errors that do not necessarily result in methodology involves first predicting the set of deviations, and therefore would not otherwise be expected nominal operator activities for the reported. Second, CATS enables airline safety current operational context, then comparing managers to 'automatically' incorporate actual operator actions to these expectations to information about a detected error into a CATS- ensure operators performed correct activities. In based training curriculum. Other pilots could some situations, various methods or techniques 'relive' a high-fidelity version of the context in may be acceptable; therefore the methodology which another crew erred. Increasing the also includes a mechanism for determining that, efficiency and fidelity of information transfer although operator actions do not match about errors to the pilot workforce in this way expectations exactly, the actions are nonetheless would likely yield safety benefits. correct. In this sense, CATS is designed to "track' flight crew activities in real time and 'understand' It is important to note that flight crews need not that they are error-free. As the example below view such an application as punitive. It is illustrates, 'errors' CATS detects include those incumbent on airline safety and training that operators themselves detect and rapidly managers to ensure that the CATS model used to correct; such errors may nonetheless be useful to detect errors exactly matches the training examine. provided to flight crews. Research indicates that much of what pilots know about some autopilot In addition to parameters that define the state of functionality currently is not formally trained the controlled system, activity tracking also (Mitchell, 2000). Thus, a safety-enhancement requires data about the dynamic set of constraints program that uses CATS to detect errors would on controlled -system behavior, as well as data improve training by requiring safety and training about actual operator actions. For flight deck managers to explicate policies about how an applications, constraint data in the form of data aircraft should preferably be flown. linked ATC clearance information will likely be widely available in the near future, as noted The report is organized as follows. It first above, but a number of legal issues impede the describes the CATS activity tracking release of pilot action data (U.S. GAO, 1998). methodology, and information flow in CATS. This report takes the view that the promise of The report then describes the CATS significant safety benefits, together with implementation for detecting pilot errors. It first anonymity provisions similar to those of the describes flight data obtained for this Aviation Safety Reporting System (ASRS), can demonstration from the NASA Langley Boeing help overcome these issues in the future. Activity 757 (B757) Airborne Research Integrated tracking also requires a valid model of nominally Experiment System (ARIES) aircraft. It next correct operator activities suitable for deriving the describes two key representations. The first is a set of 'preferred' operator actions predicted portion of a CATS model of B757 flight (expected according to the nominal model) for a operations. The second is a representation of the given operational context. For the flight deck, constraints conveyed by ATC clearances that such models may be adapted from extant plays a key role in representing the current Advanced Qualification Program (AQP) models operational context (Callantine, 2002). An (U.S. FAA, 1995) and validated in high fidelity example from the available flight data then simulations. (The original CATS B757 model, illustrates CATS detecting pilot errors. The report however, was initially derived from a training concludes with a discussion of future research program at a major airline, together with expert challenges. input from line pilots.)
2 Sherry et al., 2001), CATS relies on a correctly CATS identifies two types of errors: errors of functioning system to reflect the results of actions omission, and errors of commission. It further (or inaction) in its state. CATS identifies errors by identifies errors of commission that result when using information in the CATS model that enables it to assess actions (or the lack thereof, in the 'right action' is performed with the 'wrong value.' CATS does not base these determinations the case of omissions) in light of the current on a 'formulaic' representation of how such operational context and the future context errors would appear in a trace of operator formed as a result of operator action (or activities, nor attempt to further classify errors inaction). Thus, one might view the CATS error (e.g., 'reversals') as in some research on formal detection scheme as 'closing the loop' between a methods for identifying potential errors (Wright, representation of correct task performance and Fields, and Harrison, 1994). Indeed, this would be the controlled system, and evaluating feedback difficult, given that the CATS model does not from the controlled system to ensure it 'jibes' represent the 'steps' of procedures explicitly as with correct operator activities. Given that the 'step A follows step B;' instead it represents system is operating normally and providing procedures implicitly by explicitly specifying the 'good data,' this is a powerful concept. conditions under which operators should Crew Activity Tracking System (CATS) preferably perform each action. CATS predicts concurrent actions whenever the current context CATS implements a methodology for activity satisfies conditions for performing two or more tracking in a computer-based system that has activities. CATS interprets concurrent actions been validated to work in real time (Callantine, whenever the granularity of action data identifies Mitchell, and Palmer, 1999). Figure 1 generically them as such. depicts information flow in CATS, between a controlled system and CATS, and between CATS Like analysis techniques that rely on a and applications based on it. As described above, 'reflection' of the task specification in a formal CATS uses representations of the current state of model of a system (Degani and Heymann, 2000, the controlled system and constraints imposed by
__°nstraints _ CAtTle xt
/ I Activity Model I _i t Actions 1%_, I/f
...... -I PredictiOns I
I Integrated Aid/ Training %_stem _ Interpretations I I Analysis Tool
Figure 1. Information flow within and between CATS and a generic human- machine system, and applications to error analysis, aiding, and training.
3 the environment (including performance limits with aircraft's Mode Control Panel (MCP), setting on the controlled system) to derive the current tactical targets and engaging pitch, roll, and thrust operational context. CATS then uses this modes as required to comply with air traffic representation to generate predictions from its control clearances. High-level modes such as model of operator activities. CATS compares Lateral Navigation (LNAV) and Vertical detected operator actions to its predicted activities, Navigation (VNAV) track the FMS-programmed and it assesses actions that it cannot immediately plan; other modes, such Flight Level Change (FL interpret as matching a prediction by periodically CH), achieve a tactical target state (the MCP target referencing the activity model until it receives altitude, in the case of FL CH). A detailed enough new context information to disambiguate description of the Boeing 757 autoflight system possible interpretations. mode usage is provided in Callantine, Mitchell, and Palmer (1999); see Billings (1997), Sarter Thus, two threads comprise the activity tracking and Woods (1995), and Wiener (1989) for methodology as implemented in CATS: a discussions of mode errors and automation issues. 'prediction thread' responsible for generating the context information necessary to predict nominal B757 ARIES Flight Data activities, and an 'interpretation thread' that The NASA Langley B757 ARIES aircraft, with its interprets operator actions. Displays of the onboard Data Acquisition System (DAS), resulting interpretationsmtogether with displays provided the flight data for this research (Figure for visualizing the input data, current operational 2). The DAS collects data at rates in excess of 5 context, and activity model--comprise a CATS- Hz, using onboard computers that perform sensor based analysis tool (Callantine, 2000a, 200b). data fusion and integrity checking. In future Predictions and interpretations supply the applications such functionality may be required information necessary for an aid that is integrated within CATS. Table 1 shows the collection of into the displays of the controlled system values that comprise the data set. The data include (Callantine, 1999) or, in the case of a tutoring information from important cockpit systems. The system, a high-fidelity simulation of the rightmost column of Table 1 shows data CATS controlled system. derives from the sampled values using filtering techniques. Included are crew action events CATS CATS Implementation for Flight derives from the values of control states. Target Data Error Detection value settings on the MCP are derived with 'begin' and 'end' values, as in formal action The following subsections specifically describe specification schemes (Wright, Fields, and the implementation of CATS for detecting pilot Harrison, 1996). Like the initial CATS research errors from flight data. The first is devoted to the (Callantine and Mitchell, 1994), this application flight data itself. The second illustrates a portion focuses on interactions with the autoflight system of the CATS model, and the third describes how MCP, so it only uses some of the available data. CATS generates the current operational context using a representation of ATC clearance Absent from data in Table 1 are important flight constraints. The CATS model fragment is relevant management system (FMS) data, including to an example of CATS detecting pilot errors actions pilots perform using the flight presented in the fourth subsection. management computer (FMC) control and display units (CDUs). This is a shortcoming of The following subsections all assume some the B757 ARIES DAS that future research seeks knowledge of commercial aviation and a B757- to rectify. In the interim, tracking CDU style autoflight system. The basic scheme is that interactions with CATS is feasible with the NASA pilots first program the flight plan into the FMS Ames Advanced Concepts Flight Simulator via the CDU. After engaging the autopilot (or (ACFS), a full-motion, high-fidelity glass cockpit flight director) and the autothrottles, they interact simulator (Callantine, 2000), and its desktop
4 Figure 2. Data Acquisition System (DAS) onboard the NASA B757 ARIES aircraft (inset).
counterpart, the 'miniACFS.' To detect entries necessary down to the level of pilot actions. that a pilot types into the CDU scratchpad, CATS Figure 3 illustrates eight actions. All actions derivable from the data are included in the full uses a parsing mechanism. It detects CDU keystrokes and 'releases" a fully-formed action model. Each activity in the model is represented (e.g., 'crossing restriction entered') when the with conditions that express the context under character string created exactly matches a value which the activity is nominally preferred, given that CATS expects, or when it can be determined policies and procedures governing operation of not to match any related value. Thus, unlike the controlled system. The parenthesized CATS in general, this parsing process does numbers in Figure 3 refer to Table 2, which lists incorporate a rudimentary a priori model of what the 'and-or trees' that comprise these rules. sorts of errors a pilot might make. This is an area of further research. Also absent from Table 1 are For comparison to other work that considers data concerning ATC clearances. For the present human errors involved with CDU manipulations application, cockpit observations provide required (e.g., Fields, Harrison, and Wright, 1997), the clearance information. model fragment in Figure 3 shows just one of numerous FMS configuration tasks. Note that a CATS Model of B757 Navigation Activities CATS model can also include cognitive, verbal, and perceptual 'activities,' but CATS can only Figure 3 depicts a fragment of the CATS model used to detect errors from B757 ARIES data. The predict, not interpret, activities for which no confirmatory data exists. Thus, such activities are model decomposes the highest level activity, 'fly not relevant to the present application. glass cockpit aircraft," into sub-activities as
5 Table 1. B757 ARIES data used in the present research, including derived states and action events (rightmost column). Some variables appear multiple times, because the B757 ARIES DAS collects them from multiple sources.
Time variables NAVICOMM data AFDS modes FMC-A/T internal data Derived states time dme_range fl_ch_engd fmc_at_mach_mode_reqd ve___speed time1 left dme freq hdg_hold_engd fmc_at_airspeed_mode_req alt_cap_engaged time2 righ__dme_freq hdg_sel_engd d spd. win auto_chng time3 left dme dist land_2_green fmc_active_climb ap_cmd_engd Environmental information rig__dme_dist land_3_green fmc._climb_mode_reqd Derived MCP actions total air temp left vhf freq alt_hold_engd fmc_active_cruise set MCP hdg true_wind_dir right_vhf_freq vnav_armed_engd fmc con mode_reqd set MCP alt wind__speed FMC data inav_armed_engd fmc_crz_mode_reqd set MCP spd AC position/attitude fmc_target_airspeed speed_mode_engd fmc_active_descent set MCP mach baro_alt fmc_selected_altitude thrusLmode_engd fmc_display_annunc_on set MCP vs baro_corr fmc_selected_airspeed Ioc_engd fmc_eng_ident_l hdg sel press flight_path_angle fmc_selected_mach vert_spd_engd fmc_eng_ident_2 hdg hold press ground_speed fmc crz altitude apprch_armed_engd fmc_eng_ident_3 Inav press computed_airspeed fmc_eta Ioc_armed_engd fmc_eng_ident_4 vnav press calibrated_airspeed fmc_desired_track back_course_armed engd fmc_eng_ident_5 spd press mach fmc_wpt_bearing glideslope_engd fmc_eng_ident..6 apprch press magnetic_heading fmc_cross_track_dist MCP Speed display status fmc_eng_idenL7 Ioc press magnetic_track_angle fmc_vert_dev mep_speed display_blank fmc_eng_idenL8 alt hold press pitch_angle fmc_range_to_alt Autothrottle fmc_eng_ident_9 vs mode press radio_altitude fmc wide vert_dev alarmed fmc_eng_ident_10 fl ch press AFI_S stat'es roll_angle MCP switches fmc_ga_mode_reqd thrust mode press true_track_angle ap_cmd_ctr_engd hdg_sel_reqd fmc_idle thr reqd mach toggled iru_potential_ve___speed ap_cmd_cen_gc_huh hdg_hold_reqd fmc_msg_annunciated cap cmd switch press hybrid_lat ap_cmd_cen_gr_huh Inav_reqd throttle retard_reqd Iap cmd switch press hybrid_Ion left_ap_cmd_engd vnav_reqd pitch_speed control_engd rap cmd switch press AC configuration/controls ap_cmd_left_engd spd_reqd vnav_operational arm autothrottles left_engine epr righLap_cmd_engd apprch_reqd Inav_operational Other derived actions right_engine_epr ap_cmd_right_engd Ioc_reqd tmc_valid tune left VHF flap_pos ap_cmd_center_engd alt_hold_reqd VNAV submodes tune right VHF speed_brake_handle ap_cws_center_engd vs mode_reqd fmc_vnav_speed operation set flaps left_throttle_pos ap_cws left engd fl ch reqd al set spoilers right_throtUe_pos ap_cws_righLengd thrust_mod_reqd fmc_vnav_path_operational gross_weight ap_in control IASlMach toggle fmc_vnav_alLoperational MCP target values fd_c_on mach_toggled Thrust ratings sel_mcp_altitude fdJo_on Crew Alert levels fmc_rating_1_reqd sel_mcp_heading fd_on_c crew_ale__level_a fmc_rating_2_reqd sel_mcp_speed fd_on_fo crew_ale_...level_b fmc offset annunciated sel_mcp_vert_speed AFDS switches crew_ale_._level_c fmc_throttle_dormanLreqd mcp_flare_retard_rate ap_cmd_center_reqd Status data fmc_thr_mode_reqd sel_mcp_mach ap_cmd_right_reqd eec_valid fmc_to_mode_reqd MCP bank angle settings ap_cws_center_reqd engine not out req_1_valid_resv bank angle_lim_flaps_25 ap. cws left reqd req_2_valid_resv bank_angle_lim..flaps_l 5 ap_cws_righLreqd bank_angle_lim_auto ap_cmd left reqd
6 Access CDU LEGS page (15) LEGSPush CDUkey (30) I
Performtakeoff roll (2) I Configure crossing restriction restriction value in t Enterrestrictioncrossing(16) [Dimension = t Typescratchpadcrossing(31) I VERT] (11)
aircraftConfigure(3) I Add to crossing fix (17) t Lir°r_sssinselgfiectx(32) I
autothrotUes (4) Configure I Execute route Push EXEC key (33) J modification(s) (12)
flight director Configure (5) I Navigate Laterally [Dimension = EAT] (13) | Dial MCP altitude I cockpit altitude (18) knob (34) I Flyaircraftglass(1) Configureautopilot (6) I I
FL CH (22) CH [mode = FL CH switch Fly using _ EngageFL CH] (27)FL H Push(35) MCP I
Configure flight Achieve/ management maintain system (7) altitude (19)
! Fly using V/S I (23) I
Navigate Navigate with Vertically AP or FD (Dimension = guidance (8) VERT] (14) [Dimension = speed speed knob SPD] (24) (28) (36)
Hold altitude (20)
Configure communications (9) Fly using [mode = VNAV switch VNAV (25) t EngageVNAV] (29)VNAV I Push(37) MCP
Fly profile (21)
Perform Manage speed approach (10) [Dimension = SPD] (26)
---! Figure 3. Fragment of CATS model for B757 operations. Table2. 'And-or' treesof conditionsunderwhichtheCATSmodelin Figure3 representsactivitiesas 'nominallypreferred.'CATSpredictsan activitywhenits conditions,plusall theconditionsof its parent activities,aresatisfiedby thecurrentoperationalcontext.
(1) start-of-run (2) (not above-runway-elevation) (3) (and (not above-clean-speed) (not flight-surfaces-within-limits) (not gear-within-limits) ) (4) (not autothrotUe-armed) (5) (not flight-director-on) (6) [ (and (not autopilot-cmd-mode-engaged) above-1000-feet-AGL ) ] (7) (or (not programmed-route-within-limits) route-uplink-received ) (8) (and above-1000-feet-AGL (or autopilot-cmd-mode-engaged flight-director-on) ) (9) (not comm-frequency-within-limits) (10) (or approaching-glideslope-intercept-point approach-localizer-intercept-point) (11) (not crossing-restriction-within-limits) (12) route-modifications-within-limits (13) (or autopilot-cmd-mode-engaged flight-director-on) (14) (or autopilot-cmd-mode-engaged flight-director-on) (15) (not cdu-page-LEGS) (16) (and cdu-page-LEGS (not crossing-restriction-built) ) (17) (and cdu-page-LEGS crossing-restriction-built) (18) (not mcp-altitude-within-limits) (19) (or (and (not current-altitude-within-limits) (not profile-within-limits-for-now) ) expedite-needed ) (20) (and current-altitude-within-limits (not profile-within-limits-for-now ) ) (21) profile-within-limits-for-now (22) (or (not altitude-close-to-target) expedite-needed) (23) altitude-close-to-target (24) (or fl-ch-engaged vs-engaged) (25) profile-within-limits-for-now (26) vnav-engaged (27) (not fl-ch-engaged) (28) (not target-speed-within-limits) (29) (and (not vnav-engaged) (not capturing-required-altitude) ) (30) (not cdu-page-LEGS) (31) (not crossing-restriction-built) (32) crossing-restriction-built (33) route-modifications-within-limits (34) (not mcp-altitude-within-limits) (35) mcp-altitude-within-limits (36) (not target-speed-within-limits) (37) mcp-altitude-within-limits
8 As discussed in Callantine (2002), CATS defines Several important features of the CATS model deserve mention. First, when using the model to context from a human operator's perspective to predict the currently preferred set of activities, be the situation plus any activities the operator is CATS searches the model top-down, so that engaged in performing. The situation is defined higher level activities 'subsume' their children as the system's state, together with environmental (i.e., the conditions on an activity must be met constraints and all salient relationships between before CATS can predict any of its children). the state and constraints. Each of these elements is Thus, CATS makes predictions and additionally considered to have historic, current, interpretations at every level of abstraction the and planned (or predicted) future components. model represents. Second, the model itself is States and constraints are also decomposed 'memoryless.' Given some context, CATS can hierarchically at multiple levels of abstraction as predict what the operators need to do (as necessary. discussed below, however, historic information is contained in the context, and in some cases does CATS uses a representation of context of this impact the CATS predictions). Third, the model form to generate a summary of the current can be structured to represent the activities of an operational context suitable for evaluating the individual operator, or a team of operators (cf. conditions under which activities are preferred, in Fields, Harrison, and Wright, 1997); either way, order to predict activities, and for determining CATS is capable of detecting errors that relate to whether an operator action it did not expect is in error. Whenever the state or constraints change, assigned operator roles and responsibilities. Fourth, the model contains information--beyond CATS examines the salient relationships to that provided by its structure and preference generate a set of 'context specifiers' that conditions--to support error detection. One type summarizes the current operational context; these of information concerns the automation mode are the descriptive clauses that appear in the that should be operational if a mode-engagement conditions listed in Table 2. CATS also uses the action was successfully invoked. Another symbolic constraint representation to maintain a concerns the 'dimension' of the operational record of compliance with constraints. This is context that an activity addresses. important not only for context generation, but also for logging flight path deviations. Representation of ATC Clearance Constraints for Context Generation Error Detection Example Environmental constraints play a key role in The report now presents an example of CATS defining the goals that shape worker behavior in detecting errors from B757 ARIES flight data complex sociotechnical systems (Vicente, 1999). collected during recent flight test activities. CATS also relies on a representation of Although the data are real, in the flight test environmental constraints to construct a environment, strict procedures about how the representation of the current operational context pilots should preferably fly the airplane are (Figure 1). These factors motivated recent unreasonable. Nonetheless, by imposing the research on a symbolic representation of the model depicted in part in Figure 3, CATS was constraints ATC clearances impose on flight able to detect errors, and the errors were not operations (Callantine, 2002). Figure 4 shows the contrived. While the errors CATS detects are representation, which represents three key insignificant, because they in no way dimensions of constraints: vertical, lateral, and compromised safety, the exercise nonetheless speed. CATS employs a rule base that enables it demonstrates the viability of CATS for error modify this constraint representation to reflect the detection. It should be noted that, in this constraints imposed (or removed) by each new application, as the following 'snapshots' show, ATC clearance. CATS runs at between twelve and twenty-two times real time.
9 _ Ol_ndi_ _ ._ _.:._: ._:_.;__.:_;_:_.:: :_:_..!i__:_,_s>:_;_ _ ,:_-::,:_:_:!:i:z::_:,i:ii!,i,:_:-:_:,_,i?i:iii:_;,:i:_<_:.::!;-:_.%.._:i:_::_:_<_i:;:_:_!:__.i_,_i:i_:i:i_-_i_:_::i:?_?'_!i:_?i:i _i:i_:!:?i:_':i::"i:_:i:i 'Turn left he_dmg 010. cltmb and _nt_n 12(100it" L_____Ev _ _ __ .....F __ FLY_ I : _ ._OU'r-SP[_S :' ECIDN-Ct'B-SPD I ' E CON (_..B SP:] ,::?_:::IIE::_ICRZ._SIO,._. '_X::O':O:. :_ .._T _ _,_i ..... ,,' ';;:;; _." _'_ Figure 4. Snapshot of a CATS representation of environmental constraints constructed from the filed flight plan, and modified according to constraints conveyed by ATC clearances. Figure 5 shows the CATS interface at the start of engage until a new target altitude is set. To assess the scenario (Scenario Frame 1). The crew has the VNAV switch press with regard to the current just received a clearance to "climb and maintain context, in which airplane is still in ALT HOLD 16,000 feet." CATS modifies its representation mode at 12,000 feet, CATS searches its model to of ATC clearance constraints accordingly, and determine if any parent activities of the VNAV using the updated context, predicts that the crew switch press contain information linking the should set the new target altitude on the MCP by action to a specific context. CATS finds that the dialing the MCP altitude knob. 'engage VNAV' activity should reflect VNAV mode engagement in the current context (see In Scenario Frame 2 (Figure 6), a pilot instead Figure 3). Because this is not the case, CATS flags pushes the VNAV switch. Because CATS has not the VNAV switch press as an error. Meanwhile, predicted this action, it cannot interpret the action CATS still expects the crew to dial the MCP initially. CATS instead continues processing data. altitude knob. In Scenario Frame 3 (Figure 7), CATS has In Scenario Frame 4 (Figure 8), a pilot does received enough new data to interpret the VNAV begin setting the MCP altitude. CATS interprets switch press action. Had the action been correct, this action as matching a current prediction, but the autoflight system state would have reflected with an incorrect value, as the altitude setting has this by engaging the VNAV mode and not yet reached 16,000. commencing the climb. However, VNAV will not 10 i TRANS_TO_ Running _ 2200% af r Figure 5 (Scenario Frame 1). In response to a clearance to climb, CATS predicts the crew should set the new target altitude on the MCP by dialing the MCP altitude knob._ Figure 6 (Scenario Frame 2). CATS detects that a crew member pressed the VNAV switch instead. 11 Runm_j It 1467% Of r Figure 7 (Scenario Frame 3). CATS cannot reconcile the VNAV switch press with the current context, and therefore flags it as an error. CATS is still expecting the crew to dial the MCP altitude knob. should engage VNAV, which is preferred CATS does not flag this action as a 'wrong value' according to the CATS model. error, however, because it is only the start of the altitude setting. CATS continues to predict 'dial CATS detects a second FL CH switch press in MCP altitude knob' because the context specifier Scenario Frame 7 (Figure 11). Perhaps a pilot 'mcp-aititude-within-limits' is not generated performed this action as 'insurance' to engage a when the current MCP target altitude is compared mode to begin the climb. Because FL CH mode to the value specified by the representation of engages, and this is reflected in CATS' ATC constraints (see Figure 3 and Table 2). representation of the current context, CATS interprets both FL CH switch presses as correct In Scenario Frame 5 (Figure 9), one pilot pushes acceptable alternative actions. By this time, CATS the VNAV switch a second time before the has also flagged the second VNAV switch press as altitude setting is complete. As the other pilot an error. completes the altitude setting, CATS interprets the end of the altitude setting action as matching its In the final frame of the scenario (Scenario prediction. Frame 8, Figure 12), the aircraft has begun In Scenario Frame 6 (Figure 10), CATS detects climbing in FL CH mode. At this point the crew that a pilot has pressed the FL CH switch (perhaps opts to engage VNAV mode. At last, CATS to begin the climb in FL CH mode, since VNAV detects the predicted VNAV switch press and did not engage). Because the MCP target altitude interprets it as correct. is now properly set, CATS predicts the crew 12 Figure 8 (Scenario Frame 4). CATS detects a pilot starting to dial the MCP altitude, and interprets it as matching its prediction, but with the wrong value (This is not an error, because the action is only tile start of the altitude setting). Figure 9 (Scenario Frame 5). CATS detects a second VNAV switch press, prior to the time when the altitude settin_ is finished. 13 Figure 10 (Scenario Frame 6). CATS detects that the crew has now opted to engage FL CH mode by pressing the FL CH switch; but because the altitude is nowproperly set, CATS now predicts the crew should push the VNAV switch to engage VNAV (the preferred mode according to the CATS model). Figure 11 (Scenario Frame 7). CATS detects a second 'insurance' FL CH switch press, and interprets it as acceptable as it did the first FL CH switch press. 14 Figure12(ScenarioFrame8).Thecrewoptsto engage VNAV; CATS detects the predicted VNAV switch press and interprets it as correct (elapsed time from Scenario Frame 1 is --42 secs). Conclusions and Future Research Using CATS with flight data collected at 'continuous' rates results in better performance. The above example demonstrates that CATS can Event-based data, such as those available from the detect errors from flight data. Although the errors NASA ACFS, require more complicated CATS detects are inconsequential, this research interpolation methods to avoid temporal 'gaps' indicates CATS can provide contextual in the CATS representation of context that can information useful for disambiguating the causes adversely affect CATS performance. Important of deviations or unusual control actions that arise directions for further research involve improving in incident or accidents. Discoveries made using the coverage of flight data to include the FMS CATS can be incorporated into training curricula and CDUs, as well as work on methods to by connecting a CATS-based training system to a automatically acquire ATC clearance simulator and allowing pilots to 'fly' under information. This research indicates that, if CATS conditions that correspond the actual context of has access to data with full, high-fidelity coverage an error-related event. Such capabilities are also of the controlled system displays and controls, it useful outside the airline arena as they support can expose the contextual nuances that surround both fine-grained cognitive engineering analyses errors in considerable detail. and human performance modeling research. 15 Beach, FL, October, 5.C.3-1--5.C.3-12 (CD- References ROM). Babbage, C. (1961). On the principles of the Callantine, T. (2002). A representation of air calculator and other seminal writings by Charles traffic control clearance constraints for intelligent Babbage [b.1792-d.1871] and others. In P. Morrison & E. Morrison (Eds.), New York, NY: agents, Proceedings of the 2002 IEEE Conference Dover. on Systems, Man, and Cybernetics, Hammamet, Tunisia, October. Billings, C. (1997). Human-centered aviation Callantine, T., and Mitchell, C. (1994). A automation. Mahwah, NJ: Lawrence Erlbaum methodology and architecture for understanding Associates. how operators select and use modes of automation in complex systems. Proceedings of Busse, D., and Johnson, C. (1998). Modeling the 1994 IEEE Conference on Systems, Man, and human error within a cognitive theoretical Cybernetics, San Antonio, TX, 1751-1756. framework. In F. Ritter & R. Young (Eds.), Proceedings of the 2 "a European Conference of Callantine, T., Mitchell, C., and Palmer, E. (1999). Cognitive Modeling (ECCM-98), Nottingham, GT-CATS: Tracking operator activities in UK: Nottingham University Press, 90-97. complex systems. NASA Technical Memorandum 208788, Moffett Field, CA, NASA Ames Research Byrne, M., and Bovair, S. (1997). A working Center. memory model of a common procedural error. Cognitive Science, 21 (1), 31-61. Degani, A. and Heymann, M. (2000). Some formal aspects of human automation interaction. Callantine, T. (1999). An intelligent aid for flight NASA Technical Memorandum 209600, Moffett deck procedure performance: The Crew Activity Field, CA: NASA Ames Research Center. Tracking System "Task Caddy." Proceedings of the 1999 IEEE Conference on Systems, Man, and Fields, R., Harrison, M., and Wright, P. (1997). Cybernetics, Tokyo, October, 1095-1100. THEA: Human error analysis for requirements definition. Technical Report 2941997, York, UK: Callantine, T. (2000a). A glass cockpit crew University of York Computer Science activity analysis tool. SAE Technical Paper 200- Department. 01-5522. Warrendale, PA: SAE International. Fields, R., Wright, P., and Harrison, M. (1996). Callantine, T. (2000b). Online analysis of high Temporal aspects of usability: Time, tasks and fidelity simulation data. Proceedings of the errors. SIGCHI Bulletin 28(2). International Conference on Cognitive Systems Engineering for Process Control 2000, Taejon, Johnson, C. (1998). Representing the impact of Korea, November, 175-182. time on human error and systems failure. Interacting with Computers, 11, 53-86. Callantine, T. (2001a). Agents for analysis and design of complex systems. Proceedings of the Johnson, C. (2000). Novel computational 2001 International Conference on Systems, Man, techniques for incident reporting. In D. Aha & R. and Cybernetics, Tuscan, AZ, October, 567-573. Weber (Eds.), Intelligent Lessons Learned Systems: Papers from the 2000 Workshop Callantine, T. (2001b). The crew activity tracking (Technical Report WS-00-03), Menlo Park, CA: system: Leveraging flight data for aiding, AAAI Press. training, and analysis. Proceedings of the 20th Digital Avionics Systems Conference, Daytona 16 Johnson, C. (2000). Supporting the analysis of Smith, N., Brown, J., Poison, P., and Moses, J. human error in national and international (2001). An assessment of flight crew experiences incident reporting schemes. In P. Cacciabue with FANS-1 controller-pilot data link communication in the South Pacific. Proceedings (Ed.), Proceedings of the 19th European Annual Conference on Human Decision Making and of the 4'h USA Europe Air Traffic Management Manual Control, Ispra, Italy, June, 137-144. R&D Seminar, Santa Fe, NM, December. Johnson, C. (2001). The London Ambulance U.S. Federal Aviation Administration (1995). Service, computer aided dispatch system: A case Advanced qualification program, Advisory study in the integration of accident reports and Circular AC 120-54. the constructive design of safety-critical computer systems. Reliability Engineering and Systems U.S. General Accounting Office (1998). Aviation Safety, 7I(3), 311-326. safety: U.S. efforts to implement Flight Operational Quality Assurance programs, Flight McGreevy, M. (2001). Searching the ASRS Safety Digest, 17(7-9), 1-36. database using QUORUM keyword search, phrase search, phrase generation, and phrase Vicente, K. (1999). Cognitive work analysis: discovery, NASA Technical Memorandum Toward safe, productive, and healthy computer- 210913, Moffett Field, CA: NASA Ames based work. Mahwah, NJ: Lawrence Erlbaum Research Center. Associates. Mitchell, C. (2000). Horizons in pilot training: Wiener, E. (1989). The human factors of Desktop tutoring systems. In N. Sarter & R. advanced technology ("glass cockpit") transport Amalberti (Eds.), Cognitive Engineering in the aircraft. NASA Contractor Report 177528, Aviation Domain. Mahwah, NJ: Lawrence Moffett Field, CA: NASA Ames Research Center. Erlbaum Associates, 211-251. Wiener, E., and Curry, R. (1980). Flight deck Perrow, C. (1984). Normal accidents: Living with automation: Promises and problems. Ergonomics, high-risk technologies. New York, NY: Basic 23, 995-1011. Books. Wright, P., Fields, R., and Harrison, M. (1994). Reason, J. (1990). Human error Cambridge: Deriving human-error tolerance requirements Cambridge University Press. from tasks. Proceedings of the 1" IEEE International Conference on Requirements Sarter, N., and Woods, D. (1995). How in the Engineering, Colorado Springs, CO, April. world did we ever get into that mode? Mode error and awareness in supervisory control. Human Factors, 31(1), 5-19. Sherry, L, Feary, M., Polson, P., Mumaw, R., and Palmer, E. (2001). A cognitive engineering analysis of the vertical navigation (VNA V) function. NASA Technical Memorandum 210915, Moffett Field, CA: NASA Ames Research Center. 17 Report Documentation Page FormApproved OMB No. 0704-0188 Pubic repo_ burdenIocthis mlbctionof informationis estimated to averaget ho_ pm response,indud_gthe time Iorreviewinginstructions,searchingexistingdata source, garbing and rnaintziningN data needed,a_l cOnllp_ingand reviet_ngI_e oollec_n ofinlorrnation.Sendcommentsregarding this b_Men estinw,le _r any _hef aspect of this col]ecti_ ofinformation,includingsuggestionsforredzJc=_gthis bu_en. IoWashinglonHeadquarters Servctm, Dnciorate for k_formationOperationsandRepods, 1215 Je_rson Davis Highway, Suile 1204, Atlinglon,VA 22202-4302, and Io the Office of Manageme_ and BudgeL Paperv,_'k ReductionPm_ct (0704-0188), Weshi_, OC L:_-_03. 1.AGENCYUSE ONLY(Leaveblank), 2. REPORTDATE 3.REPORTTYPEANDDATESCOVERED June 2002 Technical Memorandum 4. TITLE AND SUBTITLE 5. FUNDINGNUMBERS Activity Tracking for Pilot Error Detection from Flight Data 728-20-10 6. AUTHOR(S) Todd J. Callantine 7. PERFORMINGORGANIZATIONNAME(S)AND ADDRESS(ES) 8. PERFORMINGORGANIATION REPORTNUMBER NASA Ames Research Center Moffett Field, California 94035-1000 IH-030 9. SPONSORING/MONITORINGAGENCY NAME(S)ANDADDRESS(ES) 10.SPONSORING/MONITORING AGENCYREPORTNUMBER National Aeronautics and Space Administration NASA/CR--2002-211406 11. SUPPLEMENTARYNOTES Point of Contact: Everett Palmer, M/S 262-4, Ames Research Center, Moffett Field, CA 94035 (650) 604-6073 12A. DISTRIBUTION/AVAILABILITYSTATEMENT 12B. DISTRIBUTIONCODE Subject Category: 03-01, 63-02 Distribution: Public Availability: NASA CASI (301) 621-0390 13. ABSTRACT(Maximum200 words) This report presents an application of activity tracking for pilot error detection from flight data, and describes issues surrounding such an application. It first describes the Crew Activity Tracking System (CATS), in-flight data collected from the NASA Langley Boeing 757 Airborne Research Integrated Experiment System aircraft, and a model of B757 flight crew activities. It then presents an example of CATS detecting actual in-flight crew errors. 14. SUBJECTTERMS 15. NUMBEROF PAGES 18 Human error detection, Activity tracking, Glass cockpit aircraft 16. PRICECODE 17. SECURITYCLASSIFICATION 18. SECURITYCLASSIFICATION 19. SECURITYCLASSIFICATION 20. UMITATIONOF ABSTRACT OF REPORT OFTHIS PAGE OF ABSTRACT Unclassified Unclassified Unclassified Unlimited NSN 754001-280-5500 Standard Form 298 (Rev 2-89) Prescribed by ANSI Std. Z-39-18 296-102