JOURNAL OF SPACECRAFT AND ROCKETS

Pilot Performance, Workload, and Situation Awareness During Lunar Landing Mode Transitions

Christopher J. Hainley, Jr.∗ Massachusetts Institute of Technology, Cambridge, Massachusetts 02139 Kevin R. Duda† Charles Stark Draper Laboratory, Inc., Cambridge, Massachusetts 02139 and Charles M. Oman‡ and Alan Natapoff§ Massachusetts Institute of Technology, Cambridge, Massachusetts 02139 DOI: 10.2514/1.A32267 Aircraft and spacecraft pilots frequently change their level of supervisory control between full autopilot and other modes, providing varying levels of manual control. Therefore, multimodal systems must transition “gracefully,” meaning without unsafe decreases in flight performance or unacceptable changes in workload or situation awareness. Thirteen subjects flew a fixed base simulation of the NASA Constellation Program Altair lunar lander that transitioned from full autopilot to one of three flight-director-guided rate-command attitude hold manual control modes. After training, each subject flew 24 approaches, half of which included a landing point redesignation at the time of the mode transition requiring the pilot to null additional guidance errors. Bedford subjective workload and two-choice embedded secondary task response times were used to quantify temporal changes in mental workload. Situation awareness transients were detected by analysis of a tertiary task, verbal callouts of altitude, fuel, and terrain hazards. Graceful transitions were particularly difficult because Altair’s large inertia made the plant dynamics relatively sluggish. Transitions to manual control increased subjective and objective workloads and decreased callout accuracy in proportion to the number of flight control axes being manually commanded.

I. Introduction Automation design heuristics [4], theories of supervisory control – ANY large transport aircraft and spacecraft, such as the [5], and level-of-automation (LOA) taxonomies [6 8] were initially M Apollo Lunar Landing Module and the U.S. Space Shuttle, developed for teleoperation, powerplant, and air traffic control were designed to be capable of almost fully automatic landing. Future applications but only bluntly characterize the manual control options spacecraft may be capable of fully automatic landings when used for available in modern autoflight systems. Employing a high LOA uncrewed missions (e.g., [1]). However, in crewed systems, pilots lowers the mental workload and improves the overall situation generally insist on the availability of manual control modes, allowing awareness (SA), but remaining engaged at a low LOA keeps the pilot them to change from full supervisory mode to some level of manual aware of the aircraft state and more able to detect and respond to control, e.g., while reprogramming an approach route or to diagnose a certain failures (e.g., [9]). It has long been recognized that the LOA system failure [2]. For example, airline pilots can choose to engage must vary during operational use depending on situational demands. However, some aircraft autoflight system modes are clumsy and heading, altitude, and/or thrust autoflight modes or control each of – them manually. NASA spacecraft human rating requirements [3] unpredictable [10], complicating mode awareness [11 14] and dictate that astronauts must be able to manually override higher- resulting in transitions that surprise even experienced operators level software control/automation and control spacecraft attitude and [10,15]. Mode transitions reduce operator performance in error flight path. Following the dictum “fly as you train, train as you fly,” detection and error mitigation in automated systems [2,9,16,17]. The need for aircraft and spacecraft automations that support graceful both astronauts and aircraft pilots practice transitioning between reversion from automatic to manual flight has long been recognized automation modes and often choose to fly part or all of an approach (e.g., [18–20]). Previous studies of the effects of the LOA (e.g., [21]) using manual modes, often aided by flight director guidance. assess performance, mental workload, and situation awareness but have focused on steady-state behavior rather than the transient. In aviation, pilots and their instructor evaluators typically speak of Received 27 October 2011; revision received 21 December 2012; accepted the “gracefulness” of a specific mode transition, referring to the Downloaded by MASSACHUSETTS INST OF TECHNOLOGY (DRAPER LAB) on June 19, 2013 | http://arc.aiaa.org DOI: 10.2514/1.A32267 for publication 31 December 2012; published online 19 June 2013. Copyright undesirable decrease in operator performance, increase in workload, © 2013 by The Charles Stark Draper Laboratory, Inc. Published by the and change in situation awareness associated with a mode transition, American Institute of Aeronautics and Astronautics, Inc., with permission. even when the transition is expected, and so automation surprise is Copies of this paper may be made for personal or internal use, on condition not an additional factor. Because each dimension of this gracefulness that the copier pay the $10.00 per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923; include the code construct can be quantified, it provides a useful three-dimensional 1533-6794/13 and $10.00 in correspondence with the CCC. metric for the evaluation of the transient effect of mode transitions. *MIT Research Assistant, Man-Vehicle Laboratory, Department of To date, no studies have attempted to quantify such transient effects Aeronautics and Astronautics, 77 Massachusetts Avenue 37-219; currently of the reversion to manual flight in terms of concurrently measured Draper Laboratory Fellow, Charles Stark Draper Laboratory, Inc. manual control performance, subjective and objective mental † Senior Member of the Technical Staff, Human-Centered Engineering workloads, and situation awareness and how these depend on Group, 555 Technology Square; AIAA Life Sciences and Systems Technical transition characteristics such as the change in the number of manual Committee. Senior Member AIAA. ‡ control loops the operator must control. This study examined these Senior Research Engineer and Senior Lecturer, Man-Vehicle Laboratory, Department of Aeronautics and Astronautics, 77 Massachusetts Avenue 37- questions in the context of a lunar landing scenario, but there are 219. potentially broader applications. §Research Scientist, Man-Vehicle Laboratory, Department of Aeronautics Automation of control functions previously assigned to a human and Astronautics, 77 Massachusetts Avenue 37-219. operator often improves the overall system performance and AIAA Early Edition / 1 2 AIAA Early Edition / HAINLEY ET AL.

reliability, can significantly decrease operator workload [22], and can use of a timed two-choice response task to probe short-term improve failure detection [23,24]. These benefits typically come at variations in mental workload, detailed next, is new. the cost of potential operator complacency [9], skill degradation [2], SA can be generally defined as “the perception of the elements in and the inability of operators to diagnose and respond to subsystem the environment with respect to time and/or space, the com- failures [25]. To enhance operational flexibility and operator prehension of their meaning and the projection of their status in the confidence, a variety of automation modes are typically provided. near future” [40,41]. The subject’s perception of their own awareness Within a particular system mode, the role of the human operator may has been assessed using of subjective scales (e.g., Situation range from the supervisory level (e.g., human specifies a series of Awareness Rating Technique [42]), but such ratings are limited by the high-level goals and simply monitors the execution) to a level in subject’s own perceptions of reality. Similarly, ratings by expert which the human both sets goals and executes low-level manual observers (e.g., SALSA [43]) are constrained because observers may commands. This range of interactions has been referred to as the LOA have only limited knowledge of the operator’s concept of the situation [8]. Systems frequently have numerous modes; for example, the [21]. Objective measures are traditionally obtained via analysis of autoflight systems on large commercial aircraft have several dozen answers to questions administered during simulation freezes (e.g., heading-, altitude-, and trust-related modes. Systems are typically Situation Awareness Global Assessment Technique [41]) or by mea- designed so that some modes operate concurrently, and certain mode suring the reaction time to probe questions related to the displayed transitions occur automatically or can be only selected if certain information (e.g., SPAM [44]). However, we wanted to be able to conditions are met. Variouserror conditions can cause the automation assess short-term temporal changes in critical SA variables without to initiate a reversion to manual flight, but the operator typically the intrusion of simulation freezes or distracting probe questions or retains authority to choose the subsequent LOA (“adaptable reliance on posttrial memories. We therefore analyzed the timeliness automation” [26]). The alternative of designing the automation so and accuracy of the required verbal callouts of altitude, fuel, and terrain, which our pilot subjects made during their approach to that it monitors or anticipates pilot performance, workload, and “ ” situation awareness and selects the appropriate LOA (adaptive landing. Many experienced pilots employ self-talk during instru- automation) is being explored [27,28]. The issue is whether such ment approaches, verbalizing important situational variables, employing phonologic working memory in order to stabilize their systems can reliably evaluate the situation and estimate pilot attention patterns and instrument scan (e.g., [45,46]). This is a new performance, workload, and situation awareness. So far, the method of assessing task critical elements of SA and has relatively consensus is that the human should usually retain the LOA decision high face validity because Apollo Lunar Module (LM) astronauts authority [29]. also called out altitude, fuel, and terrain features. The classic “crossover” mathematical model for multiloop manual The goal of the current study was to quantify flying performance, control [30] predicts that, when flying in a manual mode, the subjective and objective workloads, and situation awareness during performance of the combined human-operator-vehicle system (as various types of automation mode transitions, in order to develop measured by the closed-loop system bandwidth) depends on vehicle objective measures of mode transition gracefulness. We hypoth- dynamics and the ability of the operator to anticipate vehicle behavior “ ” esized that reversion from autoflight to a flight control mode and create control lead. Manual control performance decreases as a requiring the operator to close a larger number of manual control function of the number of information loops the operator must sample information loops would impose a higher attentional mental and close. Increasing the number of manual control loops increases workload and would therefore have a larger detrimental effect on pilot mental workload as measured experimentally by side task manual control performance, mental workload, and aspects of performance (e.g., [31]). Human operators can successfully adapt situational awareness not immediately related to the manual control to sudden changes in vehicle dynamics [32,33], though under task. We also hypothesized that the mental workload would increase conditions of high “urgency,” unexpected input disturbances, and situation awareness would decrease monotonically with the changes in vehicle dynamics, pilot attention, or mode of manual number of manual control loops the pilot had to close. Because control can cause loss of operator lead, resulting in excessive the lunar landing vehicle dynamics were relatively challenging to control gain and pilot-induced oscillations [34]. Despite these various manually control, we also wanted to confirm that, if large guidance studies, human operator performance, mental workload, and errors were suddenly introduced when the landing point was situation awareness have not been concurrently quantified during the redesignated, the additional task of reducing this error would increase period of transition from full autopilot to various manual modes. workload and decrease situation awareness, as compared to cases in Mental workload can be defined as the fraction of an operator’s which there was no redesignation. limited capacity used to perform a task [35]. An operator’s infor- mation processing and attentional capacity in various sensory channels (e.g., visual, auditory) [36] are limited. At moderate and II. Methodology high workload levels, performance normally remains constant until A. Lunar Lander Simulator Displays spare attentional capacity is nearly exhausted. Mental workload is The subjects were trained to fly approaches in a fixed-base lunar therefore typically measured via the assessment of spare attentional lander simulator. Seated subjects viewed a primary flight display capacity, either directly employing subjective scales of spare (PFD) on a monitor at eye level and approximately arm’s length attention, such as the hierarchical modified Bedford workload scale Downloaded by MASSACHUSETTS INST OF TECHNOLOGY (DRAPER LAB) on June 19, 2013 | http://arc.aiaa.org DOI: 10.2514/1.A32267 distance and a horizontal situation display (HSD), located imme- – [37 39] or by quantifying performance on a secondary task. We diately to the right of the PFD. As shown in Fig. 1 (bottom left), the employed both techniques in this study. When a secondary task is PFD depicted the vehicle pitch and roll attitude (earth/sky horizon, used to assess mental workload, it should use the same sensory pitch ladder, and roll indicator), heading and horizontal velocity resources as the primary task but be performed only as the attentional (on a horizontal situation indicator), altitude, altitude rate (vertical demands of the primary task permit. Manual control places high velocity) using symbology adapted from aircraft and helicopter demands on visual attention. Visual reaction time secondary tasks electronic flight instrument displays [47,48], percentage of fuel that impose a continuing cognitive demand are frequently used, remaining, and the estimated time until touchdown. The active involving varying degrees of processing complexity [35]. Embedded autopilot flight control mode was shown on an annunciator panel at secondary visual monitoring tasks that are part of the operator’s the top of the PFD. To reduce pilot workload, magenta “flight normal role have face validity. One technique requires the operator to director” lines were presented on the attitude indicator, heading indicate which of two peripherally displayed indicators has appeared display, and altitude rate displays showing reference values [31]. As compared to a single indicator response task, the two-choice determined by the guidance system such that, if the pilot “flew to” task requires decision making. A similar secondary task was used in them, the vehicle flight path would converge with the reference this study. By controlling the time of appearance of the secondary trajectory to the selected landing target. indicator, it is also possible to quantify how mental workload/spare As shown in Fig. 1 (right), the HSD displayed a lunar terrain attention varies temporally after the automation mode changes. This elevation map (plan view, heading up) with gray-shaded elevation AIAA Early Edition / HAINLEY ET AL. 3

Fig. 1 Top, lunar lander simulator setup. Bottom left, PFD and bottom right, HSD.

contours. The vehicle location in the horizontal plane was depicted by Classic multiloop manual control theory [30] explains why such an “ownship” symbol, which remained at the center of the HSD, and dynamics have relatively poor handling qualities. However, the dual so, as the vehicle moved, the terrain map translated and rotated cue flight director provided significant assistance because the pilot relative to it. The location of three alternative landing points, chosen could concentrate on following the inner-loop flight director attitude by the vehicle’s hazard detection system [49], were shown as cues and trust that the vehicle would descend very close to the numbered circles. The currently selected guidance system aimpoint guidance computer’s desired flight path. The PFD did not explicitly (#2 in the Fig. 1 example) was boxed in magenta, with a digital display lateral or vertical flight-path errors. display of the horizontal range in feet and time until touchdown (min: As with the Apollo LM, the pilot could manually control the rate of s). The surface areas that were overflyable but judged unsafe for descent (ROD) in increments of 1ft∕s. In our simulation, the ROD touchdown were shaded in red. The secondary task display was could be commanded using a three-position button on a left-hand located at the lower-right corner of the HSD and consisted of a small inceptor, or the autopilot could automatically control the rate of turquoise circle. Use of the secondary task display, described as a descent. As the vehicle tilted from the vertical, the guidance system communication (COMM) indicator response, is detailed later. automatically recommended an increase in descent engine thrust in proportion to the cosine of the tilt angle to maintain the commanded B. Lunar Lander Dynamics, Guidance, and Control ROD. The vehicle ground track approached the primary landing aimpoint from one of the four corners of the terrain map. The

Downloaded by MASSACHUSETTS INST OF TECHNOLOGY (DRAPER LAB) on June 19, 2013 | http://arc.aiaa.org DOI: 10.2514/1.A32267 The subject controlled vehicle attitude using a three-axis joystick held in the right hand. The vehicle dynamics represented those of guidance laws were designed to follow a reference trajectory that was NASA’s Constellation Program “Altair” lunar lander (LDAC1-Delta calculated based on the range to target and the projected time to design). The descent engine had a fixed gimbal; fuel slosh and arrival at a point 150 ft above the selected landing point [51]. This consumption effects were not modeled. As with the Apollo LM, the reference trajectory was continually updated based on the actual joystick provided the pilot with manual rate command-attitude hold location of the simulated vehicle in reference to the selected landing (RCAH) control. The thrusters on the Altair LDAC1-Delta were point, and the guidance recommended pitch, roll, yaw attitude, and similar to those on the LM, but the Altair had a larger inertia, and so rate of descent was continuously updated and displayed on the PFD. the control authority and maximum rate command was limited All approaches began at an altitude of 500 ft above the simulated (3.0 deg ∕s2 and 30 deg ∕s in pitch and roll and 2.0 deg ∕s2 and lunar surface, 417 ft from the center of the landing area map (location 20 deg ∕s in yaw). Hence, the joystick deflection determined the of the primary aimpoint), a descent rate of 16 ft∕s, a horizontal attitude rate through first-order lag dynamics with a time constant that velocity of 15 ft∕s toward the aimpoint, and pitched and rolled 19 deg was short for small inputs but could be as long as 5 s for maximum in opposition to the horizontal velocity. The descent rate decreased rate commands. This sluggish RCAH control characteristic made the linearly to 3ft∕s until the vehicle was below 150 ft [51] and within vehicle challenging to fly manually when large inputs were required 15 ft horizontally of the designated landing point. This range criterion because the dynamics approximated those of a second-order system was kept regardless of control mode. The control system always to attitude and a fourth-order to position over the lunar surface [50]. maintained the vehicle facing north. 4 AIAA Early Edition / HAINLEY ET AL.

The experiment was designed to study transitions from fully The two-choice visual secondary task used to measure mental automatic (FA) mode to one of three manual control modes. In the FA workload used a turquoise-colored circle displayed on the lower- mode, the autopilot controlled the attitude and ROD of the vehicle; right portion of the HSD. The circle was labeled “COMM” because the subject had the responsibility of monitoring the vehicle states and the subjects were told that the secondary task was a proxy for a real- performance. The following three manual modes were investigated: world mission control datalink communication task. The subjects 1) “One-axis,” which had pitch axis RCAH manual control with were instructed that, as soon as the outer portion of the circle color automatic rate-of-descent. The autopilot automatically controlled the changed to blue (as shown in Fig. 1) or green, they should respond by roll and descent rate and maintained a northerly heading, whereas the pushing a correspondingly colored button on their joystick. The pilot manually controlled the pitch. response time to each COMM stimulus was recorded as a mental 2) “Three-axis,” which had roll-pitch-yaw RCAH manual control workload measure. The COMM circle appeared a total of 10 times with automatic rate of descent. The autopilot controlled the descent during each approach trial, permitting the assessment of temporal rate, whereas the pilot manually controlled the roll, pitch, and yaw. changes. The COMM circle appearance times were deliberately 3) “Three-axis ‡ ROD,” which had roll-pitch-yaw RCAH manual varied within 2 s windows so that the subject could not anticipate the control with incremental manual control of rate of descent (requiring exact time of appearance. If the subject was distracted and failed to a total of four-axis manual control). Three-axis ‡ ROD is the analog respond within 4–6 s, the COMM circle color was reset to turquoise. of the “P66” manual control mode used in the Apollo LM during For the tertiary SA assessment task, the subjects were instructed to touchdown [52]. verbally call out vehicle altitude, fuel, and terrain status at specific intervals during the approach. The altitude was to be reported every C. Subjects and Experimental Protocol 50 ft between 450 and 250 ft, every 25 ft between 250 and 150 ft, and Thirteen volunteer subjects [10 males, 3 females; 26.2  3.4 every 10 ft below 150 ft altitude. The fuel level was to be reported (mean  standard deviation) years old] participated in the study whenever the integer percentage changed (e.g., 7 to 6%). The subjects and were compensated for their time. None withdrew or failed were also to report whenever the vehicle was overflying a red to complete the experiment. The protocol was approved by the hazardous area depicted on the terrain map, e.g., by reporting Massachusetts Institute of Technology Committee on the Use of “entering red” or “departing red”. The subjects learned the callout Humans as Experimental Subjects. schedule during their familiarization and training trials, and a placard The subjects were briefed on the experiment and then flew several detailing the callouts was available on the simulator panel in front of practice approaches to familiarize themselves with the displays, them, which the subjects could refer to as desired. Each approach trial controls, and vehicle handling qualities, beginning in FA mode and thus nominally required 21 total situation awareness callouts (14 then with each of the three manual modes. Next, each subject flew 24 altitude, 4 fuel level, and 3 hazard crossings). If the pilot kept the trials in a training session practicing the three possible mode vehicle on the reference trajectory, a situation awareness callout transitions with and without landing point redesignations (LPRs), would be required approximately every 3 s. Callouts were scored as while simultaneously performing the secondary COMM indicator correct if the report occurred within 1–2 s of the actual event (as task and the situation awareness verbal callout tertiary task. Finally, determined by the experimenter). After each approach trial, the each subject flew 24 experimental trials, identical to those in the subjects used the hierarchical modified Bedford workload scale to training session. Each approach was approximately 75 s in duration retrospectively rate their spare attention before, during, and after the and terminated just before vehicle touchdown. control mode transition. “During” the transition was defined as the Each approach trial began in FA mode. Twenty to twenty-five s period beginning with the mode transition and ending when the later, the vehicle automatically transitioned to one of three alternative guidance error needles were nulled and the subject felt in control manual modes. The subject was alerted by an auditory “beep” and of the vehicle. The modified Bedford scores could range from 1 consulted the PFD mode annunciator to determine the new control (“a piece of cake”) through 4 (“there was ample time to attend to mode. The subjects were instructed that, in responding to the mode additional tasks”) and 7 (“there was minimal spare time for additional change, they should give first priority to the flight-director-based tasks”)to10(“adequate performance was impossible”). The subjects attitude and rate-of-descent flying tasks, second priority to were allowed to report using fractions (e.g., 4.5). responding to the COMM indicator for mental workload assessment, and third priority to providing verbal situation awareness callouts of D. Experiment Design and Statistical Analysis altitude, fuel, and terrain hazards. The experiment utilized a repeated-measures within-subjects In the primary manual control task, the subjects were instructed to design. Fixed-effect independent variables included the control mode null the pitch and roll attitude errors by reference to the flight director transition type (three levels, from FA to each of the three manual needles within 1.0 deg and to keep the descent rate within 0.5 ft∕s modes) and LPR (two levels, with and without) with four repetitions of that recommended by guidance. Because these error tolerances were of each for a total of 24 trials. In order that each approach situation trained to and held constant throughout the experiment and by would seem novel, four different approach directions and eight explicitly presenting vehicle state errors on the PFD, there was no different terrain maps were used. The order of control mode, LPR, observed change in the subject’s manual control gain throughout the and terrain map presentations were in pseudorandomized order to trial. The subjects were told to disregard the heading (yaw) flight minimize possible learning effects. Downloaded by MASSACHUSETTS INST OF TECHNOLOGY (DRAPER LAB) on June 19, 2013 | http://arc.aiaa.org DOI: 10.2514/1.A32267 director cue in all flight control modes. In the FA mode, the vehicle Simulator state data were recorded at 10 Hz. The data from the first would maintain a northerly heading, and in each of the manual control 69 s of each trial were processed with MATLAB to quantify flight modes, the heading flight director cue would recommend a heading control performance and COMM indicator response times. Analysis such that if the pilot yawed the lander it would be facing the direction of on the data from 24 experimental approaches flown by each subject the currently selected landing aimpoint. However, the subjects were was performed with SYSTAT 13.0 software. Trials in which the instructed not to provide any heading (yaw) control inputs. vehicle made contact with the lunar surface were omitted from the Half of the approach trials introduced a LPR, in which, at the analysis (nominal trials terminated before landing). In addition, trials moment of change to manual mode, the automatic guidance also with exceptionally large and erroneous control inputs due to self- designated a different landing point further downrange. Because the reported subject fatigue were omitted from analysis. In total, five LPR changed the approach trajectory, it introduced a step change in trials were omitted from the analysis because of subject fatigue or flight director guidance error, which the pilot was required to a crash. manually null. The maximum guidance-commanded pitch and roll Because many of the dependent variables were not normally angles were limited to 45 deg to reduce the effect of large distributed or were inherently ranked data, we utilized robust trajectory errors [51]. When the computer commanded a LPR, the nonparametric methods, particularly the Friedman test for ranked new point was always in the general direction of travel, and the data. Bonferroni posthoc corrections were applied when needed. The distance change was similar in magnitude for every LPR trial. subjects were considered a random effect. Significant effects by the AIAA Early Edition / HAINLEY ET AL. 5

Friedman test indicate that the trends embracing the entire subject “blunders” that were not classified as outliers but contributed to errors cohort are indeed strong enough to be detected because the Friedman in addition to those of simple manual tracking. Both factors distorted test’s efficiency, like that of all nonparametric tests, is lower than that the distribution. Fortunately, the counterpart nonparametric Fried- of their parametric counterparts. man alternative gave significant, if weaker, results applicable to the The dependent variables for each trial included the 10 COMM distorted distribution. indicator response times, 21 SA callouts, and 3 modified Bedford The modified Bedford subjective mental workload ratings were workload scores. Both the COMM indictor response times and SA analyzed by trial phase (before, during, and after the mode transition). verbal callouts had clusters of responses close to or at the limits of For each of the 21 SA callouts, the percentage of correct callouts allowable performance (e.g., see Figs. 2 and 3) causing their across subjects and trials was calculated and used as a temporal distributions to deviate from normal, even after the conventional measure of situation awareness, with a higher percentage indicating (Fisher) transformation. We therefore chose the more robust but less better awareness of the state callout expected at that moment in time. efficient alternative analysis through nonparametric statistics that do Friedman tests were used to assess the subjects’ agreement on which not assume normality. To compare “inner-loop” manual control COMM indicator workload probes gave the longest response times, performance across control modes, we chose to measure the mean indicating the highest mental workload. The Friedman test was square error (MSE) between the actual pitch angle and the guidance- also applied to responses under LPR and control mode to analyze recommended pitch angle following the mode transition. To compare the agreement between subjects across independent variables. The the MSE component due to manual control with and without a modified Bedford workload ratings were analyzed across trial phase LPR, the MSE was separately measured when the autopilot flew and situational awareness callouts over time within a trial. These an entire LPR approach in the FA mode (no transition) for each within-subject comparisons using the Friedman test established each approach geometry (map and direction). The result, 82.0  6.6 deg2 subject as their own control. (mean  standard error of the mean (SEM)), varied slightly due to effects of approach geometry (map and approach direction) and time of the LPR. This mean value was subtracted from the average III. Results MSE from each run that included a LPR so that MSE data from LPR and non-LPR runs could be meaningfully compared. The average A. Situation Awareness Verbal Callouts MSE for each control mode and LPR combination (across the four We had hypothesized that SA verbal callout performance would map repetitions) was calculated for each subject. A nonparametric decrease following a mode transition by an amount depending on the Friedman test on the MSE before and after the transition tested the number of axes the subject was required to control after mode effect of the mode transition on manual control performance, as well transition. Because the LPR approaches required the pilot to null as the effects of control mode and LPR following the transition. large flight director errors, it was expected that SA on these trials Although the pitch MSE performance metric is on a continuum, the would be lower due to distraction by the manual control task. data had separate clusters based on LPR/no LPR and occasional pilot Situation awareness verbal callout results, averaged across the LPR and no-LPR conditions, are shown in Fig. 2. The abscissa plots each of the required verbal callouts in the nominal temporal order throughout each trial. The effect of mode change on the percentage of correct SA callouts was dramatic, and the change was consistently related to the number of axes being controlled following mode transition. All three SA curves in Fig. 2 exhibit a transient decrease after mode transition and then gradually recover, with the three-axis ‡ ROD case never approaching the pretransition, FA mode accuracy level. Note that the lowest percentages are associated with altitude callouts made immediately after the mode transition and with the subsequent three fuel-level callouts. When transitioning to manual flight, apparently the subjects tended to neglect the fuel-level display element, even though it was on the far left side of the PFD. Overall, the subjects’rankings on the average percentage of correct callouts were concordant over all 21 subject reports (Friedman Fig. 2 Effect of control mode on the average percentage of correct statistic ˆ 177.64, degrees of freedom df†ˆ20, p < 0.0005). The situation awareness verbal callouts made (mean  SEM, n ˆ 13,*: pairwise comparisons (before and after) showed that subjects agreed < 0 05 p . by Friedman test). that the SA performance at four of the six callouts before the mode transition (450, 400, 350, and 300 ft) had significantly higher correct percentages than those callouts after the transition (300 ft vs posttransition callout: Friedman statistic ≥2.55, df ˆ 1, p ≤ 0.011). This indicates that the transition from a fully automatic to a manual Downloaded by MASSACHUSETTS INST OF TECHNOLOGY (DRAPER LAB) on June 19, 2013 | http://arc.aiaa.org DOI: 10.2514/1.A32267 control mode significantly reduced situation awareness, as measured by the percentage of correct verbal callouts. The effect of control mode change had a relatively consistent effect on SA: the relative ranks of the three control modes on the average percentage of correct SA callouts was concordant across subjects at 11 of the 21 callout points, all of them after the mode transition (Friedman statistic ≥7.51, df ˆ 2, p ≤ 0.023, un-Bonferroni corrected). Wewere unable to demonstrate a significant additional effect of the LPR on the percentage of correct SA callouts, except immediately after the mode transition: SA callout performance decreased significantly in a concordant way across subjects at the two callouts immediately after the LPR/control mode transition (250 ft compared to 225 and 200 ft; Friedman statistic ≥3.12, df ˆ 1, p ˆ 0.004). Once the subject had the vehicle stabilized on the new reference Fig. 3 Average COMM indicator response time versus probe number trajectory after the LPR, the SA at subsequent callouts apparently was for each of the control modes (n ˆ 13; mean  SEM). similar to that without a LPR. 6 AIAA Early Edition / HAINLEY ET AL.

B. Secondary Task Response Time Measures of Mental Workload We had hypothesized that, after the transition to manual mode, the mental workload would transiently increase and then remain elevated, with the magnitude of the change depending on the number of axes being manually controlled and whether or not a LPR occurred. The results of the secondary task response time, averaged across the LPR and no-LPR conditions, are shown in Fig. 3. Because the timing of each probe was deliberately varied over a 2 s interval, the abscissa is labeled by probe number. The transition to manual control occurred just before probe 4. (The legend shows the manual control mode after transition, but in all three cases, the control mode was FA before transition.) Pairwise comparisons of the response times in workload probes 1 through 3 (before transition) were found to be significantly lower than those in probes 4 through 10 (after transition) (Friedman statistic ˆ 5.14, df ˆ 1, p < 0.0005), demonstrating that mental workload increased after the mode change. The effect of probe Fig. 4 Average modified Bedford subjective workload ratings vs before, during, and after the mode transition for each of the control modes sequence number was significant over all COMM indicator response (n ˆ 13; mean  SEM). probes (Friedman statistic ˆ 82.29, df ˆ 9, p < 0.0005), indicating that the subjects were in general agreement on which parts of the approach had the highest and lowest mental workloads. D. Mode Transition and Redesignation Effects on Manual Control The subjects’ data were not concordant on the ranked effect of Performance the manual control mode on average response time immediately The MSE pitch angle error, with LPR run data corrected to remove after mode transition (probe 4). However, by probe 5, there was the average autopilot component as described earlier, was calculated marginally significant agreement (Friedman statistic ˆ 5.69, for each subject over the four repetitions of each mode and LPR df ˆ 2, p ˆ 0.058), and at probes 7 through 9, there was significant combination. As expected, the pitch MSE was found to be greater agreement (Friedman statistic ≥ 6.0, df ˆ 2, p ˆ 0.050). This after the transition than before (Friedman statistic ˆ 13.0, df ˆ 1, suggests that, by the time of probe 7, a control mode effect emerged p < 0.0005). The pitch MSE values in the “after” phase were larger in that was consistent across subjects. The curves indicate that the trials a LPR than without one (Friedman statistic ˆ 13.0, df ˆ 1, mental workload remains high for the three-axis ‡ ROD case and p < 0.0005). The subjects were concordant in the finding that the diminishes somewhat for the other manual modes. The curves, MSE increased consistently from one-axis or three-axis (which were however, were not flat but remained essentially parallel across probe indistinguishable) to three-axis ‡ ROD when there was no LPR number after probe 5. This suggests that, if there indeed was such a (Friedman statistic ˆ 7.38, df ˆ 2, p ˆ 0.025). Pairwise compar- cross effect of control mode × prode number, it was likely small or isons of the trials without a LPR found that one-axis and three-axis was masked by the variability in the data. were not significantly different and that both were significantly less Contrary to our expectations, a Friedman test on the average than three-axis ‡ ROD (Friedman statistic ≥ 2.50, df ˆ 1, response time across probes 4–7 showed no large and consistent p ≤ 0.013). The same trend was not seen when there was a LPR effect of the LPR on mental workload as measured with our COMM (Friedman statistic ˆ 4.77, df ˆ 2, p ˆ 0.092). With a LPR, only secondary task. Using pairwise Friedman comparisons, the subjects the comparison of one-axis and three-axis ‡ ROD was significant agreed only marginally that the response times during trials (Friedman statistic ˆ 2.29, df ˆ 1, p ˆ 0.013), and because the with a LPR were larger than those without the LPR at probes 4 omnibus test was not, the significance of this pairwise comparison is (Friedman statistic ˆ 3.77, df ˆ 1, p ˆ 0.052) and 7 (Friedman also dubious. The lack of a significant result during the LPR trials statistic ˆ 3.77, df ˆ 1, p ˆ 0.052). could be due to the increased variance in the MSE during trials with a LPR. This was an unexpected result and made it more difficult to C. Subjective Mental Workload establish clearly the effects of control mode on the LPR trials. In a larger sample, it is expected that a significant effect of control mode We hypothesized that the modified Bedford subjective mental on pitch axis MSE would be found for both LPR conditions. workload reports, obtained retrospectively after each trial, would The level of automation in the control mode significantly affected correspond to the secondary task data and increase following the the average pitch axis MSE in runs without a LPR: one-axis had the mode transition in proportion to the number of axes being controlled least error, and three-axis ‡ ROD had the highest; but, there was no and be higher for LPR approaches. We do not believe there were any significant difference between the one-axis and three-axis modes. recall problems on subjective mental workload reports due to short This result is consistent with previous single- and two-axis tracking trial duration (approximately 75 s). The modified Bedford workload task research [53]; the difference recorded between one-axis and reports, when ranked, were consistent for all subjects for every three-axis ‡ ROD, however, showed that there is a limit to the combination of control mode and LPR. The subjects agreed that the number of control axes that can be taken on in a mode transition “before,”“after,” and “during” workloads increased in that order Downloaded by MASSACHUSETTS INST OF TECHNOLOGY (DRAPER LAB) on June 19, 2013 | http://arc.aiaa.org DOI: 10.2514/1.A32267 without degrading performance. Trials with a LPR were only (Friedman statistic ˆ 14.0, df ˆ 2, p ˆ 0.001). The subjects also marginally affected by the control mode, which suggests a LPR × agreed on the ranking of the effects of control mode within the control mode cross effect. Reducing the number of axes controlled by “ ” during (Friedman statistic ˆ 26.0, df ˆ 2, p < 0.0005) and the operator led to improved performance, improved SA, and lower “ ” after phases (Friedman statistic ˆ 23.8, df ˆ 2, p < 0.0005) workload: a more graceful transition. (Fig. 4). [Similar to Fig. 3, the results in Fig. 4 are presented such that, before the mode transition, the control mode (although fully automatic) is labeled as one of the manual modes for clarity.] All pairwise comparisons between phases were significant (p < 0.05). IV. Conclusions The modified Bedford workload reports shown in Fig. 4 were This study utilized several novel methods. As noted, secondary averaged across the LPR and no-LPR conditions because the LPR did tasks have been widely employed to assess mental workload and not have a significant effect on the average modified Bedford spare attention. However, timing the visual stimuli to assess temporal reports in the “before” or “after” phases. However, the subjective variations in workload during automation mode transitions has not modified Bedford data from the “during” phase did show a significant been previously attempted. The results using a two-choice visual effect of LPR (Friedman statistic ˆ 8.33, df ˆ 1, p ˆ 0.003), response secondary task were congruent with retrospective modified whereas the secondary task data during the transition phase (probes Bedford workload ratings, but the former provided greater temporal 4–6) did not. resolution of effects during automation mode changes. AIAA Early Edition / HAINLEY ET AL. 7

The analysis of pilot verbal callouts in order to assess changes in illustrate the transient and steady-state costs of “getting into the loop,” situation awareness (SA) during the approach is also a new method. and indicate that the subjects at least transiently had lost most of their The results highlighted the loss of pilot attention to fuel status, spare attention reserve. The subject pilots in this study flew the altitude, and terrain during high workload periods due to the approach solely by reference to instruments (e.g., as they would if attentional demands of the manual control task. Those demands, landing in darkness deep in a crater located near a lunar pole) and which focused attention on the attitude indicator and nulling the flight without an out-the-window view. If the same PFD information had director attitude errors, likely impaired their prospective memory: been presented on a HUD, but a daylight window view of the landing remembering to perform a planned action at the appropriate time. area had also been available, arguably their attention would have been Additionally, the focus of attention on the center of the primary flight even further divided. A window view may improve important display (PFD) likely also hindered their ability to pay attention to dimensions of SA, but because workload will be high when flying critical fuel callouts, which were on the periphery. Altair manually, flight control and secondary task performance might As one would expect, the percentage of incorrect/missed callouts be further compromised. Distributing the system and environmental and mental workload were correlated, but the two measures did not monitoring tasks between two pilots and assuming manual control track each other reliably. For example, mental workload remained early enough so that the transient SA decrease has abated might elevated even for single-axis control although callout accuracy had provide an acceptable operational solution, if verified by higher- substantially recovered. The nature of the landing task is such that fidelity simulations. Providing automated monitoring and callouts of educated guesses as to the vehicle altitude, fuel, or terrain status are fuel status, rate of descent, and altitude, or assigning these tasks to a inappropriate. Had the simulation been frozen after missed reports nonflying pilot would also help. However, the fundamental problem and blanked the displays, perhaps the subjects could have guessed potentially results from Altair LDAC1-Delta’s sluggish rate their approximate altitude, fuel, and terrain status, but it is doubtful command-attitude hold (RCAH) manual control dynamics, which they could have made estimates with the required accuracy. effectively change from first-order to second-order control of attitude Offloading the callout task to a nonflying pilot (as in Apollo) or to an when large control inputs are employed and afford only limited automated fuel/altitude/terrain annunciator, thereby using a control authority. Manual control dynamics can largely be tuned to nonvisual information channel to communicate to the flying pilot, mission requirements. RCAH control of attitude is appropriate in is an obvious workaround. The SA assessment methodology could orbit and during the initial phases of lunar landing approaches. also be used to explore alternative design placement of display However, during the final vertical phases of approach to touchdown, elements related to SA variables. The technique has limitations even if RCAH lag can be reduced, based on the results, a transition to because it is possible to assess only a limited number of situation incremental horizontal translational velocity control (TVC) is awareness dimensions, as compared with the traditional scenario- recommended. TVC (also known as “translational rate command” or freeze methods (e.g., Situation Awareness Global Assessment “auxiliary hover trim”) and hover position hold modes have become Technique). However, the method is nonintrusive and has face widely utilized in modern search-and-rescue helicopters and validity because similar callouts were employed during the Apollo analogous modes have been proposed for lander spacecraft. lunar landings. Experimental comparisons should demonstrate that transitions to The experiment demonstrated that, even when lunar landing TVC rather than sluggish RCAH significantly reduces pilot autopilot mode transitions were expected and “automation surprise” workload (and hence permit an increase in SA) during the final safety and mode confusion were not contributing factors, transitions from critical phases of the approach. fully automated supervisory control to various manual control modes The focus was defining the transient effects on performance, increased mental workload metrics and decreased situation workload, and SA during the transition from full automation to awareness measures. The changes were monotonic with the number alternative manual modes. However, this method could be adapted to of manual control loops the pilot was responsible for closing. Manual study performance, workload, and situation awareness transients control performance, mental workload, and situation awareness during transitions at higher levels of automation and other metrics provide useful measures of the “gracefulness” of a mode applications, e.g., powerplant or ATC applications. As demonstrated transition. Because this was only a part task simulation conducted in a in a previous study of steady-state effects, one cannot assume that fixed-base simulator and the pilots were not astronauts, it was not performance, workload, and situation awareness always improve attempted to define absolute criteria for gracefulness in terms of the when transitioning from a lower to a higher level of supervisory maximum acceptable change in manual control performance, mental control because certain aspects of situation awareness may be workload, and situation awareness callouts. However, setting enhanced by involvement in lower level tasks. criterion levels of performance, workload, and SA could be con- The primary manual control task workload and overall situation sidered in an analogous real-world training situation. The research awareness are naturally inversely related. When a transition to a goal was to understand how the number of manual control loops manual mode takes place, the balance between them is perturbed, and closed impacted the time course of subsequent manual control the operator must find a way to bring both back to an acceptable performance, mental workload, and situation awareness, as well as balance. Vehicle manual control dynamics, cockpit controls, and the impact of a landing point redesignation. displays must be designed using manual and supervisory control Even though aided by an attitude flight director, flying in any of the theories and cognitive task analysis and then verified using quantitative human-in-the-loop simulations to be sure that, when

Downloaded by MASSACHUSETTS INST OF TECHNOLOGY (DRAPER LAB) on June 19, 2013 | http://arc.aiaa.org DOI: 10.2514/1.A32267 three manual modes studied required far greater visual attention on “ ” the PFD attitude display than when simply monitoring the fully mode changes occur, they are graceful and that workload, task automatic mode, due to the sluggish dynamics of the simulated Altair performance, and situation awareness always remain within lander. Assuming manual control increased both the objective mental acceptable limits. workload (secondary COMM task response time) and subjective mental workload (modified Bedford ratings) measures. After the pilots completed their transition to manual flight by nulling the flight Acknowledgments director errors, the subjective workload scores slightly improved, and The Draper Laboratory fixed-base lunar landing simulator was the secondary task response times remained elevated by several used for this study. We thank Justin Vican of the Draper Laboratory seconds. Despite extensive training, the transition to manual control for his support in the development and integration of fixed-base immediately reduced performance on the tertiary situation awareness simulation; Alex Stimpson, Hui Ying Wen, and Justin Kaderka of the callout task such that they missed altitude and fuel callouts, even Massachusetts Institute of Technology (MIT) and the Draper when flying single axis. After the transition phase was completed, SA Laboratory for display development and pilot testing, and, of course, measures remained less accurate than before the manual control all our experimental subjects who were recruited from the MIT and transition, particularly when manually controlling more than one Draper Laboratory communities. This research was supported by the axis. Presumably, the impaired callout performance was due to National Space Biomedical Research Institute though NASA NCC9- continuing preoccupation with the manual control task. These results 58, Project HFP02001. 8 AIAA Early Edition / HAINLEY ET AL.

References Transactions on Systems, Man, and Cybernetics, Vol. SMC-9, No. 1, 1979, pp. 24–34. [1] NASA, “Constellation Design Reference Missions and Operational ” doi:10.1109/TSMC.1979.4310070 Concepts, CxP 70007, Oct. 2008. [24] Wickens, C. D., and Kessel, C., “Processing Resource Demands of [2] Bainbridge, L., “Ironies of Automation,” Automatica, Vol. 19, No. 6, ” – Failure Detection in Dynamic Systems, Journal of Experimental 1983, pp. 775 779. Psychology: Human Perception and Performance, Vol. 6, No. 3, 1980, doi:10.1016/0005-1098(83)90046-8 pp. 564–577. [3] “Human-Rating Requirements for Space Systems,” NASA, Rept. NPR- doi:10.1037/0096-1523.6.3.564 8705.2B, May 2008. [25] Funk, K., Lyall, B., Wilson, J., Vint, R., Niemczyk, M., Suroteguh, C., “ ” Aerospace [4] Fitts, P. M., Functions of Man in Complex Systems, and Owen, G., “Flight Deck Automation Issues,” International Journal Engineering, Vol. 21, No. 1, 1962, pp. 34–39. – “ of Aviation Psychology, Vol. 9, No. 2, 1999, pp. 109 123. [5] Sheridan, T. B., Supervisory Control of Remote Manipulators, doi:10.1207/s15327108ijap0902_2 ” Advances in Man-Machine Systems Vehicles and Dynamic Processes, [26] Oppermann, R., and Simm, H., “Adaptability: User Initiated Research, Vol. 1, JAI Press, New York, 1984, pp. 49–137. Individualization,” Adaptive User Support: Ergonomic Design of [6] Sheridan, T. B., and Verplank, W. L., “Human and Computer Control of ” Manually and Automatically Adaptable Software, edited by Undersea Teleoperators, Massachusetts Institute of Technology, Rept. Oppermann, R., Lawrence Earlbaum, Hillsdale, NJ, 1994, pp. 14–66. TR-N00014-77-C-0256, Cambridge, MA, March 1978. “ [27] Bailey, N. R., Scerbo, M. W., Freeman, F. G., Mikulka, P.J., and Scott, L. [7] Endsley, M. R., and Kaber, D., Level of Automation Effects on A., “Comparison of a Brain-Based Adaptive System and a Manual Performance, Situation Awareness and Workload in a Dynamic Control Adaptable System for Invoking Automation,” Human Factors, Vol. 48, ” Ergonomics – Task, , Vol. 42, No. 3, 1999, pp. 462 492. No. 4, 2006, pp. 693–709. doi:10.1080/001401399185595 “ doi:10.1518/001872006779166280 [8] Parasuraman, R., Sheridan, T. B., and Wickens, C. D., A Model for [28] Parasuraman, R., Mouloua, M., and Molloy, R., “Effects of Adaptive ” IEEE Types and Levels of Human Interaction with Automation, Task Allocation on Monitoring of Automated Systems,” Human Transactions on Systems, Man, and Cybernetics, Vol. 30, No. 3, 2000, – – Factors, Vol. 38, No. 4, 1996, pp. 665 679. pp. 286 297. doi:10.1518/001872096778827279 doi:10.1109/3468.844354 [29] Hancock, P. A., “Procedure and Dynamic Display Relocation on [9] Endsley, M. R., and Kiris, E., “The Out-of-the-Loop Performance Performance in a Multitask Environment,” IEEE Transactions on Problem and Level of Control in Automation,” Human Factors, Vol.37, – Systems, Man, and Cybernetics, Part A: Systems and Humans, Vol. 37, No. 2, 1995, pp. 381 394. No. 1, 2007, pp. 47–57. doi:10.1518/001872095779064555 doi:10.1109/TSMCA.2006.886341 [10] Weiner, E. L., “Human Factors of Advanced Technology (“Glass “ ” ” [30] McRuer, D. T., and Jex, H. R., A Review of Quasi-Linear Pilot Cockpit ) Transport Aircraft, NASA CR-177528, June 1989. Models,” IEEE Transactions on Human Factors in Electronics, Vol. 8, [11] Sarter, N. B., “Cockpit Automation: From Quantity to Quality, from No. 3, 1967, pp. 231–249. Individual Pilot to Multiple Agents,” Automation and Human doi:10.1109/THFE.1967.234304 Performance: Theory and Applications , Erlbaum, Mahwah, NJ, 1996, [31] Ephrath, A. R., and Curry, R. E., “Detection by Pilots of System Failures pp. 267–280. ” “ During Instrument Landings, IEEE Transactions of Systems, Man, and [12] Sarter, N. B., and Woods, D. D., Pilot Interaction with Cockpit Cybernetics, Vol. SMC-7, No. 12, 1977, pp. 841–848. Automation: Operational Experiences with the Flight Management doi:10.1109/TSMC.1977.4309640 ” International Journal of Aviation Psychology System, , Vol. 2, No. 4, [32] Young, L. R., “On Adaptive Manual Control,” IEEE Transactions on 1992, pp. 303–321. Man-Machine Systems, Vol. MMS-10, No. 4, 1969, pp. 292–331. doi:10.1207/s15327108ijap0204_5 doi:10.1109/TMMS.1969.299931 [13] Sarter, N. B., and Woods, D. D., “How in the World Did We Ever Get “ ” [33] Weir, D. H., and Phatak, A. V., Model of Human-Operator Response to into That Mode? Mode Error and Awareness in Supervisory Control, Step Transitions in Controlled Element Dynamics,” NASA CR-671, Human Factors – , Vol. 37, No. 1, 1995, pp. 5 19. Jan. 1967. doi:10.1518/001872095779049516 [34] McRuer, D. T., “Pilot-Induced Oscillations and Human Dynamic [14] Sherry, L., Feary, M., Polson, P., and Palmer, E., “What’s It Doing Now? Behavior,” NASA CR-4683, July 1995. Taking the Covers Off Autopilot Behavior,” Proceedings of the 11th [35] O’Donnell, R. D., and Eggmeier, F. T., “Workload Assessment International Symposium on Aviation Psychology , Columbus, OH, Methodology”, Handbook of Perception and Human Performance, 2001, pp. 1–6. “ ” Volume II: Cognitive Processes and Performance, edited by Boff, K. R., [15] Sarter, N. B., Woods, D. D., and Billings, C. E., Automation Surprises, Kaufman, L., and Thomas, J. P., Wiley and Sons, New York, 1994, Handbook of Human Factors and Ergonomics, 2nd ed., Wiley, New – – pp. 42-1 42-49. York, 1997, pp. 1926 1943. [36] Wickens, C. D., “The Structure of Attentional Resources,” Attention and [16] Endsley, M. R., “Situation Awareness in Aviation Systems,” Handbook Performance VIII, Earlbaum, Hillsdale, NJ, 1980, pp. 239–257. of Aviation Human Factors, 1st ed., Lawrence Erlbaum Associates, – [37] Oman, C. M., Huntley, M. S. Jr., Rasmussen, S. A., and Robinson, S. K., Mahwah, NJ, 1999, pp. 257 276. “The Use of Analog Track Angle Error Display for Improving [17] Billings, C. E., “Human-Centered Aviation Automation: Principles and ” ” Simulated GPS Approach Performance, John A. Volpe National Guidelines, NASA TM-110381, Feb. 1996. Transportation Systems Center, Rept. DOT-VNTSC-FAA-95-29, [18] Palmer, M. T., Rogers, W. H., Press, H. N., Latorella, K. A., and Abbott, “ Cambridge, MA, Aug. 1995. T. S., ACrew-Centered Flight Deck Design Philosophy for High-Speed “ ” [38] Roscoe, A. H., and Ellis, G. A., A Subjective Rating Scale for Civil Transport (HSCT) Aircraft, NASA TM-109171, Jan. 1995 ” “ Assessing Pilot Workload in Flight: A Decade of Practical Use, Royal [19] Besco, R. O., and Funk, K., Conceptual Design Guidelines to Aerospace Establishment, Rept. TR-90019, Farnborough, England,

Downloaded by MASSACHUSETTS INST OF TECHNOLOGY (DRAPER LAB) on June 19, 2013 | http://arc.aiaa.org DOI: 10.2514/1.A32267 ” Rediscover Systems Engineering for Automated Flight Decks, U.K., March 1990. International Journal of Aviation Psychology, Vol. 9, No. 2, 1999, – [39] Newman, R. L., and Greeley, K. W., Cockpit Displays: Test and pp. 189 198. Evaluation, Ashgate Publishing Limited, Hants, England, U.K., 2001, doi:10.1207/s15327108ijap0902_7 pp. 70–72. [20] Bilimoria, K. D., and Mueller, E. R., “Handling Qualities of a Capsule [40] Endsley, M. R., “Automation and Situation Awareness,” Automation Spacecraft During Atmospheric Entry,” AIAA Guidance, Navigation and Human Performance: Theory and Applications, Lawrence and Control Conference , Toronto, AIAA Paper 2010-8308, 2010, Erlbaum, Mahwah, NJ, 1996, pp. 163–181. pp. 1–14. “ “ [41] Endsley, M. R., Situation Awareness Global Assessment Technique [21] Endsley, M. R., and Kaber, D. B., Level of Automation Effects on (SAGAT),” Proceedings of the National Aerospace and Electronics Performance, Situation Awareness and Workload in a Dynamic Control Conference (NAECON), Dayton, OH, IEEE, 1988, pp. 789–795. Task,” Ergonomics, Vol. 42, No. 3, 1999, pp. 462–492. doi:10.1109/NAECON.1988.195097 doi:10.1080/001401399185595 [42] Taylor, R., “Situation Awareness Rating Technique (SART): The [22] Wickens, C. D., and Hollands, J. G., “Complex Systems, Process ” ” Development of a Tool for Aircrew Systems Design, Proceedings Control, and Automation, Engineering Psychology and Human of the AGARD Symposium on Situational Awareness in Aerospace Performance – , Prentice Hall, Upper Saddle River, NJ, 2000, Operations: Aerospace Medical Panel Symposium, CP478, Copen- pp. 513–556. “ hagen, Denmark, 1990. [23] Wickens, C. D., and Kessel, C., The Effects of Participatory Mode and [43] Hauss, Y., and Eyferth, K., “Securing Future ATM-Concepts’ Safety by Task Workload on the Detection of Dynamic System Failures,” IEEE Measuring Situation Awareness in ATC,” Aerospace Science and AIAA Early Edition / HAINLEY ET AL. 9

Technology, Vol. 7, No. 6, 2003, pp. 417–427. Conference, Big Sky, MT, IEEE, 2008, pp. 1–7. doi:10.1016/S1270-9638(02)00011-1 doi:10.1109/AERO.2008.4526297 [44] Durso, F. T., and Gronlund, S. D., “Situation Awareness,” Handbook of [50] Duda, K. R., Johnson, M. C., and Fill, T. J., “Design and Analysis of Applied Cognition, edited by Durso, F. T., Nickerson, R. S., and Lunar Lander Manual Control Modes,” Proceedings of the IEEE Schvaneveldt, R. W., Wiley & Sons, 1999, pp. 283–314, Chap. 10. Aerospace Conference, Big Sky, MT, IEEE, 2009, pp. 1–16. [45] Machado, R., “Cockpit Conversations: Self Talk for Survival,” doi:10.1109/AERO.2009.4839356 Rod Machado's Plane Talk: The Mental Art of Flying an [51] Bilimoria, K., “Effects of Control Power and Guidance Cues on Lunar Airplane, 3rd ed., Aviation Speakers Bureau, San Clemente, CA, Lander Handling Qualities,” Proceedings of the AIAA Space 2008 2003, pp. 3-1–3-14. Conference, AIAA, Reston, VA, 2008, pp. 1–18. [46] Bergqvist, P., “Talk to Yourself,” Flying, Sept. 2011. [52] Hackler, C., Brickel, J. R., Smith, H. E., and Cheatham, D. C., “Lunar [47] Mueller, E., Bilimoria, K., and Frost, C., “Improved Lunar Lander Module Pilot Control Considerations,” NASA TN-D-4131, Handling Qualities through Control Response Type and Display Feb. 1968. Enhancements,” Proceedings of the AIAA Guidance, Navigation, and [53] Todosiev, E., Rose, R., and Summers, L., “Human Performance in Control Conference, AIAA, Reston, VA, 2010, pp. 1–19. Single- and Two-Axis Tracking Systems,” Proceedings of the Second [48] Szoboszlay, Z. P., McKinley, R. A., Braddom, S. R., Harrington, W. W., Annual NASA University Conference on Manual Control, Scientific and Burns, H. N., and Savage, J. C., “Landing and H-60 Helicopter in Technical Information Division, NASA, 1966, pp. 143–158. Brownout Conditions Using 3D-LZ Displays,” Proceedings of the American Helicopter Society 66th Annual Forum, Phoenix, AZ, 2010, D. Thunnissen pp. 1–30. Associate Editor [49] Epp, C. D., Robertson, E., and Brady, T., “Autonomous Landing and Hazard Avoidance Technology,” Proceedings of the IEEE Aerospace Downloaded by MASSACHUSETTS INST OF TECHNOLOGY (DRAPER LAB) on June 19, 2013 | http://arc.aiaa.org DOI: 10.2514/1.A32267