Modeling Task Prioritization Behaviors in a Time-Pressured Multitasking Environment

AN ABSTRACT OF THE DISSERTATION OF

Takeaki Toma for the degree of Doctor of Philosophy in Industrial Engineering

Presented on June 9, 2015

Title: Modeling Task Prioritization Behaviors in a Time-Pressured Multitasking Environment

Abstract approved success

Kenneth H. Funk II

Funk’s (1991) cockpit task management (CTM) theory is structurally consistent with cognitive multitasking models: it addresses managing multiple and concurrent tasks in three stages: situation awareness, response selection, and response execution. Based on CTM theory, Colvin, Funk and Braune (2005) hypothesized that the following six factors may affect task prioritization: 1. expectations, 2. importance, 3. salience, 4. status

(performance status), 5. required Time/Effort, and 6. urgency.

Based on the above two research studies, the following three research questions were investigated: RQ-1) Can perceived task priority be explained by the following five factors? perceived task importance, urgency, performance status, salience, and workload RQ-2) Is there any relationship between the perceived task priority and the chance of noticing task-related cockpit instrument malfunction signals? If so, how much does the perceived task priority affect the chance of noticing task-related signals considering the following factors: salience of task-related signals, expectancy of task-related signals, and the number of concurrent tasks? and RQ-3) Can actual task execution and task performance be explained by the perceived task priority? A medium fidelity flight simulation study was conducted to test the above research questions.

For RQ-1, the perceived task importance, perceived task urgency, and the perceived salience of the tasks were significantly related to the perceived task priority.

For RQ-2 and RQ-3, the pilots were more likely to execute the tasks and notice malfunction signals within a shorter time when the task was highly prioritized. Findings from this study are consistent with other multitasking studies: concurrent multitasking eliminates the benefits that result from alternating and integrating stimulus-driven bottom-up and goal-driven top-down processing, which is regarded as critical in task prioritization.

©Copyright by Takeaki Toma

June 9, 2015

Modeling Task Prioritization Behaviors in a Time-Pressured

Multitasking Environment

Takeaki Toma

A DISSERTATION

Submitted to

Oregon State University

in partial fulfillment of

the requirements for the

degree of

Doctor of Philosophy

Presented June 9, 2015

Commencement June 2016

Doctor of Philosophy dissertation of Takeaki Toma presented on June 9, 2015

APPROVED:

Major Professor, representing Industrial Engineering

Head of the School of Mechanical, Industrial and Manufacturing Engineering

Dean of the Graduate School

I understand that my dissertation will become part of the permanent collection of Oregon State university libraries. My signature below authorizes release of my dissertation to any reader upon request.

Takeaki Toma, Author

ACKNOWLEDGEMENTS

First and foremost, I am very grateful for the warm encouragement and assistance from my graduate committee members. I am thankful for the continued support and understanding of my advisor and mentor Dr. Kenneth Funk who provided me with such a wonderful research topic. I also appreciate Dr. Funk for providing me with teaching / research assistantships that were both rewarding and interesting experiences. I especially would like to thank Dr. Sara Emerson for her help and support through the design of experiments and data analysis process, including validating assumptions, limitations, interpretation of statistical data analysis, such as regression analysis, mixed models, and survival data analysis. I greatly appreciate Dr. Mei Lien’s advice on cognitive psychology, literature review, definition of tasks, and how to write the methodology section based on other methodologies in the literature. I appreciate Dr.

Anthony Veltri who was always supportive and expected a good study result. I appreciate Dr. Chinweike Eseonu for his encouragement and help. In addition, I appreciate the warm help and encouragement from Dr.David Cann, MIME associate head, and Dr. Brenda.McComb, the graduate school dean. I appreciate my mother,

Sachiko Toma, and my father, Isamu Toma, and my siblings Kiyoka Morita, Shoko

Toma, and Koji Toma for all of their support. I also appreciate people who warmly encouragement me, particularly Yoko Miyagi, Phuong Nguyen, Rebecca Ott, Patricia

Lacy, and Dr. Marcey Bamba.

I also would like to thank pilots Mr. Saher Bishara, Mr. Forrest Anderson and Mr.

Michael Laviolette for fruitful help and advice on the design of flight simulation experiments as well as cockpit task management. I am thankful to Dr. Christopher

Wickens and Dr. Pamela Tsang for their advice. I am also thankful to the Okinawa International Exchange and Human Resource Development Organization, and the Japan

Student Services Organization for their financial support. Finally, I'm thankful for writing help from OSU's Graduate Writing Center assistants, particularly Aimee Clark,

John Osborne, Coral Rost, and Robert Asinjo.

TABLE OF CONTENTS Page CHAPTER 1: INTRODUCTION ...... 1

CHAPTER 2: LITERATURE REVIEW ...... 3 COCKPIT TASK MANAGEMENT (CTM) THEORY ...... 3 COGNITIVE VIEW OF HUMAN TASK PRIORITIZATION BEHAIVOR ...... 7 Task Prioritization Factors in the Situation Awareness Stage...... 7 Task Prioritization Factors in the Response Selection Stage ...... 11 Task Prioritization Factors in the Response Execution Stage ...... 18 Workload and Strategic Control for Task Prioritization ...... 25 Summary of Task Prioritization Factors at the Cognitive Level ...... 28 CRITICAL TASK PRIORITIZATION FACTORS IN THE AVIATION DOMAIN30 LITERATURE SUMMARY AND RESEARCH OPPORTUNITY ...... 40 RESEARCH QUESTIONS ...... 41

CHAPTER 3: RESEARCH METHODOLOGY ...... 44 PARTICIPANTS ...... 44 EQUIPMENT ...... 45 PROTOCOL ...... 46 FLIGHT SCENARIO ...... 47 MEASUREMENT AND HYPOTHESIS-TESTING ...... 49 Definition of Terms ...... 49 Methodology for RQ-1 ...... 52 Methodology for RQ-2 ...... 55 Methodology for RQ-3 ...... 65

TABLE OF CONTENTS (Continued) Page CHAPTER 4: RESULTS ...... 67

RESULTS FOR RQ -1 ...... 68 Relationship Between Perceived task importance and Perceived task priority ...... 69 Relationship Between Perceived task urgency and Perceived task priority ...... 71 Relationship Between Perceived Task performance Status and Perceived task Priority ...... 73 Relationship Between Perceived task salience and Perceived task priority ...... 75 Relationship Between Perceived workload and Perceived task priority ...... 77 Characteristics of Perceived task urgency ...... 79 Individual Difference and Situation Difference ...... 87 Relative Importance of Five Candidate Factors ...... 93 Significant task prioritization criteria when considering five factors together ...... 96 Summary of the Result of RQ -1 ...... 99

RESULTS OF RQ -2 ...... 101 Task Priority Effect on Awareness of Task-Related Signals ...... 102 Signal Expectancy Effect on Awareness of Task-Related Signals ...... 104 Signal Salience Effect on Awareness of Task-Related Signals...... 105 Number of Concurrent Task Effect on Awareness of Task-Related Signals ...... 106 Interaction of number of tasks against the Salience and Expectancy Factors ...... 107 Predicting the Awareness of Task-Related Signals with Priority, Expectancy, Salience and the Number of Concurrent Tasks ...... 109 Summary of the Result of RQ-2 ...... 114

TABLE OF CONTENTS (Continued) Page RESULTS OF RQ - 3 ...... 115 The Perceived task priority and Executed Task at the Simulation Freeze Moment ...... 115 The Perceived task priority and Task Performance ...... 118 Summary of the Result of RQ-3 ...... 122

SUMMARY OF MAIN FINDINGS ...... 124

CHAPTER 5: DISCUSSION ...... 127

WHY DID PARTICIPANTS NOT ALWAYS NOTICE THE PROBLEM SIGNALS EVEN WITH HIGH TASK PRIORITY ...... 128 Cockpit Display Design and Visual Sampling Frequency ...... 128 Concurrent Multitasking Weakened Situation Awareness ...... 132 Different Visual Scanning Pattern ...... 136 Required Situation Awareness Level for Noticing Instrument Malfunction Signals143 Interrupting Tasks ...... 145

WHY DID PARTICIPANTS NOT ALWAYS EXECUTE THE PRIORITIZED TASKS? ...... 147 Top-Down Habitual Task Priority ...... 147 Forgetfulness and Prospective Memory ...... 151

WHY AREN’T THE PERCEIVED TASK PEROFRMANCE STATUS AND WORKLOAD RELATED TO THE PERCEIVED TASK PRIORITY?...... 153 INTEGRATED MODEL OF TASK PRIORITIZATION BEHAIVOR ...... 155 ASSUMPTIONS AND LIMITATIONS OF THIS STUDY ...... 163 FUTURE STUDY ...... 165

TABLE OF CONTENTS (Continued) Page CHAPTER 6: CONCLUSIONS AND RECOMMENDATIONS ...... 166 CONCLUSIONS ...... 166 RECOMMENDATIONS ...... 167

BIBLIOGRAPHY ...... 169

APPENDICEIS………….……………………..…………………..…………………. 182 APPENDIX A: GLOSSARY...... 183 APPENDIX B: INFORMED CONSENT DOCUMENT FOR EXPERIMENT ...... 187 APPENDIX C: IDEF0 DIAGRAM FOR MODELING COCKPIT TASK MANAGEMENT ...... 192 APPENDIX D: EXPLANATION OF FLIGHT SIMULATION TO PARTICIPANTS ...... 198 APPENDIX E: ATC COMMUNICATION SCRIPT ...... 209 APPENDIX F: QUESTIONNAIRE ...... 215 APPENDIX G: CODING VARIABLES AND HYPOTHESIS TESTING METHODOLOGIES ...... 228 APPENDIX H: PARTICIPANTS RESPONSES TO TASK PRIORITY QUESTIONNAIRES ...... 246 APPENDIX I: ADDITIONAL ANALYSIS ...... 249

LIST OF FIGURES Figure Page Figure 2.1 Funk (1991)’s Cockpit Task Management Model ...... 5 Figure 2.2 An example of task prioritization behavior in SA stage...... 10 Figure 2.3 An example of wrong task prioritization in the response selection stage: Working Memory, Time Constraint, Rare Situation (skewed hypothesis distribution) may lead to the wrong prioritization of tasks ...... 15 Figure 2.4 An example of task prioritization behavior in the response selection stage: When executive function cannot inhibit interrupting stimuli, wrong task prioritization may occur ...... 16 Figure 2.5 An example of task prioritization behavior: Sluggish movement & slow response ends in failure of task prioritization (time-out problem) ...... 19 Figure 2.6 Task Status in CTM (Funk, 1991) ...... 23 Figure 2.7 Example of task prioritization behavior in task execution stage ...... 24 Figure 2.8 IDEF0 Box format ...... 35 Figure 2.9 The first Level of IDEF0 Hierarchy ...... 36 Figure 2.10 The Second Level of IDEF0 Hierarchy ...... 38 Figure 2.11 The Third level of IDEF0 Hierarchy at “Manage Tasks and Allocation” component based on CTM (Funk, 1991) ...... 39 Figure 2.12 Mapping of Three Research Questions on the Three Stage Model ...... 43 Figure 3.1 X-plane flight simulator ...... 45 Figure 3.2 Flight Path Used in the Flight Simulator ...... 47 Figure 3.3 Coding the urgency in the model...... 53 Figure 3.4 Coding of the Perceived Task Performance Status ...... 54 Figure 3.5 Airspeed indicator, VOR indicator, and vacuum pump indicators were ...... 57 Figure 3.6 The number of tasks while pilot did not notice a task-related signal...... 61 Figure 4.1 Scatter plots of perceived task priority (Y-axis) and importance for each task (X-axis) ...... 70 Figure 4.2 Scatter plot of perceived task priority (Y-axis) and reported buffer time ..... 72 Figure 4.3 Scatter plot of the perceived priorities (Y axis) and task performance status (X axis) for each task ...... 74 Figure 4.4 Scatter plots of the perceived task priority (Y axis) and perceived salience (X axis) for each task ...... 76

LIST OF FIGURES (Continued) Figure Page Figure 4.5 Scatter plots of the perceived workload (X axis) and priority (Y axis) for each task ...... 78 Figure 4.6 Biplot of the Perceived Task Importance Score ...... 80 Figure 4.7 Biplot of the Reported Task Buffer Time...... 81 Figure 4.8 Self-Evaluated Performances of Aviate and Navigate tasks in two dimensional plots of reported buffer time (i.e., urgency) in X-axis and perceived task priority on Y-axis ...... 83 Figure 4.9 Aviate Task and Navigate Task Priorities in VFR/IFR Participants ...... 85 Figure 4.10 Communicate and Manage Systems Task Priorities in VFR/IFR participants ...... 86 Figure 4.11 Individual Difference of the perceived task priority score at each task ...... 88 Figure 4.12 Difference of the perceived task priority score at each situation ...... 90 Figure 4.13 Relationship Between Aviate Task Priority and Aviate Task Importance at Each of the Eight Different Flight Simulation Situations ...... 91 Figure 4.14 Relationship Between Aviate Task Priority and Aviate Task Importance from Participant Perspective ...... 92 Figure 4.15 Bar plot of relative weights for the perceived Aviate task priority score .... 94 Figure 4.16 Bar plot of relative weights for the perceived Navigate task priority score 94 Figure 4.17 Bar plot of relative weights for the perceived Communicate task priority score ...... 95 Figure 4.18 Bar plot of relative weights for the perceived Manage Systems task priority score ...... 95 Figure 4.19 Relationship Between the Unnoticed Time Period of Cockpit Instrument Malfunction Signals (Y-Axis) and Perceived task priority (X-Axis) ...... 103 Figure 4.20 Relationship Between the Unnoticed Time Period of Cockpit Instrument Malfunction Signals (Y-Axis) and Signal Expectancy Condition (X-Axis) ...... 104 Figure 4.21 Relationship Between the Unnoticed Time Period of Cockpit Instrument Malfunction Signals (Y-Axis) and Signal Salience Condition (X-Axis).. 105 Figure 4.22 Relationship Between the Unnoticed Time Period of Cockpit Instrument Malfunction Signals (Y-Axis) and Observed Workload (X-Axis) ...... 106

LIST OF FIGURES (Continued) Figure Page Figure 4.23 The relation between the Log(Unnoticed Time) and the mean number of tasks in four situations: 1. Neither Expected or Salient (black squares), 2. Expected Only, (orange triangles) 3. Salient Only (red circles), and 4. Expected & Salient (dark triangles) ...... 108 Figure 4.24 Comparing the model fitting performance between the training data (left) and the test data (right) ...... 112 Figure 4.25 A plot of prediction versus actual data of unnoticed time with Unit: log (Seconds)...... 113 Figure 4.26 Mean scores of the perceived task priorities at eight different flight situations ...... 116 Figure 4.27 Observed Frequency of Task Execution (Y-axis) When the Estimated First Priority (FP) was Aviate, Navigate, Communicate, and Manage Systems Tasks ...... 117 Figure 4.28 Normalized Data of Observed Executing Tasks at the Frozen Moments . 118 Figure 4.29 Mean Altitude Deviation as the Aviate Task Performance (Left), and Mean Directional Deviation as the Navigate Task Performance Measures ...... 120 Figure 4.30 Perceived task priority Score When Checklist was Executed ...... 121 Figure 4.31 Three research questions and potential task prioritization factors at each stage ...... 124 Figure 5.1 Screen Shot of Analog Flight Instrument Panel of Cessna 172 RG ...... 129 Figure 5.2 Time Chart of Sequential Scanning in Analog Instrument Panel, and Concurrent Scanning in Integrated Digital Instrument Panel ...... 130 Figure 5.3 The relation between task Priority (Y-Axis) ...... 132 Figure 5.4 Negative slope trend of task priority with task urgency in four tasks ...... 133 Figure 5.5 Conceptual sketch of task priority and urgency relationships when multiple tasks have negative slopes simultaneously ...... 133 Figure 5.6 The effect of perceived Aviate task priority (Y-axis) and the mean number of tasks...... 135 Figure 5.7 Bubble Plot Proportional to the Unnoticed Time (Seconds) in Navigation Indicator Problem...... 137

LIST OF FIGURES (Continued) Figure Page Figure 5.8 Bubble Plot Proportional to the Unnoticed Time (Seconds) in Low Fuel Problem ...... 138 Figure 5.9 Bubble size Proportional to the Unnoticed Time (Seconds) for Vacuum Pump Indicator Problem ...... 139 Figure 5.10 Bubble Plot Proportional to the Unnoticed Time (Seconds) for Altimeter Problem ...... 140 Figure 5.11 Bubble Plot Proportional to Unnoticed Time (Seconds) in Attitude/Horizontal Indicator Problem ...... 142 Figure 5.12 Task Prioritization in Bayesian Manner ...... 149 Figure 5.13 The relationship between the proportion of failure of checklist task execution and the mean number of concurrent multitask ...... 152 Figure 5.14 Task Prioritization Behavior Model ...... 156

LIST OF TABLES

Table Page Table 2.1 Potential Task Prioritization Factors from Colvin et al., (2005) and supporting literatures from cognitive science ...... 34 Table 3.1 Tasks at each Situation ...... 48 Table 3.2 Definition of Priority and Five Factors ...... 50 Table 3.3 Probe questions used at the moment the simulation was frozen...... 52 Table 3.4 Challenges to participants ...... 59 Table 3.5 Data Coding Methodology for Measuring the Awareness of Signals ...... 63 Table 3.6 Collected data for Task Execution Measurement ...... 65 Table 3.7 Collected data for Task Performance Measurement ...... 66 Table 4.1 P-values of five potential factors for four tasks ...... 100 Table 5.1 Summary of Awareness of task-related signals Problems ...... 144

Modeling Task Prioritization Behaviors in a Time-Pressured Multitasking Environment

1. CHAPTER 1: INTRODUCTION

Human multitasking in transportation can be dangerous. A recent National Safety

Council (NSC) white paper (2010) reported that 25% of all car crashes were caused by the use of a mobile device while driving, and an estimated 1.4 million crashes and

645,000 injuries were related to multitasking. Relatedly, many aviation incidents and accidents are reported to occur during multitasking. Chou, Madhavan, and Funk (1996) reviewed National Transportation Safety Board (NTSB) aircraft accident reports and

NASA Aviation Safety Reporting System incident reports using cockpit task management (CTM) theory (Funk, 1991); they reported that 23% of aviation accidents and 49% of aviation incidents were rooted in CTM errors.

One reason for dangerous multitasking in transportation operations might be the difficulty of task prioritization during multitasking. A well-known example used in aviation studies is the crash of Eastern Air Lines Flight 401 on December 29, 1972 in the

Florida Everglades. When the airplane was approaching the Miami airport, the pilots noticed that a landing gear indicator light did not turn on. The pilots communicated with the airport approach controller who gave a clearance to maintain an altitude of 2000 feet and hold west over the Everglades. The cockpit crew proceeded to put the plane in autopilot mode and believed that the airplane was holding at an altitude of 2000 feet.

However, the pilots did not notice that the autopilot setting had changed (probably due to

2 a pilot mistakenly moving the yoke) and the airplane gradually descended towards the ground (NTSB, 1973). The root cause of the accident can be interpreted as a task prioritization error because the pilots wrongfully prioritized their attention to the landing gear problem instead of controlling the airplane; the pilots did not pay attention to the altitude because they focused too much on the landing gear problem (NTSB, 1973).

This accident raises several questions: Why did the pilots prioritize diagnosing the landing gear problem higher than controlling the airplane? What is the mechanism behind task prioritization behavior in aviation multitasking environments?

It was hypothesized that the aviation task prioritization process is influenced by five factors: task importance, task urgency, task status, task salience, and multitasking workload based on the antecedent study conducted by Colvin, Funk and Braune (2005).

The objective of this research was to test the above hypotheses. This dissertation consists of the following chapters. In Chapter 2, human task prioritization studies in multitasking are reviewed with research questions. In Chapter 3, the proposed methodology is described. In Chapter 4, the main findings are presented. In Chapter 5, the main findings are discussed. In Chapter 6, a summary and recommendation for future research are presented.

2. CHAPTER 2: LITERATURE REVIEW

In this chapter, current knowledge about multitasking and task prioritization is reviewed. The first section provides an overview of Funk’s (1991) CTM normative theory for ideal task management. The second section reviews cognitive research about task management. The third section reviews critical task prioritization factors in the aviation domain. Finally, the fourth section summarizes the literature and presents the research opportunity.

COCKPIT TASK MANAGEMENT (CTM) THEORY

How should pilots manage multiple tasks in time-pressured and dynamic situations? How should pilots prioritize tasks? Using a systems engineering approach,

Funk (1991) developed Cockpit Task Management (CTM) theory as “the process by which the flight crew manages an agenda of cockpit tasks” (p. 277). CTM consists of two major components: agenda management and cognitive processing (Figure 2.1).

In agenda management, a pilot creates an initial agenda that consists of a hierarchical structure of tasks for the purpose of achieving the mission goal. The mission goal is decomposed into sub-goals (sub-tasks) that may be decomposed further until the tasks are small enough to manage. Each task has a priority, and generally, a

4 more important or urgent task has a higher priority. It should be noted that the priority of each task may change dynamically.

In CTM the following procedures are conducted, as shown in Figure 2.1. The first process is to assess the current situation (CTM 2.a). The second process is to assess the status of active tasks (CTM 2.b). The third process is to assess task resource requirements (CTM 2.e). The fourth process is to make a response selection by prioritizing tasks (CTM 2.f). Funk (1991) defined task prioritization as a pairwise comparison of tasks based on the importance, urgency, and goals of the tasks. The fifth process is to make a response execution. Based on the determined priority, the pilot will conduct one of the following: activating a task (CTM 2.b), terminating a task (CTM 2.d), or allocating resources to a task (CTM 2.g).

Funk’s (1991) CTM theory is structurally consistent with cognitive models, such as Wickens, Hollands, Banbury, and Parasuraman (2013)’s human information processing stage model and Endsley’s (1995a) situation awareness (SA) model. For example, CTM theory has situation awareness stages (CTM 2.a, 2.c, and 2.e) that correspond to Level-2 and Level-3 in the SA model. On the other hand, while the resource scope in CTM theory includes human resource (pilots) and equipment resources

(autopilots, radio, displays and controls), the scope of SA theory and other human cognitive theories focus only on human sensory cognitive and motor resources.

Figure 2.1 Funk (1991)’s Cockpit Task Management Model

Bishara and Funk (2002) conducted flight simulation experiments and confirmed the effectiveness of CTM training for better task prioritization performance. Participants in the CTM training group outperformed the control group in task prioritization error rates and prospective memory error rates.

What factors cause CTM errors? Based on a CTM taxonomy, Chou et al. (1996) reported that at least two types of CTM errors (task initiation errors and task prioritization errors) occurred when the amount of required cognitive resources was large and the number of concurrent tasks and task difficulty (flight path complexity) were high. Chou et al. (1996) concluded that high workload generated CTM errors and proposed the necessity for pilots to develop strategies to predict and handle high workload situations.

Wilson (1998) and Funk, Wilson, Vint, Niemczyk, Suroteguh and Owen (1999) reported that the automation of pilots’ tasks might increase task prioritization errors. For example,

6 inappropriately designed automation may make it difficult for pilots to detect, diagnose and evaluate the consequence of automation failures.

Then what is the limit of people’s ability to compute task priority? Shakeri and

Funk (2007) tested how people can calculate the tradeoffs among CTM’s task prioritization criteria (importance, urgency and status of tasks) in multitasking with a juggler’s paradigm. A participant monitored six hypothetical tasks on a computer screen. The status and urgency of each task were displayed with bars similar to a battery icon charge level bar. Each task had a different level of importance, urgency, and status to be taken into consideration for task prioritization. Shakeri and Funk (2007) reported four main findings. First, participants were not able to achieve perfect task- prioritization (they scored 71% to 87% in task prioritization performance compared to a near optimal task-prioritization). Second, participants were more aware of the importance of tasks that were static and displayed on the screen and failed to recognize the dynamically changing urgency or status of tasks that required mental computation and prediction. Third, participants overemphasized the penalty score of the task-prioritization decision, which indicated the difficulty participants had ignoring the salient task; and fourth, participants “learned” which tasks should be more prioritized than others

(strategic task management).

COGNITIVE VIEW OF HUMAN TASK PRIORITIZATION BEHAIVOR

What is the cognitive view of human task prioritization behaviors? Is it the way that pilots actually behave in task management as opposed to how they should manage task prioritization? Cognitive factors are reviewed that may affect task prioritization from three CTM main stages: the situation awareness stage (CTM 2.a, 2.c, and 2.e), the response selection stage (CTM 2.f), and the response execution stage (CTM 2.b, 2.d, and

2.g). Finally, workload and strategic control of tasks is reviewed for the agenda management component (CTM 1).

Task Prioritization Factors in the Situation Awareness Stage

The situation awareness stage is the foundation for making any decision that relates to task prioritizations. Similar to CTM (1991), Endsley (1995a) proposed three sub-stages in human cognitive situation awareness: perception of elements in the current situation (Level-1 SA), comprehension of the current situation (Level-2 SA), and projection of future status (Level-3 SA). Endsley assumed that people hold memories of prototypical situations as coherent frameworks for understanding information (i.e., schemata) and sequences of appropriate actions (i.e., scripts, Schank, 1980; Abelson,

1981) in their long-term memory. If a person thinks that a given situation matches a pattern of prototypical situations (schemata, Bartlett, 1995) in their long-term memory, a recognition-primed decision (e.g., Klein, 1993) may occur. Endsley (1995a) claimed

8 that a person with a well-developed mental model knows the relevant elements of the system that require attention and knows how to integrate the elements to understand the meaning of the current situation (Level-2 SA) and the meaning of the predicted future status (Level-3 SA).

Gugerty (2011) claimed that three types of thinking mechanisms are engaged in visual situation awareness: automatic thinking, heuristics or recognition-primed type of thinking, and conscious and controlled thinking. The first type of cognitive process is immediate automatic thinking (Shiffrin & Schneider, 1977) that places almost no burden on cognitive resources and is used in ambient vision for tracking fuzzy images. For example, experienced drivers can be unconsciously aware of the changing shape of a road to control speed and direction in an automatic manner. The second type of cognitive process is recognition-primed thinking (Klein, 1993) that takes less than one second and places little demand on cognitive resources. For example, a driver recognizing a red light signal will decide to stop the car. The third type of cognitive process is conscious and controlled thinking that demands cognitive resources. For example, navigational decisions in unfamiliar locations place a heavy cognitive burden on the controlled cognitive thinking process.

What then are the factors that affect the situation awareness of pilots? Level-1

(perception of elements in the current situation) and Level-2 SA (comprehension of the current situation) can be deteriorated with “blindness”; a pilot may not notice important stimuli from the external environment, such as an altimeter indication that the current altitude has deviated from the cleared altitude. Inattentional blindness may occur when a pilot’s attention is diverted to another task. Strayer, Drews, and Johnston (2003)

9 conducted high-fidelity driving simulation studies and reported that cellular phone usage affected explicit recognition memory for roadside billboards, and eye-tracking data showed this to be because of reduced attention to foveal information. In the famous

“invisible gorilla” experiments, Simons and Chabris (1999) showed that people do not notice an object not only because the stimuli are unclear but also because an event is unexpected or the primary task is difficult (i.e., attention requiring). It may be difficult to prevent cognitive blindness because paying attention to the object is not enough; further careful encoding of the changed target is required (Simons and Levin, 1998).

Simons and Levin (1998) hypothesized that change blindness may occur because people often categorize the obtained stimuli information quickly and with only minimal attention to details most of the time. Changes are more likely to be noticed if the target object is located in the center of interest with salient stimuli; yet even still, change blindness may occur (Simons and Chabris, 1999). Inattentional/change blindness can be mitigated if the person has the right expectation (Simons and Levin, 1998).

Finally, Level-3 SA (projection of future status) is innately difficult to achieve.

Fennema and Kleinmuntz (1995) reported that even experienced people might have difficulty anticipating the required effort and correctness of a chosen task.

Endsley (1995a) claimed that the key to good SA is the ability to characterize the given situation and the ability to determine the similar prototypical situation quickly and correctly. Endsley suggested that a pilot should have many prototypical situation models (i.e., case studies) in his/her long-term memory; if a pilot does not have an appropriate prototypical model for the current situation they will often fail to resolve a new problem on time. Reason (1990) pointed out that the innate risk of choosing the

10 wrong schema is because “a schema only contains evidence of how a particular recollection or sensory input should appear. It has no representation of what it should not look like.” (Reason, 1990, P36). Second, Endsley (1995a) claimed that the way information is presented in the cockpit is very important for SA because different problem framing prompts different information integration (situation comprehension), which will lead to choosing different mental models to solve the problem. Third, Endsley

(1995a) noted that meta-awareness of SA improved the quality of the decision. For example, if a novice pilot realizes his/her lack of SA and modifies his/her behavior it is possible to reduce the probability of poor performance.

To summarize the role of situation awareness, any situation awareness problem

(e.g., inattentional blindness, change blindness, wrong mental models, and etc.) can be the foundation for flawed task prioritization behavior. Figure 2.2 shows the Flight 401 accident in Miami example, where unnoticed low altitude with inattention/change blindness led the pilots to prioritize the trouble-shooting task of the front gear problem over the aviation task to maintain the aircraft’s altitude (NTSB 1973).

Figure 2.2 An example of task prioritization behavior in SA stage.

Task Prioritization Factors in the Response Selection Stage

The response selection stage is where a pilot makes a task-prioritization decision based on updated situation awareness. In this study, the “priority” is defined as the pilot’s intention (rightly or wrongly) of allocating his or her cognitive resource to the intended focal task(s) at specific time t ; a pilot makes a decision to choose the next task (i.e., prioritized task) or to keep attending the focal task (i.e., keep the current prioritized task) at the response selection stage (CTM 2.f).

When two or more tasks should be executed simultaneously (i.e., concurrent multitasking) task prioritization is the decision to allocate more cognitive resources to tasks that meet the subjective target performance criteria used for the problem domain

(Wickens, Kramer,Vanasse, and Donchin 1983).

How then does a pilot calculate the tradeoff between tasks A and B? What are the factors that affect the effectiveness and efficiency of this decision? Multi-attribute utility theory (MAUT) has been regarded as normative for decision making by considering the value (i.e., importance) and cost (e.g., required effort ) of tasks (e.g., Clemen, 1996;

Wallenius, Dyer, Fishburn, Steuer, Zionts, and Deb, 2008). MAUT can be executed with controlled and effortful thinking or System-2 thinking (Kahneman, 2003) when people consciously conduct an effortful consideration of alternative choices, criteria, and plausible explanations of the given situation (Allport, Styles, and Hsieh, 1994).

What factors may affect prioritization decisions? Wickens, Goh, Helleberg,

Horrey, & Talleur, (2003) modeled visual attention decisions with four factors: salience,

12 effort, expectancy, and value of tasks (the SEEV model). Similar to the SEEV model,

Kushleyeva, Salvucci, and Lee (2005) developed a task prioritization model that computes the priorities of each task with an expected utility gain, represented as PG-C.

Here, P, G, and C represent the probability that selecting a task will lead to a successful completion of the global goal, the subjective value of the Goal, and the expected Cost of completing the goal if a task is chosen.

In some cases, a task prioritization decision is attention-requiring, involving careful calculation, while in other cases, an automatic decision is based on past experience. When people do not have enough time or enough cognitive resources (e.g., available working memory), they use heuristics as a “shortcut” for task prioritization decision making to minimize the associated resource consumption while keeping the required performance in multitasking (e.g., Allport, Styles, and Hsieh, 1994; Fennema &

Kleinmunts, 1996; Simon 1979). Task prioritizations can be conducted automatically for routine work. When a pilot encounters a particular stimulus it may evoke the performance of tasks that are habitually associated with it (Shiffrin and Schneider, 1977).

Many studies have shown that people make decisions by considering both stimuli and experience in a Bayesian manner (e.g., Anderson, 1998; Chater and Oaksford, 1990;

Wallsten & Barton, 1982). Wallsten and Barton (1982) reported that people first process the stimulus information sequentially from the most salient to the least salient before considering the importance of the criteria associated with the stimuli in a Bayesian manner. Thus, in cases where more salient information is also more diagnostic, people could make appropriate decisions; but in cases where less salient criteria are diagnostically the most useful, people have difficulty making correct decisions. The

13 number of criteria for decision-making changes systematically with payoffs, costs, and available time. Little is certain about which criteria are used for task prioritization in aviation multitasking.

What are the factors that affect task prioritization decision in the response selection stage? Dougherty and Hunter (2003 a, b) reported that people often make wrong decisions when only a small number of hypotheses about the problem are generated due to three factors: working memory capacity, time constraints, and the shape of the hypothesis distribution. For example, Mehle (1982) reported that auto mechanics were able to generate only four to six hypotheses about why a car would not work.

Dougherty and Hunter (2003 a, b) hypothesized this mechanism as people making judgments by comparing the focal hypothesis with alternative hypotheses, and that people with low- working memory capacity generate fewer alternative hypotheses than people with high- working memory capacity. Dougherty and Hunter (2003 a, b) gave the following example to explain the above mechanism. Suppose a person wanted to make a probability judgment of it raining tomorrow by comparing the focal hypothesis (rain) with alternative hypotheses. Dougherty and Hunter (2003 b) assumed that people first generate alternative hypotheses for “not rain”. When one alternative hypothesis (e.g., sunny) is generated, the subjective probability judgment for rain is

MS(Rain) Pr(Rain | One _ alt _ hypo)  (1) MS(Rain)  MS(Sunny)

14 where, “MS” stands for the “memory strength” of the hypothesis. On the other hand, when three alternative hypotheses are generated the subjective probability judgment for rain is

MS(Rain) Pr(Rain | Three _ alt _ hypo)  (2) MS(Rain)  MS(Sunny)  MS(sleet)  MS(Snow)

Thus, when only one alternative hypothesis (sunny) is generated the focal hypothesis is compared with only two total sets of hypotheses, and the probability of the focal hypothesis is magnified.

Experiments by Dougherty and Hunter (2003 b) indicated three human cognitive problems (Figure 2.3). Their first main finding was that people with little working memory capacity made decisions with fewer alternative hypotheses. Dougherty and

Hunter (2003 b) reported that working memory capacity was a statistically significant factor for the number of generated hypotheses while “static” short-term memory capacity was not an important factor. This is consistent with many studies that have shown working memory to be an important predictor of multitasking performance (e.g.,

Baddeley, 2002, 2006; Konig, Buhner, and Murling, 2005; Hambrick, Oswald, Darowski,

Rench, and Brou, 2010).

In addition to working memory capacity, time constraints also affected the number of generated alternatives in decision-making. Dougherty and Hunter (2003 b) assume that time constraints “truncated” the process of recalling alternative hypothesis from long-term memory. Dougherty et al., (2003 b) also reported that participants tended

15 to ignore hypotheses that were less likely to happen (in case of “long-tailed” distribution) even when they could recall from their past experience.

Figure 2.3 An example of wrong task prioritization in the response selection stage: Working Memory, Time Constraint, Rare Situation (skewed hypothesis distribution) may lead to the wrong prioritization of tasks

The second factor of task prioritization decision behavior in the response selection stage is the compromised executive function in cognitive resource allocation (Figure 2.4).

The executive function plays an important role in multitasking: the role of task prioritization, sequencing behaviors, inhibiting inappropriate behaviors, thinking about what task is most relevant for the current purpose, resisting distractions or irrelevant information, switching between task goals, and handling novel information (Banich,

2009). The executive function is vital for interruption management because a pilot must have the power to inhibit irrelevant interruptions.

Figure 2.4 An example of task prioritization behavior in the response selection stage: When executive function cannot inhibit interrupting stimuli, wrong task prioritization may occur

In basic psychological science, the Stroop task demonstrates the role of the executive function for inhibiting inappropriate behaviors or interrupting tasks (e.g.,

Banich, 2009). In the Stroop task, a series of color words (red, blue, green, brown and purple) are printed in conflicting colors. For example, suppose the word “red” is printed in blue ink. When a participant is asked to name the color of the word, the participant might say the printed color (red) rather than the true color (blue) because reading the word is a more automatic process than carefully recognizing the true color of the word.

The third factor of task prioritization decision behavior is psychological stress.

Lipshitz and Strauss (1997) claim that when people sense uncertainty (i.e., a sense of doubt), the response of decisions will be delayed or decisions will be blocked, depending on the type of uncertainty.

The fourth factor of task prioritization decision behavior is perceived uncertainty.

Keinan (1987) reported that even without time constraints people were affected by stress; regardless if the stress was controllable or not, people did not consider all available alternatives, and their scanning pattern of alternative choices were in a nonsystematic fashion.

In the response selection stages, task prioritization using CTM factors

(importance, urgency and status of tasks) may be bounded by cognitive factors (working memory shortage, executive function burden, time-pressure, difficulty/novelty of the task, and psychological stress).

Task Prioritization Factors in the Response Execution Stage

The response execution stage is when a pilot executes the selected response with motor (e.g., manual control, speech, etc.). In CTM, the response execution stage corresponds to activating a task (CTM 2.b), terminating a task (CTM 2.d), or allocating resources to a task (CTM 2.g).

What are the factors that affect task prioritizations in the response execution stage? The first factor is the slow response time that leads to “time-out” problems because multitasking often generates an interference of resources. For example, multitasking pilots may run out of time for prioritizing tasks or fail to execute tasks

(Figure 2.5). In the case of Eastern Airlines Flight 401, the pilots were late in checking the altitude for at least 4 minutes (NTSB, 1973). Even if the pilots had the correct task prioritization intention of checking the altitude, a late execution of an appropriate task might have generated this catastrophic accident. Similarly, Stanton and Baber (2008) studied the operator’s response time in a catastrophic rail accident that occurred in

England using the alarm initiated activity (AIA) and critical path analysis (CPA) methods and concluded the operator made all the correct decisions in task prioritization. Stanton et al. (2008) reported that the railroad accident was inevitable even though all of the correct tasks were prioritized correctly.

Figure 2.5 An example of task prioritization behavior: Sluggish movement & slow response ends in failure of task prioritization (time-out problem)

Generally speaking, some sets of tasks are difficult to execute concurrently as people suffer sluggish movements and slow response in concurrent multitasking scenarios. Strayer and Johnston (2003) showed that concurrent multitasking of cell- phone conversation and driving generated not only inattentional blindness but also sluggish movement and slower response in the driving task itself. Strayer, Watson, and

Drews (2011) conducted a driving simulation study in a car-following situation to test how much driving performance is impaired by conversations on a cellular phone. They measured the driver’s speed, following distance, and braking reaction time. The result showed clear evidence of sluggish behavior in multitasking drivers who talk on cellular phones while driving. Their braking reaction time was significantly slower, and they needed longer time to recover speed lost after braking. According to the study of a car- collision prediction model, sluggish braking reactions lead to a higher probability of fatal collisions (Lee, McGehee, Brown, & Reyes, 2002).

In concurrent multitasking, an interference of resource occurs not only because of motor interference (e.g., manual control of hands for typing and writing), but also from the interference of cognitive resources (e.g., Wickens, 2002a;Salvucci & Taatgen, 2010).

In basic psychological experiments, Pashler (1984, 1989) showed the response selection stage is the “bottleneck” that generates cognitive interference; only one task can be processed at a time.

Allport, Elizabeth, Styles, and Hsieh (1994) reported that it takes more switching time for people to switch to the dominant task A (i.e., an easy automatic task) from a non- dominant task B (i.e., a difficult, attention-requiring task). Many factors may affect the switching time. Allport et al. (1994) claimed that when simultaneously conducting a difficult (attention/resource-requiring) task and an easy task executive function needs to inhibit the easy task to better focus on the difficult task. However, the inertia of inhibiting the easy task persists for a long period of time (switching time). Thus, it takes more time to return to easy task A. Allport et al. (1994) showed this “proactive interference” in a wide variety of task pairs including Stroop-like switching experiments.

The study of Allport et al. (1994) indicated that accidents might happen even when the pilot intends to prioritize the right task (i.e., driving or aviating) at the emergent moment if they are engaged in the multitasking of attention-requiring secondary tasks.

Levy and Pashler (2008) conducted experiments that required participants to multitask: driving (Task A) and secondary tasks (Task B: reporting the number of sound signals). Every participant was instructed to ignore secondary tasks (Task B) and prioritize the braking task (Task A) in an emergency situation. However, most people

21 were unsuccessful in returning to braking (Task A) on time. The failure to abort Task B caused significant delays in braking.

Wickens and Kessel (1980) and Wickens (2002a, 2008) integrated past multitasking cognitive studies and developed a multiple resource theory to model what types of multitasking would generate interference. Wickens and Kessel (1980) indicated that tasks in the perceptual and recognition stages did not much affect another task that required response selection-execution stages. For example, when people are engaged in the concurrent multitasking of speech recognition (perception) and speech production

(response), the used resource for these two tasks is separated even though speech recognition and production share the same verbal resources.

Wickens’s (2002a) multiple resource theory predicts the task interference levels between two tasks. The multiple resource model consists of three cognitive stages: perception, cognitive, and response. For example, if Task B requires audio and visual factors (AV) there is a 0.8 level of task interference (very high interference) with Task A that requires audio and Visual (AV). On the other hand, if task A and B both involve response and visual factors (RV) then the degree of interference is 1.0 (impossible to execute concurrently).

Such high cognitive and motor resource interference might make people transit to sequential multitasking from concurrent multitasking (e.g., Salvucci, Taatgen 2010;

Salvucci, Taatgen, Borst, 2009; Wickens, et al., 2013). Salvucci and Taatgen (2010) summarized the interference magnitudes for each cognitive component and insisted that problem-state memory interference (temporary information needed during the execution of task, e.g., an operator needs to keep remembering the information needed for the task) is much higher than other interferences. Salvucci and Taatgen claimed that such interference reaches the saturation point where the operator cannot make adequate progress in multitasking. Then, he or she would suspend one or more tasks for later resumption when able to resume the task with less interference. Task suspension also occurs when people are engaged in hierarchical problem solving (e.g., means-end analysis) or when environmental constraints (e.g., unexpected interruptions) stop the progress of tasks, which requires people to perform multitasking sequentially (e.g.,

Altmann & Trafton, 2002; Salvucci, Taatgen, Borst, 2009).

In Funk’s (1991) CTM theory, this transition is depicted as the CTM task status

(Figure 2.6). After the task is initiated it can be either in active or non-active status, which some scholars (e.g., Salvucci, Taatgen, 2010; Salvucci, Taatgen, and Borst, 2009) call “sequential multitasking”.

Figure 2.6 Task Status in CTM (Funk, 1991) Adapted from “Cockpit Task Management: Preliminary definitions, normative theory, error taxonomy, and design recommendations.” By Funk (1991), The International Journal of Aviation Psychology, 1(4), p.275

The second factor for task prioritization behavior in the response execution stage is the omission of tasks or forgetfulness of the suspended tasks (Figure 2.7) in sequential multitasking. Occasionally, people forget to execute suspended tasks even if those tasks should have a high priority. In 1987, Northwest Airlines Flight 255 crashed in Detroit because the pilots forgot to execute the checklist item regarding flaps during taxiing

(NTSB, 1987). Similarly, in 2008, Spanair Flight 5022 crashed in Madrid because the pilots omitted the taxiing checklist item for flaps (CIDEA, 2015). Memory for goal theory indicates that the suspension and resumption of tasks may generate forgetfulness in task execution (Altmann & Trafton, 2002). In the Flight 255 accident, the pilots suspended the checklist task to respond to Air-Traffic Controller (ATC) communication.

When the pilots resumed the checklist, they forgot to execute the task of extending flaps

(NTSB, 1987).

Figure 2.7 Example of task prioritization behavior in task execution stage

Altmann and Trafton (2002) claimed that people might have difficulty recalling the suspended goals/tasks due to the following three cognitive mechanisms. The first mechanism is the decay of memory activation level with time; the suspended tasks will be forgotten gradually. The second mechanism is memory interference between suspended old and new goals (or tasks); people have a hard time recalling the goals/tasks that were suspended at some time in the past because the activation level of the old task is interfered with by the new suspended task. The third mechanism is situational cues to recall the suspended tasks. Suspended old tasks can be recalled when associated cues are noticed. It is assumed that people use the Bayesian approach for cue interpretation

(e.g., Anderson, 1998; Chater & Oaksford, 1999). If people notice the cue and integrate their prior experience with current evidence of the cue appropriately then people can recall suspended tasks. However, the cue may also become an interrupting or distracting factor that may activate irrelevant tasks.

To summarize the task prioritization factors in the response execution stage, task prioritization behavior may be affected by forgetfulness (e.g., slips and lapses; Reason,

1990) and slow task execution.

Workload and Strategic Control for Task Prioritization

Heavy workload may affect multitasking performance and task prioritization decision quality. Ma and Kaber (2005) developed an operational definition of SA in the driving domain based on the Endsley’s (1995a) SA theory and tested how much adaptive cruise control (ACC) and cell phone conversation affected SA when driving. Ma et al.,

(2005) showed cell phone conversation increased the driver’s workload and impaired diving SA. On the other hand, ACC relieved the workload of vehicle monitoring and motor control and more attention was paid to driving. Ma et al., (2005) hypothesized

ACC helped drivers develop more accurate and complete knowledge of driving SA.

When the workload approaches a cognitive limitation for multitasking, how do people change their task prioritization behaviors? Robert and Hockey (1997) proposed compensatory control theory to explain how people prioritize tasks under stress and in high workload environments.

In Robert and Hockey’s (1997) theory, first the long-term and short-term goal determines the target status. Here, the target state may change after considering the pilot’s perceived costs and benefits associated with the actions. The control feedback system compares the target state and actual output and controls the allocation of resources. There are two types of control feedback loops: an automatic control loop A, and an attention demanding control loop B. While automatic loop A will control the gap when the discrepancy between the target status and actual status is small, a large discrepancy requires attention-demanding control loop B, which involves cost-benefit decision-making. Robert and Hockey (1997) proposed a two-level control model that

26 uses two thresholds levels of task performance discrepancy. The lower level is the threshold when a pilot starts feeling the necessity of further effort (i.e., allocating more mental resource to the problem task) until the upper threshold level. Robert and Hockey

(1997) claims that a pilot will conduct “active control” between the lower and high threshold level that involves increased executive control or working memory capacity and the use of rule-based or knowledge-based responses.

In case the performance discrepancy is too great and it exceeds the upper level of the threshold, Robert and Hockey (1997) claims that people would use one of these two strategies: the strain coping mode, or the passive coping mode. In the strain coping mode a pilot exerts the maximum effort needed to achieve the target performance criteria for the targeted task by allocating more resources (i.e., effort) that involves distress for a brief period of time. In the passive coping mode, the pilot lowers the upper target threshold for protecting the system from disruption of task performance, which may involve the reduction of required levels of speed and accuracy.

Wickens et al., (2013) discussed other possible strategies for task management that a pilot may use. The first strategy option is to allow slight performance degradation. For example, a multitasking driver may feel fine if the center of a car deviates from the center between traffic lanes when the workload of using a GPS device increases. The second strategy option is to use more efficient and less resource consuming procedures. For example, people may stop using time-consuming optimal methods for decision-making and adopt heuristic methods that provide satisfactory results in a timely manner. The third strategy is to shed the low-priority tasks and focus on high- priority tasks. For example, a driver may stop talking on a cellular phone while

27 driving and instead focus on driving. In this case, it is possible that people may choose to focus on using their cellular phone and stop paying attention to driving their cars.

The fourth strategy is to perform tasks in a non-optimal manner; for example, prioritizing cellular phone conversations over safe driving. However, very little is known about the strategy that people actually choose (Wickens et al., 2013).

Summary of Task Prioritization Factors at the Cognitive Level

In summation, task prioritization factors exist not only in the decision-making response selection stage, but also at each of the three information processing stages: situation awareness, response selection, and response execution. Endsley (1995a) noted

“Even the best-trained decision makers will make inappropriate decisions if they have inaccurate or incomplete SA [situation awareness]” (Endsley, 1995a, p36).

In the situation awareness stage, any inappropriate situation awareness may potentially become a factor of task prioritization. Pilots may fall into inattention or change blindness problems and lose situation awareness when salient stimuli are not presented or when they don’t have many prototypical situations in their long-term memory. The way information is presented in the cockpit is also important because framing problems differently prompts different information integration (situation comprehension), which will lead to choosing different mental models to solve the problem (Endsley, 1995a).

In the response selection stage, task prioritization is the decision making problem to allocate cognitive resources to tasks to meet the target performance with criterion used for the problem domain (e.g., Wickens et al., 1983). Thus, task prioritization may be affected by different sets of decision factors or biases (e.g., Kahneman, 2003).

Furthermore, because people make decision-judgments by comparing the focal hypothesis with alternative hypotheses, working memory capacity problems can distort task prioritization decisions (e.g., Mehle, 1982, Dougherty and Hunter 2003 a, b).

In the response execution stage, interference of cognitive and motor resources caused by multitasking can be so great that people may not be able to execute prioritized tasks accurately and on time, and severe cognitive or motor resource interferences make people suspend the ongoing task for later resumption. Altmann and Trafton (2002) reported that people are likely to forget to execute suspended tasks. Such behaviors may be interpreted as inappropriate task prioritization behaviors even when people intentionally prioritized those tasks.

CRITICAL TASK PRIORITIZATION FACTORS IN THE AVIATION DOMAIN

Among the many potential factors that affect task prioritization, what are the most relevant and useful ones for the rational design of the aircraft cockpit? Colvin, Funk and

Braune (2005) conducted a flight simulator study in which they asked pilot participants to report possible task prioritization factors and they summarized and hypothesized the six primal candidate factors.

The first factor participants reported in Colvin et al.’s study was the perceived salience of stimuli that relates to a task. Colvin et al., hypothesized that “the priority of a task is directly proportional to its salience”. As introduced before, Shakeri and Funk

(2007) reported that people could not ignore salient stimuli in task prioritization. In fact, when task-related stimuli were not salient, inattention /change blindness phenomena occured (Simons & Chabris, 1999; Simons and Levin 1998; Strayer, Drews & Johnston,

2003).

The second factor participants reported in Colvin et al.’s study was the expectation of a task. Colvin et al. (2005) hypothesized that “the priority of a task is directly proportional to its consistency with procedure or with other pilot expectations”

(Colvin et al., p335). Simons and Chabris (1999), and Simons and Levin (1998) showed that lacking the expectation of stimuli may generate inattention/change blindness.

Endsley (1995a) also argued that an expected mental model (i.e., expectation) affects situation awareness that will influence where attention is directed and how perceived information is interpreted in a top-down cognitive process. Thus, the above two factors

(salience and expectation factors) could be vital in signal perception stage in Endlsey’s

(1995a) situation awareness model. This in turn affect task prioritization because without perceiving important task-related signals, pilots would not prioritize the task appropriately (e.g., Flight 401 accident).

The third factor participants reported in Colvin et al.’s study was the perceived importance of a task. They hypothesized that “the priority of a task is directly proportional to its importance” (Colvin et al.2005, p334). In multi-attribute utility decision, people tradeoff tasks considering pluses (i.e., importance or value) against minuses (i.e., costs). For example, Kushleyeva, Dario, Salvucci, and Frank (2005) constructed a task-prioritization model that trades off a cost factor and the value factor multiplied by their probabilities. Wickens et al. (2003) and Funk (1991) also used importance / value factor for a visual attention prioritization model and the CTM theory, respectively.

The fourth factor reported by Colvin et al.’s participants was the perceived performance status of a task. Colvin et al. hypothesized that “the priority of a task is directly proportional to its importance” (p335). The perceived status of a task may influence the task prioritization decision, because understanding of the current status is situation awareness Level-2, which is a foundation of sound decision-making (Endsley,

1995a). The status of tasks (CTM 2.c) is used as a task prioritization factor in CTM theory (Funk, 1991). Wickens et al.,(2003) also noted that pilots need to have situation awareness of task status for sound decision-making along with spatial awareness and system awareness. Endsley (2000) claimed that the perceived goal (that can be measured by the perceived task importance) at the moment plays an important role to

32 understand the current situation. Thus, the perceived importance of task, and the perceived performance status of task factors could reflect the Level-2 situation awareness

(i.e., comprehension of current situation) in Endsley (1995a)’s model, which could be important in task prioritization decisions.

The fifth potential task prioritization factor is the perceived urgency of a task.

Colvin et al. hypothesized that “the priority of a task is directly proportional to its urgency” (p335). The perceived urgency of task may be defined as the buffer time, or time remaining until the deadline of the task (Wickens, et al., 2013), which reflects the projected Level-3 situation awareness (Endsley, 1995a). Inappropriate perception of task urgency may lead to fatal aviation accidents (e.g., In the Flight 401 accident case, the pilots did not notice the urgency of the task). The Threaded Cognition Multitasking model uses the task urgency factor for task prioritization (Salvucci & Taatgen, 2010;

Salvucci, Taatgen and Borst, 2009). Funk (1991) also used it as the task prioritization factor in CTM. Thus, the perceived urgency of task factor could be vital in Endley

(1995a)’s Level-3 situation awareness, which could be important in task prioritization decision.

The sixth potential factor is the perceived cost or effort of a task or its workload.

Colvin et al., (2005) stated that “the priority of a task is proportional to the time/effort required to perform it” (Colvin et al., 2005, p336). Chou et al. (1996) reported that high workload adversely affects multitasking performance. Furthermore, high workload or high switching costs of tasks (e.g., Allport et al.,1994) will delay the execution of prioritized tasks in time (e.g., Lee, McGehee, Brown, & Reyes, 2002). When people cannot make adequate progress in concurrent multitasking they may suspend one or more

33 tasks for later resumption (e.g., Altmann & Trafton, 2002; Salvucci, Taatgen & Borst,

2009). Altmann and Trafton (2002) reported that people tend to forget to recall, resume, and execute tasks in a suspended status. For example, the pilots of Spanair Flight 5022 forgot to complete a flaps checklist item while taxiing, resulting in its crash (CIDEA,

2015). Thus even high priority tasks may be forgotten and observers (e.g., accident investigators) may regard it as an “inappropriate task prioritization decision”. As mentioned before, outside observers (e.g., accident investigators) may potentially regard delayed execution as inappropriate task prioritization decisions in incidents or accidents

(e.g., Flight 401; NTSB, 1973). Thus, the time/effort required to perform it (i.e., workload) factor could affect actual task prioritization behavior against the pilot’s task prioritization intention.

As discussed, the above six factors proposed by Colvin et al. (2005) could be promising candidate factors because they are based on the pilot’s perceptions and they are also consistent with other cognitive studies (Table 2.1).

Table 2.1 Potential Task Prioritization Factors from Colvin et al., (2005) and supporting literatures from cognitive science

Colvin, et al. Related Literatures from Cognitive Science (2005)’s factors -If stimulus is not salient, “Inattentional/ change blindness” cognitive phenomena Perceived may occur (Simons & Chabris, 1999; Strayer, Drews, & Johnston, 2003; Simons Salience of and Levin, 1998) stimuli that -People cannot ignore salient stimuli in task prioritization (Shakeri and Funk, relates to 2007) tasks -Salient stimuli is one of the visual attention prioritization factors in SEEV model (Wickens, Helleberg, Horry, & Talleur, 2003) -Lack of the right expectation of stimuli that relate to tasks may generate “Inattention/Change blindness” cognitive phenomenon (Simons & Chabris, 1999; Strayer, Drews, & Johnston, 2003; Simons and Levin, 1998) Perceived -Expectation affects situation awareness that will influence where attention is Expectations directed and how perceived information is interpreted in a top-down cognitive of tasks process (Endsley, 1995a,1995b) -Projected future situation of the environment will become an expectation that affects the above points (Endsley, 1995a,1995b) -“Expectation” is one of the major factors in the SEEV visual attention prioritization model (Wickens, et al., 2003) -People use the importance (or value) criteria in multi-attribute utility theory (e.g., Perceived Kushleyeva et al.,2005) Importance of -Importance of tasks (or value criterion) often used as prioritization factor such as tasks visual attention models (e.g., Wickens et al., 2003) -Importance of tasks is a task prioritization factor in CTM (Funk, 1991) -Performance status of task is a task prioritization factor in CTM (Funk, 1991) Perceived -Understanding of the current status is critical in situation awareness (Endsley, Performance 1995a) Status of -Pilots need to have situation awareness of task status (Wickens, 2003) tasks -Assessing status of active tasks are critical component in CTM (Funk, 1991) -Perceived urgency of the task is the buffer time until the deadline (Wickens, et al., 2013), which reflects the projected situation awareness at Level-3 SA (Endsley, Perceived 1995a) Urgency of -Urgency of tasks is the task prioritization factor in CTM (Funk, 1991) tasks -Threaded cognition multitasking model uses urgency of task for task prioritization (Salvucci & Taatgen, 2010; Salvucci, et al., 2009) -High workload adversely affects the task prioritization (Chou et al., 1996) -Switching cost or high workload (e.g., Allport, 1994) may delay the execution of prioritized task on time (e.g., Lee, McGehee, Brown, & Reyes, 2002) Perceived -According to the memory for goal theory, people tend to forget to recall and Costs of tasks execute suspended tasks (Altmann & Trafton, 2002) (Workload) -According to Strategic control theories, people may change their task prioritization behavior such that providing more resource (i.e., make efforts) or lower the target threshold (e.g., Robert and Hockey, 1997; Wickens et al., 2013) -Effort (i.e., the cost or workload) factor is used for SEEV visual attention prioritization model (Wickens, Helleberg, Horry, & Talleur, 2003) -Cost (i.e., workload) factor is an important tradeoff criterion against value criterion (i.e., importance) in multivariate utility theory (e.g.,SEEV model, Wickens, 2002a)

In order to summarize the cognitive mechanism of the six factors provided by

Colvin et al (2005), an IDEF0 model of CTM was constructed. An IDEF0 (Icam

DEFinition for Function Modeling-0) model depicts task functions in process flow map using inputs, outputs, controls and mechanisms (National Institute of Standards and

Technology, 1993). Figure 2.8 shows one task functional unit labeled as a verb. If needed, this task can be further decomposed into subtasks in a hierarchical fashion.

Arrows entering the left side of the box are inputs, and arrows coming out from the right side of the box are outputs. Control arrows specify the required condition for the functions that enter the box from the top. The arrows connected to the bottom side of the box are mechanisms that support the function of the box.

Figure 2.8 IDEF0 Box format

Based on this explanation of an IDEF0 model, the hierarchical structure of

Cockpit Task Management (Funk, 1991) is depicted below. Figure 2.9 (called the A-0 diagram) shows the top level of the hierarchy for the ultimate function “Fly aircraft” based on Colvin et al’s (2005) factors. Here, the salience of stimuli related tasks (Factor

1) is depicted as an input to this function, and the predetermined aviation procedures

(Factor 5) and task complexity and workload (Factor 6) are depicted as control factors.

Figure 2.9 The first Level of IDEF0 Hierarchy

Figure 2.10 shows the second level of the IDEF0 hierarchy (called the “A0” diagram) with five sub-tasks: A1 task (Manage Tasks and Allocate Resource), A2 task

(Aviate), A3 task (Navigate), A4 task (Communicate), and A5 task (Manage Systems).

A2 to A4 tasks are generally recognized aviation subtasks (Aviate, Navigate,

Communicate, and Manage Systems, or “ANCS” for short), and A1 task allocates cognitive resources and motor resources to these tasks (Wickens 2003). It should be noted “”Manage Tasks and Allocate Attention” (Box A11) is also a task, meaning that it also competes for cognitive resources against the remaining four tasks (A2-A5).

What is the current best understanding of the task prioritization behavior?

Figure 2.11 depicts the third level of the IDEF0 hierarchy for the A1 task (Manage Tasks and Allocate Resource) based on the CTM (Funk, 1991) framework.

First, the “① salience of task” factor is sensed at Box A11. The perceived information is then used in three types of functions: “Assess Current situation” in Box

A12, “Assess Status of Active Tasks in Box A13, and “Forecast and Assess Task

Resource Requirement in Box A14.

“Assess Current Situation (CTM 2.a)” (Funk, 1991) integrates many perceived elements in the light of the goal/task (Endsley, 1995a) and outputs the understood current situation, which in turn may impact the ② importance of task. Thus, the ② importance of task factor may reflect the pilot’s current situation awareness. The ③ performance status of tasks factor may reflect the output of the “Assess Status of Active

Tasks (CTM 2.c)”. The ④ urgency of task factor may reflect the output of “Forecast and

Assess Task Resource Requirements (CTM 2.e)”.

The ⑤ expectation factor and ⑥ cost/workload factor connect to all five box functions (A11-A15). This means that factors ⑤ and ⑥ have interaction effects on each of the five functions. In the A11 box (sense and perceive elements), the ① salience of stimuli factor is perceived only when the ⑤ expectation factor is present.

Similarly, when the “⑥ cost/workload” factor affects the A15 box (prioritize active tasks), pilots may use “shortcuts” (or heuristics) by dropping any of the three criteria from ②Importance, ③ Urgency, and ④ Status of tasks.

Figure 2.10 The Second Level of IDEF0 Hierarchy

Figure 2.11 The Third level of IDEF0 Hierarchy at “Manage Tasks and Allocation” component based on CTM (Funk, 1991)

LITERATURE SUMMARY AND RESEARCH OPPORTUNITY

A medium fidelity flight simulator experiment was conducted. As shown above, these six factors are consistent with many theories that relate to human task prioritization and they reflect the factors Colvin et al.’s (2005) participants said they used to prioritize tasks. In the CTM model, Funk (1991) claimed that ideal aviation task prioritization should be based on three factors: importance of tasks, urgency of tasks, and performance status of tasks (Factors 2-4 in Table 2). However, CTM is a normative theory (i.e., ideal task management theory), which needs to be validated empirically. Salvucci and

Taatgen (2010) constructed the “threaded cognition” model of human multitasking cognition and claimed that “urgency of task” (Factor 4) should be used for task prioritization. However, their model is not compared with any empirical study in the aviation domain. Wickens et al. (2003) developed the SEEV visual attention model for prioritization of visual attention with similar factors: Salience, Expectation, Effort, and

Value (Colvin et al.’s (2005) factors 1, 6, 5, and 2, respectively). However, SEEV factors are used in a human “visual attention” prioritization model that may not be applicable to task prioritization.

In summation, six candidate task prioritization factors identified by Colvin et al.’s

(2005) participants seem to be consistent with the basic cognitive science literature: the perceived salience of stimuli related to tasks, perceived importance of tasks, perceived status of tasks, perceived urgency of tasks, expectation of tasks, and perceived costs of tasks. However, it is still unknown if these factors really do affect task prioritization in the cockpit.

RESEARCH QUESTIONS

Based on important insight from the above literature, the following research questions were raised about the task prioritization behavior in aviation human multitasking. Figure

2.12 maps each research question on the three stage model.

Research question-1 (RQ-1): Can perceived task priority be explained by the following five factors?

1.1 Perceived importance of task: Is the priority of a task directly proportional

to its Importance?

1.2 Perceived urgency of task: Is the priority of a task directly proportional to

its Urgency?

1.3 Perceived performance status of task: Is the priority of a task inversely

proportional to its Status?

1.4 Perceived salience of task: Is the priority of a task directly proportional to

its Salience?

1.5 Perceived workload of task: Is the priority of a task proportional to the

Time/Effort required to perform it?

Research question-2 (RQ-2): Is there any relationship between the perceived task priority and the chance of noticing task-related cockpit instrument malfunction signals?

If so, how much does the perceived task priority affect the chance of noticing task-related signals considering the following factors?

2.1 Salience of task-related signals

2.2 Expectancy of task-related signals

2.3 The number of concurrent tasks

Research question-3 (RQ-3): Can actual task execution and task performance be explained by the perceived task priority?

Figure 2.12 Mapping of Three Research Questions on the Three Stage Model

3. CHAPTER 3: RESEARCH METHODOLOGY

A medium fidelity flight simulator experiment was conducted. In order to answer the first research question (RQ-1), questionnaires were used to estimate task priority as well as task importance, task urgency, task performance status, task salience, and the task workload in a flight simulation experiment. In order to answer the second research question (RQ-2), the participants’ awareness of task-related signals and task execution data were collected by questionnaires, flight recorder data, and camcorder data. In order to answer the third research question (RQ-3), the perceived task priority was compared with the actual task execution and task performance observed in flight recorder data and video recordings.

PARTICIPANTS

Sixteen pilot participants (15 males and 1 female) were recruited for the simulation experiments. This is because a statistical power analysis (for testing with

Power=0.8, α=0.05 on regression models) estimated that at least 128 simulation data points were needed. Since each participant experienced eight situations, sixteen people were needed (128 8 16 ). All the participants already possessed a private pilot’s license with an average of 4,508 hours of total flying hours (minimum 65 hours, maximum 31,000 hours) with average 2,274 hours of single pilot hours. Their ages were between 24 and 82 and mean age was with a 49.3. Each simulation experiment lasted

45 for 2 hours and each participant was compensated with a $25 gift card. The experiment was conducted from September 1st, 2014 through September 29th, 2014 in the Human

Factors Engineering laboratory at Oregon State University.

EQUIPMENT

Participants operated a Cessna 172 RG airplane in a X-plane®-based general aviation flight simulator (Figure 3.1). In order to collect multitasking behavioral data, the following four instruments were prepared: a computer-synthesized voice for the ATC communication (this was operated by the experimenter), a flight checklist, and flight charts.

Figure 3.1 X-plane flight simulator

PROTOCOL

Before the experiment, participants conducted a practice session. Each participant learned how to fly the X-plane flight simulator, how to communicate with Air Traffic

Control (ATC), as well as learning and familiarizing themselves with the flight plan, and aircraft checklists. After the practice session was completed, the participant could ask for clarification regarding simulator operation.

Next, data collection was conducted using the flight simulator. Flight data (e.g., headings, altitudes, airspeeds, engine parameters, radio frequencies, and flight control movements) was automatically recorded by the simulator. Every few minutes the flight simulation was randomly frozen and the participant was asked to rate their task prioritization factors: the perception of the importance of tasks, urgency of tasks, performance status of tasks, salience of tasks, workload of tasks, and estimated priority of tasks. Here, we followed Endsley (1995b)’s query guideline such that the frozen timing was randomly determined at each of the eight experimental blocks. Because of its unpredictable interruption, the participants could not anticipate or prepare for queries beforehand, which provided unbiased estimates of the participant’s task-prioritization decisions. Furthermore, behavioral audio and video data on the pilot’s behavior were recorded throughout the flight.

FLIGHT SCENARIO

After becoming familiar with the flight simulation in the practice session, participants conducted a simulated flight scenario using two VORs (VHF Omni

Directional Radio Range) (Figure 3.2). The participants flew from the Eugene airport

(EUG) to Bend Roberts Field Airport (RDM) in the computer flight simulation. As seen in the Figure 3.2, this flight path was split into eight situations. Participants communicated with the experimenter who played the role of the ATC controller with the computer-synthesized voice based on a predetermined standardized communication script. A simple flight path was prepared during which the participants were presented with unexpected flight instrument problems. The participant landed at the destination airport by following a vectored approach guided by ATC.

Figure 3.2 Flight Path Used in the Flight Simulator

The participant was reminded that in the simulation, flight safety was the ultimate goal, and it was assumed that the participating participants prioritized among four tasks:

Aviate (aircraft vertical control), Navigate (aircraft lateral control), Communicate, and

Manage Systems. At each of the eight situations, participants were challenged to perform subsets of the four tasks (Table 3.1).

Table 3.1 Tasks at each Situation Situations Vertical Control Lateral Control ATC calls Checklist (Aviate) (Navigate) (Communicate) (Manage Systems) 1. Climb 1 0 1 1 2. Climb 1 0 0 0 3. Cruise 0 1 1 1 4. Cruise 0 1 0 0 5. Descend 1 1 1 1 6. Pre-landing 1 1 1 0 7. Pre-landing 1 1 1 1 8. Landing 1 1 1 1

MEASUREMENT AND HYPOTHESIS-TESTING

Definition of Terms

In this study, the term “priority” was defined as the superior rank, position, or privilege of a task. Following Wickens et al. (2013)’s definition, the priority of tasks was defined as the allocation policy of dividing resources among tasks. Similar to

Wicken’s SEEV model (e.g., Wickens, Helleberg, Horry and Talleur, 2003), the importance of tasks was defined as the value of tasks or the weight of the contribution of a task to the ultimate goal: “safe flight to the destination.” Following Wickens et al.

(2013), the urgency of tasks was defined as the buffer time: time difference between the deadline time of a task and the time required to finish the task (i.e., the smaller the buffer time of the task, the more urgent the task is). The performance status of tasks was defined as how successful the task was in accomplishing the goal. The salience of tasks was defined as the degree of attention-catching stimuli that relates to each task.

Following Wickens et al.(2013), the workload of tasks was defined as the task demand imposed on the limited information processing capacity of the brain. The above definitions are summarized in Table 3.2.

Table 3.2 Definition of Priority and Five Factors Terms Definitions Priorities of tasks Allocation policy of dividing resources among tasks. Importance of task Weight of contribution of a task to the ultimate goal “safe flight to the destination”. Urgency of task The time difference (i.e., buffer time) between the deadline time of a task and the time required to finish that task Performance status of How successful in accomplishing the goal of the task task Salience of task The degree of attention-catching stimuli that relates to the task. Workload of Task Business of the operator who is engaged in the task. “The demand of task imposed on the limited information processing capacity of the brain” (Wickens et al., 2013), P347

Four tasks were chosen based on the Colvin et al. (2005)’s second study using

Aviate-Navigate-Communicate-Manage Systems (ANCS) classification taxonomy. The

Aviate task consisted of those activities related to control of aircraft’s vertical motion.

Navigate task consisted of those activities related to control the lateral motion from the present location to arrive at an intended location. Here, the difference of Aviate and

Navigate activities were such that Navigate task was related to controlling the geographical location from a point A to point B on lateral direction control while Aviate task related to control of the aircraft’s motion particularly in the vertical altitude plane.

Communicate tasks consisted of those activities related to transmitting information to or receiving information from another human such as ATC. Manage Systems task consisted of those activities related to the operating the aircraft’s secondary equipment such as its electrical system, hydraulic system, etc.

Methodology for RQ-1

In order to answer the first research question “RQ-1) Can perceived task priority be explained by the following five factors?: 1.1 Perceived importance of task, 1.2

Perceived urgency of task, 1.3 Perceived performance status of task, 1.4 Perceived salience of task, and 1.5 Perceived workload of task”, it was necessary to estimate each participant’s perceptions of task priority and five criteria. Table 3.3 shows the probe questions used in the questionnaire. It was assumed that during the probe question time, the participants could recall what they perceived as pertaining to priority, importance, urgency, performance status, salience, and workload. Thus, the scope of data analysis inference based on the probe questions was constrained by the participant’s honest and accurate memory of the tasks.

Table 3.3 Probe questions used at the moment the simulation was frozen

Variables Probe Questions Y: Estimated priorities of Which task did you prioritize at this moment? tasks X1: Perceived importance Based on your comprehension of the current situation, which task is more of tasks important? X2: Reported Buffer Time Based on your projection of the future status, rate the urgency of each task of tasks (i.e., task urgency) by its "buffer time"; the amount of time you could delay the task before it requires your attention to maintain safe flight. X3: Perceived performance Based on your comprehension of the current situation, rate the Status of tasks performance of task, how successful you believe in accomplishing the goal of the task set by yourself? X4: Perceived salience of Based on your current perception, which task is more salient and draws tasks your attention at the moment? X5: Perceived workload of NASA’s TLX workload probe questionnaires that have a multi- Tasks dimensional rating procedure to measure an overall workload score. See the details in Appendix.

The task importance factor (X1), task salience factor (X4), and the task priority

(Y) were numerically coded using a unidimensional version of the Analytic Hierarchy

Process (AHP: Saaty, 1990) methodology (See the coding details in Appendix G -1 for task importance factor, G-4 for task salience factor, and G-6 for task priority). This methodology was used because it is known for good sensitivity and reliability (e.g.,

Vidulich & Tsang, 1987).

The task urgency (X2) was coded as the time difference (i.e., buffer time) between the perceived deadline time of a task and the time required to finish that task

(Figure 3.3). Thus, when the time difference (i.e., available buffer time) approaches zero, the task was regarded as more urgent. On the other hand, when the available buffer time increases, the task was regarded as less urgent (See the coding details in Appendix G

-2).

Figure 3.3 Coding the urgency in the model

The task performance (X3) data was coded numerically based on the questionnaire response as shown in Figure 3.4.

Figure 3.4 Coding of the Perceived Task Performance Status

Finally, the task workloads (X5) for each of the four tasks, were measured by

NASA TLX methodology (See Appendix G-5).

Based on the numerically coded data, the linear mixed models were constructed for the task priority (Y) using the five candidate factors. As fixed effects, the following explanatory variables were used in the model: X1(task importance), X2 (task urgency),

X3(task performance status), X4 (task salience), and X5 (task workload). As random block effects, the following random factors were used in the model: participants and flight situations. P-values of the likelihood ratio tests were used to estimate the effect of each of the five candidate factors.

Methodology for RQ-2

Answering the second research question “RQ-2) Is there any relationship between the perceived task priority and the chance of noticing task-related cockpit instrument malfunction signals? If so, how much does the perceived task priority affect the chance of noticing task-related signals considering the following factors?: 2.1.

Salience of task-related signals, 2.2 Expectancy of task-related signals, 2.3 The number of concurrent tasks” required an assessment of whether the participants were aware of any task-related signals.

Seven probe malfunctions were programmed to occur during the flight simulation to observe if the participants could notice the problem. Awareness of task-related signals was estimated by checking if participants noticed the task-related cockpit instrument problems by on-line and off-line evidence as follows.

The offline evidence was obtained at the simulation freeze moment with recall- based memory probe question: “Based on your perception in the current situation, did you notice any equipment malfunction?” If a participant reported the malfunction event

푝 on the questionnaire, then a binary number 1 was assigned, otherwise a binary number

0 was assigned.

On-line real-time awareness of task-related signals was determined from the recorded video by measuring the unnoticed time 푇푝, which is the amount of time it took for a participant to notice a malfunction problem p (recall that participants were asked to report any noticed problems as soon as possible). The participants were asked to promptly report any abnormality in the cockpit instruments. It was assumed that the

56 longer unnoticed time indicated worse awareness of task-related signals, and the shorter unnoticed time indicated the better awareness of task-related signals.

The four potential factors for awareness of task-related signals (signal salience, signal expectation, related task-priority, and the number of tasks per second) were coded as follows.

The salient signal factor 푆푆푝 was coded with binary numbers: 0 for a non-salient signal, and 1 for a salient signal. Figure 3.5 depicts the method of coding salient signal and non-salient signal conditions. Since the needle moved rapidly during malfunctions it was coded as salient signal condition in the airspeed indicator (when pitot-tube clog was the root cause), vacuum pump indicator, and VOR indicator. Since their needles froze during malfunctions, it was coded as non-salient signal condition in the airspeed indicator (caused by static-port blockage) altimeter, fuel indicator and vertical speed indicator.

Figure 3.5 Airspeed indicator, VOR indicator, and vacuum pump indicators were coded as salient signals as needles moved rapidly when they malfunctioned

The expectation of the problem event 퐸푝 was coded as 1 when the problem was expected and 0 when the problem was not expected, where the expectancy was artificially generated in the following way. Participants were challenged with problems shown in

Table 3.4, of which no information was given to participants about which problem(s) might occur or repeat again at each situation.

The problems in the left column of Table 3.4 were artificially repeated multiple times during the flight simulation. The repeating problem column includes those problems that participants could expect to happen because they re-occurred multiple times. Those problems were pitot-tube clog that caused the airspeed indicator problem at situations 2, 4, 7, the low fuel problem occurred at situations 6, 7, and 8, and altimeter problem at situations 4 and 8. Once a participant noticed the problem, it was coded as

58 expected and a binary number “1” was recorded in the data-recording spreadsheet. On the other hand, if a participant never noticed the problem, it was coded as unexpected and a binary number “0” was recorded.

The non-repeating problem column includes those problems that participants could not easily expect to happen because each problem occurred only once. Those problems were artificial horizontal indicator (or attitude indicator), vacuum pump indicator, vertical speed indicator, and navigation instrument malfunctions. Because of single occurrence, problems in the right column in Table 3.4 were coded as unexpected and a binary number “0” was recorded.

Table 3.4 Challenges to participants Situation Repeating problems or abnormal None-repeating problems or abnormal situation situation 1 - Artificial horizon indicator 2 Pitot tube clogairspeed Vacuum pump indicator 3 - Static port clog problem slow airspeed indicator, slow vertical speed indicator, inaccurate altimeter 4 Pitot tube clogairspeed Navigation instrument malfunction indicator 5 Altimeter - 6 (Low fuel) - 7 Pitot tube clogairspeed - indicator, (Low fuel) 8 Pitot tube clogairspeed - indicator, Altimeter, (low fuel)

As a measuring of task loading, for each instrument malfunction problem p , the number of concurrent tasks per second was estimated in the following way. First, all task activities were recorded with a binary number (0 or 1) at each second, and they included Aviate (single subtask), Navigate (two subtasks), Communicate (four subtasks), and Manage Systems (single subtask). The subtasks’ granularity was determined by considering the experimenter’s capability of distinguishing the smallest and meaningful task chunk from the recorded video (Figure 3.6). While the participants did not notice

an instrument malfunction problem at the time from t1 to tn : the unnoticed time period

was recorded with a binary code 1 for each second as a set S  {st1 1, st2 1,...stn 1}.

After a participant reported the occurrence of the correct problem, a binary number 0 was recorded in the rest of the signal awareness data filed in the spread sheet (i.e., s  0,s  0,... ). In the example of Figure 3.6, a hypothetical participant attended to tn1 tn2 the following tasks: the Aviate task from 0:01 to 0:30, the Navigate task from 0:01 to

0:02, and the Communicate subtask “Listen to ATC call” from 0:06 to 0:22, “Speak to

ATC” from 0:26 to 0:30. Since, the altimeter malfunction was not noticed from 0:13 to

0:24, a binary number “1” was recorded at the altimeter malfunction row in the spreadsheet indicating the unnoticed time period set S .

1st Level Task Sub tasks Sub-sub tasks 1. Aviate Altitude control subtask 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Altimater malfunction 1 1 1 1 1 1 1 1 1 1 1 1 Noticed Fly

2. Navigate Navigation system control The pilot did not notice the altimater malfunction during this time period. Total Unnoticed Time (Tp) is, Tp=1+1+1+....+1= 12 (seconds)

3. Communicate Listen to ATC 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 Speak to ATC 1 1 1 1 1 Change Comm Frequency Push Ident Button

Total tasks time e.g.,Concurrent Tasks TTp=2+2+...1+1=22 task =1+0=1 4. Manage System Checklist seconds Look at map / write memo

Total Tasks 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 2 2 2 2 2

Time (seconds) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Figure 3.6 The number of tasks while pilot did not notice a task-related signal.

Second, total unnoticed time 푇푝 was defined as the time duration where a participant did not report the instrument malfunction problem p . 푇푝 was formulated as the conditional summation of the binary number 1 for each second during the time when the problem was unnoticed ( s 1).

(3) Tp  (s) s1

Third, total tasks time 푇푇푝 was defined as the total tasks-seconds while a participant did not notice the instrument malfunction problem . 푇푇푝 was formulated as,

(4) TTp  (Task _ x)(s) AllX s1

Finally, the observed concurrent tasks per second 퐶푇푝 was coded as the mean value of total tasks time 푇푇푝 divided by the unnoticed time period 푇푝 . 퐶푇푝 was formulated as,

 (Task _ x)(s) TTp AllTask s1 CTp   (5) Tp (s) s1

In the example of Figure 3.6, a participant does not notice the signal for 12 seconds, and 22 tasks-seconds are executed, thus the unnoticed time is 푇푝 =12 , total task

time is 푇푇푝 = 22 , and the observed concurrent tasks per second is CTp = 22/12=1.83.

This means that the participant did not notice the signal of the problem p while executing on average 1.83 tasks per second. This coding is summarized in Table 3.5

Table 3.5 Data Coding Methodology for Measuring the Awareness of Signals

Variables Measurement Methodologies 푨(풑): Awareness of the task- Obtain binary data (1 or 0) for the probe question; “Based on your related problem 풑 and abnormal perception in the current situation, did you notice any equipment situations (Binary: 0 or 1) malfunction? Please click a checkbox(s) if you noticed any equipment failure.”

푻풑: Unnoticed time period of During the flight simulation, a participant verbally reported as soon problem p as they noticed any problems or abnormal situations. We measure the time between the malfunction onset and problem reported time.

푺푺풑 : Signal salience condition of Airspeed indicator (caused by pitot tube clog) Vacuum Pump problem indicator, and RMI indicator problems were regarded as salient instrument panels because the needles move quickly to NG zone. The salient signal was coded as 1, and non-salient signals were codded as 0.

푬풑: Signal expectancy condition When the problem occurred at the first time, it was coded as 0 (not of problem expected). If the problem occurred again, it was coded as 1 (expected).

푪푷풑,: Concurrent tasks observed 퐶푇푝 was coded as the fraction of total multitask time 푇푇푝 divided by per second the total unnoticed time 푇푝.

Based on the above data coding, regression models were constructed to determine

if the unnoticed time periodTp could be explained with the four potential factors: signal salience factor( 푆푆푝) , signal expectation factor (퐸푝), number of concurrent tasks (퐶푃푝), and perceived task priority (푃푝).

Methodology for RQ-3

In order to answer the third research question “RQ-3) Can actual task execution and task performance be explained by the perceived task priority?”, evidence of task execution was collected with the camcorder and flight recorder based on the criteria in

Table 3.6. As introduced in RQ-2, task execution activities during each second were coded with binary number after observing the recorded video. The recorded task activities included Aviate, Navigate, Communicate (subtasks: Listen to ATC, speak to

ATC, change frequency, press Ident key), and Manage Systems tasks (subtasks: look at a map/write memo, and perform a checklist, such as LANDING – see Appendix D). For each subtask activity, if the task execution was observed, then a binary number “1” was recorded at each second. In the example of Figure 3.6, a pilot spoke to ATC from

“00:06” to “00:22”, a binary number “1” was recorded during this time zone in the data- recording spreadsheet.

Table 3.6 Collected data for Task Execution Measurement Variables Task Execution Evidence 푬풙풆(푨): Aviate Task Execution 1: If the participant holds yoke control from the recorded video (Binary: 0 or 1) 0: Otherwise 푬풙풆(푵):Navigate Task 1: If direction was systematically and continuously changed (more Execution (Binary: 0 or 1) than 1 degree per second) 0: Otherwise 푬풙풆(푪ퟏ): Listen to ATC 1: If the participant kept quiet and listened to the ATC call request. 0: Otherwise Exe (C2): Speak to ATC 1: If the participant said something to ATC 0: Otherwise Exe(C3): Change 1: If the participant changed the communication frequency communication frequency 0: Otherwise Exe (C4): Push Ident Button 1: If the participant pushed the Identification button 0: Otherwise 푬풙풆(푺푴ퟏ): 1: If the participant conducted checklists. 0: Otherwise 푬풙풆(푺푴ퟐ): 1: If the participant did either looking at the map, or writing memo 0: Otherwise

Task performance data was obtained from the flight recorder and recorded video based on the criteria in Table 3.7. Aviate task performance was measured by the altitude deviation from the target cruising altitude. Navigate task performance was measured as the directional deviation from the target navigation path. In both Aviate and Navigate tasks, a smaller deviation was regarded as better performance, while a larger deviation was regarded as poorer performance. Communicate task performance was measured based on pilot’s responses to ATC’s request. If the pilot responded appropriately to

ATC’s request, a binary number of “1” was recorded on the spread sheet. Manage

Systems task performance was measured based on how a participant conducted required checklist tasks (which are subtasks of Manage Systems). If a participant conducted the checklist task, then a binary number of “1” was recorded on the spreadsheet.

The obtained data set was compared with the perceived task priority scores to determine if the task execution and task performance could be explained by the perceived task priority score.

Table 3.7 Collected data for Task Performance Measurement

Variables Task Performance Measurement Methodologies Altitude deviation from the Measured altitude deviation from the target cruising altitude as the cruising altitude Aviate task performance (Continuous) Directional deviation from the Measured deviation from the target navigation path as the Navigate target path task performance Communicate Task 1: If a participant responded to ATC request Performance 0: Otherwise (Binary: 1 or 0) Manage Systems task 1: If a participant conducted required checklists performance evaluation 0: Otherwise (Binary: 1 or 0)

4. CHAPTER 4: RESULTS

In this chapter, the main findings for the three research questions (RQ-1, RQ-2, and RQ-3) are presented. The first section answers RQ-1; it reports how the perceived task priority scores related to five task priority decision factors (perceived task importance score, perceived task urgency, perceived task performance status score, perceived task salience score and perceived task workload score) for each task from the questionnaire data. The second section answers RQ-2; it reports how the perceived task priority correlated with actual awareness of task-related signals from the recorded video.

The third section answers RQ-3; it reports how the perceived task priority correlated with the actual execution of tasks from recorded video and the simulation flight recorder.

Finally, the main findings are summarized.

RESULTS FOR RQ -1

The first research question was, “RQ-1) Can perceived task priority be explained by the following five factors?: 1.1 Perceived importance of task, 1.2 Perceived urgency of task, 1.3 Perceived performance status of task, 1.4 Perceived salience of task, and 1.5

Perceived workload of task” The data of task priority and five factors was obtained by providing probe questions to the participants at the simulation-frozen moments.

The first five subsections show the relationships between the perceived task priority and each of the five potential task-prioritization factors. With statistical analysis programs R and lme4, hypothesis tests for linear relationships were conducted between the perceived priority score of four tasks (Aviate, Navigate, Communicate, and Manage

Systems) and each of the five potential factors. The sixth subsection shows the characteristics of estimated task urgency factor. The seventh subsection presents the individual and flight situation differences in perceived task priority. The eighth subsection analyzes the relative importance of task prioritization factors. The ninth subsection shows an analysis to investigate significant task prioritization factors when all five factors are taken into consideration. The above analysis is summarized in the final subsection.

Relationship Between Perceived task importance and Perceived task priority

The first part of the RQ-1 analysis was to determine if there was a linear relationship between perceived task priority and perceived task importance. Perceived task priority scores for each of the four tasks (Aviate, Navigate, Communicate, and

Manage Systems) were plotted and fitted with mixed linear regression models in Figure

4.1. Here, the perceived task importance score was used as the fixed effect. The participants were used as random effects for the perceived task importance. A residual plot verified no violation of homoscedasticity or normality assumptions. A likelihood ratio test was used for testing a linear relationship between the perceived task priority and the perceived task importance for each of the four tasks.

There was strong evidence of a linear relationship between the perceived task priority score and the perceived task importance score for four tasks (P-Value<0.001 for

Aviate, Navigate, P-Value=0.004 for Communicate task, and P-Value=0.04 for Manage

Systems task). The test results were consistent with Colvin, Funk, and Braune’s (2005) hypothesis that “The priority of a task is directly proportional to its importance” (Page

334).

Figure 4.1 Scatter plots of perceived task priority (Y-axis) and importance for each task (X-axis)

Relationship Between Perceived task urgency and Perceived task priority

The second part of the first question was to determine if there was a linear relationship between the perceived task priority and perceived task urgency. The perceived task priority score and the reported buffer time of tasks were plotted and fitted with linear regression models for each task in Figure 4.2. Likelihood ratio tests confirmed a linear relationship for Aviate (P-value=0.03), Navigate (P-value=0.07),

Communicate (P-value=0.02), and Manage Systems (P-value<0.01) tasks. The test results were consistent with Colvin, Funk, and Braune’s (2005) hypothesis that “the priority of a task is directly proportional to its urgency.” (Page 336)

Figure 4.2 Scatter plot of perceived task priority (Y-axis) and reported buffer time

Relationship Between Perceived Task performance Status and Perceived task

Priority

The third part of the first research question was to determine if there was a linear relationship between perceived task priority and perceived task performance status. The perceived task priority scores and the perceived task performance status scores were plotted and fitted with linear regression models (Figure 4.3). Likelihood ratio tests confirmed a weak linear relationship for the Navigate task only (P-value =0.06), while there was not enough evidence for Aviate (P-value =0.9), Communicate (P-value =0.6), and Manage Systems (P-value=0.11) tasks. Thus, the linear regression test results indicated that only the Manage Systems task was consistent with Colvin, Funk and

Braune’s (2005) hypothesis that “The priority of a task is inversely proportional to its

Status.” (Page 335)

Figure 4.3 Scatter plot of the perceived priorities (Y axis) and task performance status (X axis) for each task

Relationship Between Perceived task salience and Perceived task priority

The fourth part of the first research question was to determine if there was a linear relationship between perceived task priority and perceived task salience. A mixed model was used for this analysis. The perceived task priority scores and the perceived task saliences scores were plotted and fitted with linear regression models in Figure 4.4.

Likelihood ratio tests confirmed linear relationships for all four tasks; Aviate (P-value

<0.001), Navigate (P-value <0.001), Communicate (P-value<0.001), and Manage

Systems (P-Value<0.001). Thus, linear regression test results were consistent with

Colvin, Funk, and Braune’s (2005) hypothesis “The priority of a task is directly proportional to its Salience.” (Page 335)

Figure 4.4 Scatter plots of the perceived task priority (Y axis) and perceived salience (X axis) for each task

Relationship Between Perceived workload and Perceived task priority

The fifth part of the research question was to determine if there was a linear relationship between the perceived task priority and perceived task workload. A mixed model was used for this analysis. The perceived task priority scores and the perceived task workload scores were plotted and fitted with linear regression lines (Figure 4.5).

Likelihood ratio tests confirmed a significant linear relationship for Aviate (P- value=0.03), while no significant effect was observed for Navigate (P-value=0.3), and

Communicate (P-Value=0.23) and Manage Systems (P-value =0.22) tasks. Thus the linear regression test results indicated that only the Aviate task was consistent with

Colvin, Funk, and Braune’s hypothesis (2005) “The priority of a task is proportional to the Time/Effort required to perform it.” (Page 335 )

Figure 4.5 Scatter plots of the perceived workload (X axis) and priority (Y axis) for each task

Characteristics of Perceived task urgency

The characteristics of the perceived task urgency factor were investigated. For each of the task-importance and task-urgency factors, the variance-covariance matrix of the four tasks were analyzed by Principal Component Analysis (PCA) in order to visualize the relationships of the four tasks on a two-dimensional graph. The biplots

(Figure 4.6and Figure 4.7) use two artificial axes called principal components (PCs);

PC1 and PC2 are primary and secondary eigenvectors of the correlation matrix.

Figure 4.6summarizes the perceived importance of the four tasks. The Aviate

Task vector heading left and Communicate task vector heading right (the opposite direction) indicates that when a participant perceives the Aviate task as more important, they would perceive the Communicate task as less important.

Figure 4.7 summarizes perceived Aviate task buffer time (i.e., least urgent) of four tasks. Here, the orthogonality of directions between Aviate and Communicate tasks indicates that the reported buffer time of the Communicate task is independent from the buffer time of the Aviate task.

The above analysis might indicate the following points on participant’s perception in task management. First, participants might make a task importance tradeoff between

Aviate and Communicate tasks because the vectors of the perceived importance of Aviate and Communicate tasks head in opposite directions. Second, participants might not trade-off on urgency criteria between Aviate and Communicate (or, Navigate or Manage

Systems tasks). Given the orthogonal vector directions, participants might perceive the buffer time of the Aviate task independently from the one of the Communicate task.

From a multitasking point of view, this trend might indicate that participants thought they were able to set high urgency on two tasks simultaneously; the Aviate task and the other task (either Communicate, Navigate, or Manage Systems task).

Figure 4.6 Biplot of the Perceived Task Importance Score

Figure 4.7 Biplot of the Reported Task Buffer Time

Furthermore, the relationship between the perceived task priority score and the perceived urgency (i.e., buffer time) was investigated conditionally with the perceived task performance score. Figure 4.8 plots perceived task priority (Y-axis) and reported task buffer time (X-axis). Square boxes indicate when participants reported unsatisfactory task performance (Low), and triangles indicate when participants reported neutral performance (Middle). Circles indicate when participants reported satisfactory task performance (High).

Graph A in Figure 4.8 shows the characteristics of the Aviate task; when the participants felt the Aviate task was satisfactory (High), the regression slope was steeper

(Wald’s test for the slope was the slope P-Value=0.06 after taking account of individual block effect) than the slopes of neutral (Wald’s test for the slope was P-Value=0.79) or unsatisfactory (Low) Aviate task performance (Wald’s test for the slope was P-

Value=0.96). This might indicate that participants felt the Aviate task was more successful when the high priority aviate task was not deferred.

Graph B in Figure 4.8 shows the characteristics of Navigate task; there were no significant differences among the three regression slopes: unsatisfactory (Wald’s test for the slope was P-Value=0.08), neutral (Wald’s test for slope was P-Value=0.12) and satisfactory (Wald’s test for the slope was P-Value=0.15).

Figure 4.8 Self-Evaluated Performances of Aviate and Navigate tasks in two dimensional plots of reported buffer time (i.e., urgency) in X-axis and perceived task priority on Y-axis

The experienced participants who had a rating for instrument flight rules (IFR) and the novice participants with a rating only for visual flight rules (VFR) showed different task priority structures against task urgency. Graph A in Figure 4.9 shows that

IFR participants kept a higher level of the perceived Aviate task priority score than VFR participants over the entire range of Aviate task Buffer time (the perceived Aviate buffer time was less than 30 seconds). IFR participants had flat slopes in Aviate (P-

Value=0.94), Navigate (P-Value=0.51), and Communicate (P-Value=0.08) tasks while the slope effect was significant on Manage Systems task (P-value=0.01). This may indicate that IFR participants took consideration of the task buffer time for the Manage

Systems task while the task priorities were not related much with the reported buffer time in Aviate, Navigate, and Communicate tasks. On the other hand, VFR participants had flat slopes on Aviate (P-value=0.85) and Navigate (P-value=0.22), but they had steeper slopes for Communicate (P-value=0.09), and Manage Systems (P-value=0.05) tasks in

Figure 4.10. This may indicate that VFR participants took consideration of the task buffer time for both Communicate and Manage Systems tasks but not on Aviate and

Navigate tasks. The above difference may indicate that both IFR and VFR participants might prioritize tasks automatically in ANCS order, and IFR participants prioritize Aviate task more than VFR participants.

Figure 4.9 Aviate Task and Navigate Task Priorities in VFR/IFR Participants

Figure 4.10 Communicate and Manage Systems Task Priorities in VFR/IFR participants

Individual Difference and Situation Difference

Significant individual difference was observed between the participants in perceived task priorities as well as perceived importance, urgency, performance status, salience, and workload for each task. The individual difference of the perceived task priority scores was huge. Figure 4.11 shows the difference between individual participants, with regards to perceived task priority scores of Aviate, Navigate,

Communicate, and Manage Systems (ANCS tasks). The top left graph in Figure 4.11 shows small variance for the perceived Aviate task scores among the following participants: No. 1, 2, 4,12, and 14, while the following participants had a large variance

:No. 3, 5, 10, 11, 13, 15, and 16. A horizontal line at the center of the graph shows the global mean task priority score of all the sixteen participants. This line indicated the diversity of the perceived Aviate task priority, because the boxplots of six out of sixteen participants did not include the global center line.

Figure 4.11 Individual Difference of the perceived task priority score at each task

On the other hand, differences of the perceived task priority scores were small in each situation when all participants are pooled (Figure 4.12). The 95% intervals of the perceived task priority included the mean task priority score in the most of the situations.

Thus, the individual difference of the perceived task priority score was larger than the one of the situational difference. These individual differences and situation differences were taken into consideration in the mixed model analysis of the perceived task priority score and the five task prioritization decision factors.

Figure 4.12 Difference of the perceived task priority score at each situation

Correlation of the perceived task priority and perceived importance were different in individual perspective and flight situation perspective. In the flight situational perspective, a consistency was observed; all eight situations had positive correlation between the perceived task priority and task importance (Figure 4.13).

Figure 4.13 Relationship Between Aviate Task Priority and Aviate Task Importance at Each of the Eight Different Flight Simulation Situations

On the other hand, individual difference was observed between the participants.

Eleven participants had a positive correlation, while four participants had no correlation and one participant had a negative correlation (Figure 4.14).

Figure 4.14 Relationship Between Aviate Task Priority and Aviate Task Importance from Participant Perspective

Relative Importance of Five Candidate Factors

Among the five task prioritization decision factors, which variables were the most important in predicting the perceived task priority scores? Relative importance of coefficients was investigated by using the relative weights methodology (Johnson, 2004) with R-code provided by Kabacoff (2011). The R-code was used to calculate the mean increase of the R-square value by adding one of the five task prioritization decision criterion across the set of all possible combinations. Figure 4.15 shows the relative importance of five candidate factors for the estimated Aviate task priority score. The estimated Aviate task importance criterion explained 57.4% of the Aviate task full model, and the estimated Aviate task salience criterion explained 37.3% of the Aviate task full model. As can be seen in Figure 4.15, Figure 4.16, and Figure 4.17, the perceived importance and salience criteria had the highest relative weights in the Aviate, Navigate, and Communicate tasks. In Manage Systems (Figure 4.18), the perceived importance and workload were the most important criteria.

Figure 4.15 Bar plot of relative weights for the perceived Aviate task priority score

Figure 4.16 Bar plot of relative weights for the perceived Navigate task priority score

Figure 4.17 Bar plot of relative weights for the perceived Communicate task priority score

Figure 4.18 Bar plot of relative weights for the perceived Manage Systems task priority score

Significant task prioritization criteria when considering five factors together

Assuming that participants considered all five task prioritization factors, which were significant ones? The sampled dataset was fitted with the following statistical model to find the most significant factors for the perceived task priority score.

yij  β0  βImp x Imp  βBTx BT  βPfm x pfm  βSalxsal  βWLx WL  S ubject(i)  Situation(j)  εij (6)

Where yij represents the perceived priority score of task by a participant i at situation j .

The coefficients β Imp , β BT , βPfm βSal , and β WL represent the perceived importance score, reported buffer time, perceived task performance score, perceived salience score, and perceived task workload score. The subject(i) and situation( j) represent the random effects,

and  ij represents the residual error of the model. A linear mixed effects analysis was used to test the relationship between perceived task priority and the five task priority factors (perceived task importance, reported task buffer time, perceived task performance, perceived task salience and perceived task workload). Five task priority factors were entered into the model as fixed effects. Participants were used as random effects. Visual inspection of residual plots did not reveal any obvious deviations from homoscedasticity or normality. A likelihood ratio test was used for testing the relationship between the perceived task priority and each of the five factors in Aviate task.

For the perceived Aviate task priority, the following full model was obtained.

Priority (Aviate) = - 0.19 + 0.85* Avi_Imp + 0.0002* Avi_BufferTime + 0.015* Avi_Perfm

- 0.222* Avi_Sal + 0.042* Avi_Wrkld + (participant) +  (7)

Likelihood ratio tests revealed that the estimated Aviate task importance score (P- value=0.01), the reported Aviate task performance status score (P-value=0.04), and the estimated Aviate task workload score (P-value=0.01) were statistically significant; while the reported Aviate task buffer time (P-value =0.074), and the estimated Aviate task salience score (P-value=0.064) were moderately significant.

For the estimated Navigate task priority, the following full model was obtained.

Priority(Nav) = -0.13 + 0.76*Nav_Imp + 0.001*Nav_BufferTime + 0.004*Nav_Perfm

+ 0.314* Nav_Sal + 0.012* Nav_Workload + (participant) +  (8)

Likelihood ratio tests revealed that the estimated Navigate task importance (P- value<0.001) and the estimated Navigate task salience (P-value=0.01) were statistically significant while the reported Navigate task buffer time (P-value =0.31), the reported

Navigate task performance status (P-value=1), and the estimated Navigate task workload

(P-value=0.29) were not significant.

For the estimated Communicate task priority, the following full model was obtained.

Priority(Com)  0.060.70*Com_Imp 0.0006*Com_BufferTime 0.013*Com_Perfm 0.11*Com_Sal 0.002*Com_Workload (participant)ε (9)

A likelihood ratio tests revealed that the estimated Communicate task importance

(P-value=0.002) was the only statistically significant factor while the reported

Communicate task buffer time (P-value =1), the reported Communicate task performance status (P-value=0.36), the estimated Communicate task salience (P-value=0.42), and the estimated Communicate task workload (P-value=0.36) were not significant.

For the perceived Manage Systems task priority, the following full model was obtained.

Priority(MS) = 0.03 + 0.60*MS_Imp + 0.0002*MS_BufferTime - 0.002*MS_Perfm

+ 0.08* MS_Sal + 0.002* MS_Workload  (participant)   (10)

Likelihood ratio test revealed that the estimated Manage Systems task importance

(P-value=0.0003) was the only statistically significant factor while the reported Manage

Systems task buffer time (P-value =1), the reported Manage Systems task performance status (P-value=1), the estimated Manage Systems task salience (P-value=0.28), and the estimated Manage Systems task workload (P-value=1) were not significant.

The above analysis indicates that the perceived task importance was the only criterion that was constantly significant across all types of tasks assuming that participants used five task prioritization criteria in task prioritization.

Summary of the Result of RQ -1

Perceived workload of task” The data of task priority and five factors was obtained by providing probe questions to the participants at the simulation-frozen moments.

Table 4.1 summarizes simple linear relationships between the four perceived task priority scores and the five potential prioritization criteria after taking individual difference and flight situation difference into account. Each P-value shows the linearity test result between the perceived task priority and each of the five factors. A small P- value (P-Value<0.05) indicates a significant linear relationship. Table 4.1 indicates that the perceived importance (X1), perceived urgency score (X2), and perceived salience

(X4) indicated significant linear relationships for almost all four tasks. The relative weights methodology also confirmed that the perceived importance (X1) and the perceived salience scores (X4) explained most of the variance of the perceived task priority scores. There was not enough evidence of a linear relationship between the perceived performance status score (X3) and the perceived workload score (X5).

100

Table 4.1 P-values of five potential factors for four tasks

Y: X1: Perceived X2: Perceived X3: Perceived X4: Perceived X5: Priority of Importance Urgency Performance Salience of Perceived each task Status tasks Workload Aviate task P-Value<0.001 P-value=0.03 P-Value=0.9 P-Value<0.001 P-Value=0.03 R2  0.69 R2  0.62 0.49 0.64 0.57 Navigate task P-Value<0.001 P-Value=0.07 P-Value<0.06 P-Value<0.001 P-Value=0.3 0.53 0.33 0.36 0.45 0.43 Communicate P-Value=0.004 P-Value=0.02 P-Value=0.6 P-Value<0.001 P-Value=0.23 0.77 0.47 0.42 0.79 0.11 Manage P-Value=0.04 P-Value<0.01 P-Value=0.11 P-Value<0.001 P-Value=0.22 Systems 0.66 0.65 0.46 0.63 0.59

Some task prioritization factors had conditional relationships with the perceived task priority. For example, the reported Aviate task buffer time was significantly related to the estimated Aviate task priority when the perceived task performance score was high.

However, other tasks (Navigate, Communicate and Manage Systems) did not show this trend. This may indicate that for the pilot participants, urgency factor was important for

Aviate task, but not for other tasks when task performance was satisfactory. A principal component analysis (PCA) indicated that two tasks could have high urgency simultaneously in conditional way: the Aviate task as the ongoing task (OT), and any other task as the interrupting task (IT). This may indicate that participants felt that a combination of the Aviate (OT) and the Communicate (IT) tasks could be simultaneously urgent (i.e., it is possible to conduct concurrent multitasking). However, participants would not feel any two or more combination of IT (Navigate, Communicate, and Manage

Systems) could be simultaneously urgent, thus they would conduct these IT tasks sequentially.

101

RESULTS OF RQ -2

The second research question was “RQ-2) Is there any relationship between the perceived task priority and the chance of noticing task-related cockpit instrument malfunction signals? If so, how much does the perceived task priority affect the chance of noticing task-related signals considering the following factors?: 2.1. Salience of task- related signals, 2.2 Expectancy of task-related signals, 2.3 The number of concurrent tasks”

In order to answer this question, relationships between the actual awareness of task-related signals of each task and four factors (perceived task priority, task-related expectation, task-related salience, and the number of concurrent tasks) were investigated.

102

Task Priority Effect on Awareness of Task-Related Signals

First, the effect of the perceived task priority on the awareness of task-related signals was investigated. Figure 4.19 shows a logarithm scatter plot of the unnoticed time period versus the perceived task priority score with the estimated linear regression line. As the perceived task priority increased by one unit, the median of unnoticed time period of the malfunction signal decreased by a factor of 0.141 (Bonferroni’s 95% confidence interval was between 0.04 and 0.48) of which, the participant’s block effect was taken into account. This means that the priority score change from 0.06 (the minimum) to 0.70 (the maximum) would reduce the expected median of unnoticed time from 100 seconds to 28 seconds2 in this experiment. This result is consistent with the study of Iani and Wickens (2007). Next, other factors’ effects were investigated and compared with the task priority effect.

1 Because this model took the logarithm of the unnoticed time, the effect of the explanatory variable should take an exponential value: e (xmax xmin )  e1.97(0.70.06)  0.28 2 It should be noted that the task priority data was retrospectively obtained with probe questionnaires at the simulation freeze points, thus confounding factors may exist (e.g., memory interference), and the statistical statement of significance may not imply an effect or causation for the perceived task priority effect on awareness of the signal.

103

Figure 4.19 Relationship Between the Unnoticed Time Period of Cockpit Instrument Malfunction Signals (Y-Axis) and Perceived task priority (X-Axis)

104

Signal Expectancy Effect on Awareness of Task-Related Signals

The effect of the estimated task-related signal expectancy on the awareness of task-related signals was investigated. Figure 4.20 shows the logarithm scatter plot of the unnoticed time period versus the signal expectancy condition (“not expected” v.s.

“expected”) with an estimated linear regression line. When the signal was expected, the median of unnoticed time of the malfunction signal decreased by a factor of 0.46

(Bonferroni’s 95% confidence interval was between 0.28 and 0.77) after accounting for the participant’s block effect. This means that a change of expectancy condition from

“not expected” to “expected” would reduce the expected median unnoticed time from 100 seconds to 46 seconds, which is 1.6 times longer than the task priority effect in this experiment.

Figure 4.20 Relationship Between the Unnoticed Time Period of Cockpit Instrument Malfunction Signals (Y-Axis) and Signal Expectancy Condition (X-Axis)

105

Signal Salience Effect on Awareness of Task-Related Signals

Figure 4.21 shows a logarithm scatter plot of the unnoticed time period versus the signal salience condition (not salient V.S. salient) with the estimated linear regression line. As the condition of signal salience changes from non-salient to salient, the expected median of unnoticed time period of malfunction signal will be decreased by a factor of

0.09 (Bonferroni’s 95% confidence interval was between 0.06 and 0.13) after accounting for the participant individual’s block effect. This means that a salient signal would reduce the expected median of unnoticed time period from 100 seconds to 9 seconds, which was even shorter than the task priority effect (28 seconds) in this experiment.

Figure 4.21 Relationship Between the Unnoticed Time Period of Cockpit Instrument Malfunction Signals (Y-Axis) and Signal Salience Condition (X-Axis)

106

Number of Concurrent Task Effect on Awareness of Task-Related Signals

Figure 4.22 shows a logarithm scatter plot of the unnoticed time period versus mean number of observed concurrent tasks per second with the estimated linear regression line. As the number of concurrent tasks increases by one unit, the expected median of unnoticed time of malfunction signal (seconds) was 2.7 times longer

(Bonferroni 95% confidence interval was between 1.7 and 4.3) after accounting for the participant’s block effect. This means that adding one more concurrent task would increase the expected median of unnoticed time period from 100 seconds to 270 seconds in this experiment.

Figure 4.22 Relationship Between the Unnoticed Time Period of Cockpit Instrument Malfunction Signals (Y-Axis) and Observed Workload (X-Axis)

107

Interaction of number of tasks against the Salience and Expectancy Factors

The above analysis indicates that both signal expectation and signal salience lowered the unnoticed time of problems, but each factor had unique characteristics.

Signal expectancy and signal salience factors had an effect of shortening the unnoticed time, but this effect disappeared as participants were engaged in additional concurrent tasks. However, when the signal was both expected and salient, participants could robustly notice the signals even if they were engaged in more concurrent tasks.

Figure 4.23 shows the unnoticed time with mean number of concurrent tasks per seconds in four different conditions: 1. neither expected or salient (squares), 2. expected only (triangles), 3. salient only (dark circles), and 4. expected and salient (dark triangles).

When the signals were neither expected nor salient, the mean time to notice the task- related problem signal was 137.8 seconds and no significant effect of mean number of concurrent tasks was observed (P-value=0.9). When the instrument panel malfunction signals were expected but not salient, the mean time to notice the task signal was 122.5 seconds but it increased as the number of mean concurrent tasks increased (P- value=0.01). When the instrument panel malfunction had a salient signal but it was not expected, the mean time to notice the task signal was 24.8 seconds, but it increased as the number of concurrent tasks increased (slope P-value=0.03). When a problem was expected with a salient signal, the mean time to notice the task signal was 23.9 seconds and it did not increase with additional tasks (slope P-value=0.7).

108

Figure 4.23 The relation between the Log(Unnoticed Time) and the mean number of tasks in four situations: 1. Neither Expected or Salient (squares), 2. Expected Only, (triangles) 3. Salient Only (circles), and 4. Expected & Salient (dark triangles)

109

Predicting the Awareness of Task-Related Signals with Priority, Expectancy, Salience and the Number of Concurrent Tasks

Based on the above findings, an integrated model was constructed for predicting the awareness of task-related signals with priority, expectancy salience and the number of concurrent tasks factors. Considering the possible interaction among signal expectancy, signal salience, and the number of concurrent tasks which were discussed above, the following model was constructed.

log(t)  5.58 - 2.0*Priority 0.53*Expected -1.88 *Salient -1.39*Expected :Salient

0.72*Salient :Tasks(participant)(situation)  ε (11)

On the left hand side of the model, log(t) is the logarithm of unnoticed time period t , and the explanatory variables are written on the right hand side. Statistically significant effects were observed in the following factors: perceived task priority

(Wald’s test P-Value=0.05), signal expectancy (Wald’s test P-value=0.41), signal salience (Wald’s test P-value=0.04), an interaction effect of expectancy and salience

(Wald’s test P-value=0.04), and an interaction effect of signal-related salience with number of tasks (Wald’s test P-value=0.11) after taking individual and flight situation block effects into account.

According to the model (11), an increase of 1.0 unit of the perceived task priority score will be decreased the median of the unnoticed time by a factor of 0.14

(Bonferroni’s 95% confidence interval was between 0.10 and 0.18). When the task- related signal was expected, the median of unnoticed time is increased by a factor of 1.69

110

(Bonferroni’s 95% confidence interval is 1.43 and 2.0). If the task-related signal is salient, the median of unnoticed time is decreased by a factor of 0.15 (Bonferroni’s 95% confidence interval is 0.12 and 0.19). If the task-related signal is both expected and salient, the median of unnoticed time is reduced by a factor of 0.25 (Bonferronis 95% confidence interval is 0.21 and 0.3). When the signal is salient, an additional concurrent task increases the median of unnoticed time by a factor of 2.05 (Bonferronis 95% confidence interval is 1.82 and 2.30). Consequently, perceived task priority, signal- expectancy, signal-salience, and an interaction of salience-expectancy, had a significant decreasing effect, while the number of tasks had a significant increasing effect on the unnoticed time period.

111

In order to validate the usefulness of the obtained model in the real world, a cross- validation technique was used for the model (11). The obtained data was randomly split into the training data set ( n 104 ) and test data set ( n 121)3. Figure 4.24 compares the model fitting performance with training data and test data. The obtained model explained 74% of the variance in the training data set ( R2  0.74 or r  0.86 ), and 34% of the variance in the test data ( R2  0.34 or r  0.58 ).

Figure 4.25 plots the actual unnoticed time (Y-axis) with the model-predicted unnoticed time (X-axis) using the test data. If the model predicts well, then the plotted dots will be located near the dotted diagonal line Y=X. Although the dots were scattered around the diagonal line, the smoothing curve lies along the line of the perfect prediction, which indicates that the obtained model (11) predicts correctly on average.

3 It should be noted that multiple problems occurred at each frozen point ( n 128). After removing the lost data points, the total number of signal detection events was n  225 .

112

Figure 4.24 Comparing the model fitting performance between the training data (left) and the test data (right)

113

Figure 4.25 A plot of prediction versus actual data of unnoticed time with Unit: log (Seconds))

114

Summary of the Result of RQ-2

In order to answer the second research question “RQ-2) Is there any relationship between the perceived task priority and the chance of noticing task-related cockpit instrument malfunction signals? If so, how much does the perceived task priority affect the chance of noticing task-related signals considering the following factors?: 2.1.

Salience of task-related signals, 2.2 Expectancy of task-related signals, 2.3 The number of concurrent tasks”, the awareness of task-related signals level was measured in different conditions of the above factors.

As an individual factor, a maximum increase of the task priority score had an effect of reducing unnoticed time from 100 seconds to 14 seconds. Signal expectancy and signal salience factors had an effect of reducing the unnoticed time from 100 seconds to 46 seconds and 9 seconds respectively. However, when participants were conducting concurrent multitasking, and either the salient signal factor or expectancy signal factor was present individually, their effect of reducing the unnoticed time period disappeared.

On the other hand, when the signal-related salience factor and signal-expectancy factor were both present, participants noticed the problems in a shorter period of time even during multitasking. Cross-validation technique suggests the usefulness of the obtained model in a real-world situation.

115

RESULTS OF RQ - 3

The third research question was “RQ-3) Can actual task execution and task performance be explained by the perceived task priority?” In order to answer this question, relationships between the estimated task priority score and actual task execution and task performance were investigated for each of the four tasks.

The Perceived task priority and Executed Task at the Simulation Freeze Moment

Analysis of all simulation freeze points revealed that Aviate task priority was estimated as the highest prioritized task with a mean task priority score of 0.50, followed by Navigate (0.24), Communicate (0.12) and Manage Systems (0.15) tasks. Similarly, a consistent trend was observed in task execution time periods on the video; the mean proportion of the observed task execution time was 59.6% for Aviate task, 26.4% for

Navigate task, 12.2% for Communicate task, and 1.8% for Manage Systems tasks. This trend was consistent across the eight situations (Figure 4.26).

116

Figure 4.26 Mean scores of the perceived task priorities at eight different flight situations

117

Figure 4.27 shows that the Aviate task was estimated to be the most prioritized task (among Aviate, Navigate, Communicate, and Manage Systems) in 95 out of all 120 frozen moments (79%). On the other hand, Navigate, Communicate and Manage

Systems were the highest task priorities in only 15, 6, and 4 times out of 120 frozen times, respectively.

Aviate task execution was always observed even when another task received the highest priority. Even when the Navigate task was the highest perceived task priority, the

Aviate task was always executed (15/15=100%). Similarly, even when the

Communicate or the Manage Systems task received the highest priority, the Aviate task was always executed.

Figure 4.27 Observed Frequency of Task Execution (Y-axis) When the Estimated First Priority (FP) was Aviate, Navigate, Communicate, and Manage Systems Tasks

118

When the participants reported the Navigate task as the highest prioritized task, it did not mean that the participants focused only on the Navigate task while abandoning

Aviate, Communicate, and Manage Systems tasks. Instead, it increased the chance of observing Navigate task execution (Figure 4.28). There were only 15 times when

Navigate task was reported as the highest priority and the participants were executing the

Navigate task 9 times (9/15=60%), which was a clear jump from when other tasks were the first priority; when Aviate was the first priority, the execution of the Navigate task was only 41%; when Communicate was the first priority, the execution of the Navigate task was 50%, etc. A similar trend was observed when participants prioritized

Communicate or Manage Systems tasks.

Figure 4.28 Normalized Data of Observed Executing Tasks at the Frozen Moments The Perceived task priority and Task Performance

119

The relationship between the perceived task priority and the associated task performance was investigated. Aviate task performance was measured by the altitude deviation from the ideal altitude during flight cruising phase (situation three and situation four). Similarly, Navigate task performance was measured by the lateral deviation from the ideal path at flight situations one, two, and four. A smaller deviation from the target altitude or flight path was regarded as a better task performance, while larger deviation was regarded as a compromised task performance.

The left graph in Figure 4.29 shows the scatter plots of altitude deviation for

Aviate task performance, and the right graph in Figure 4.29 shows the lateral deviation for Navigate task performance. In the Aviate task, a weak linear relationship was observed between the altitude deviation and the perceived Aviate task priority

( P  Value  0.07 and R2  0.16) . When the perceived Aviate task priority became higher, less altitude deviation was observed. On the other hand, there was no significant relationship between the perceived Navigate task priority and the lateral deviation (P- value=0.4, R2  0.01) in this study.

120

Figure 4.29 Mean Altitude Deviation as the Aviate Task Performance (Left), and Mean Directional Deviation as the Navigate Task Performance Measures

121

Communicate task performance was measured by confirming if the participant appropriately responded to ATC calls, which was determined by reviewing the recorded video. Fourteen out of sixteen participants responded to ATC requests appropriately on the Communicate task while one participant who is a none-native English speaker and one senior participant had a hard time on the Communicate task. There was no significant evidence of any relationship between the estimated Communicate task priority and ATC communication performance.

Manage Systems task performance was measured by confirming if the participant conducted a checklist task at the required flight points. Figure 4.30 shows the relationship between the estimated Manage Systems task priority score (Y-axis) and execution of checklist subtask condition (X-axis). No significant difference of the perceived task priority of Manage Systems task score was observed between failure to execute condition and execute conditions.

Figure 4.30 Perceived task priority Score When Checklist was Executed

122

Summary of the Result of RQ-3

The third research question was “RQ-3) Can actual task execution and task performance be explained by the perceived task priority?” Task execution was investigated by reviewing the simulation-frozen moment in the recorded video, and confirming if participants conducted the task. In 89% of the time, the participants executed the highest prioritized task at the frozen moment. In 120 instances of simulation-frozen points, the Aviate task had the highest perceived task priority (95 times), followed by Navigate (15 times), Communicate (6 times) and Manage Systems tasks (4 times). This trend was consistent with the observed executed tasks.

The relationship between the task performance and the perceived task priority was investigated for each task. For Aviate and Navigate tasks, the altitude deviation and lateral deviation from the ideal altitude and path respectively, were used for estimating the task performances. Aviate task performance was weakly related to the estimated

Aviate task priority score, while there was no significant relationship between the

Navigate task performance and the estimated Navigate task priority. Communicate task performance was measured by confirming if the participants appropriately responded to the ATC requests. There was no significant relationship between the estimated

Communicate task priority and the Communicate task performance. Manage Systems task performance was measured by observing if the participants conducted a checklist subtask at the instructed flight situations. There was no statistically significant relationship between the estimated Manage Systems task priority and Manage Systems task performance. In summary, the participants were likely to execute the task which had a high perceived task priority. However, the performance of Navigate,

123

Communicate, and Manage Systems tasks were not clearly related to the perceived task priority.

124

SUMMARY OF MAIN FINDINGS

This experiment attempted to answer the three questions: “which factors affect task prioritization behavior in a time-pressured multitasking environment”. As Figure 4.31 shows, this experiment tested the above research question on each of the three cognitive stages: perception and situation awareness (SA), response selection stage (i.e., decision- making or DM stage), response execution (RE) stage.

Figure 4.31 Three research questions and potential task prioritization factors at each stage

The result of RQ-2 addressed the task priority behavior in the perception and situation awareness (SA) stage. SA stage is critical for task-prioritization because if a pilot does not notice a task-related signal, he/she would not even think of prioritizing the signal-related task in the decision-making stage. The obtained recorded video revealed that the signal salience factor and signal expectation factor had significant effect on noticing the task-related signals, but these effects disappeared as the number of concurrent tasks increased. When the signal was both expected and salient, the

125 participants could notice the signal even when they were engaged in concurrent tasks.

When the perceived task priority was high, the participant could notice the task-related problem signals within a shorter time period compared to low perceived task priority condition. Thus, signal salience, signal expectation, and number of concurrent tasks have an indirect impact on task prioritization decision because these factors would affect a pilot for noticing important signals.

The result of RQ-1 addressed the task-prioritization behavior in Decision-Making

(DM) stage with probe-questions. The DM stage is critical as it is where participants actually make decisions on which task should be prioritized. Data obtained from the questionnaires revealed that the task importance (top-down factor) and task salience

(bottom-up factor) were significant factors for the task priority. The task urgency factor was weakly related to the task priority in general, but it became significantly related to the task priority score when the participants assessed their task performances as satisfactory. However, the perceived task performance and the perceived task workload factors did not have significant relationship with the perceived task priority. It was concluded that the perceived task importance, perceived task urgency, and the perceived task salience were critical factors for the perceived task priority in decision making stage.

The result of RQ-3 addressed the task-prioritization behavior in the response execution (RE) stage. The response-execution stage is important because multitasking may delay or defer the task execution against the pilot’s intention, and the observer may regard it as wrong task-prioritization behavior. The observed task execution was consistent with the perceived task priority. In the relationship between the observed task

126 performance and the perceived task priority, only the Aviate task had a weak relationship while Navigate, Communicate, and Manage Systems did not have enough evidence of relationship. Thus, the perceived task priority related to the observed task execution, but it did not relate to the observed task performance in this study.

Comparing the result of RQ-2 (SA) and RQ-1(DM), the following characteristics were observed. In both SA and DM stages, signal-salience factor, and task-salience factor were significant as the bottom-up factors. Also, in both SA and DM stages, the signal-expectation factor, and the task importance factor were significant as the top-down factors. In the SA stage, the number of observed concurrent tasks was a significant negative factor in the signal detection, while in the DM stage, there was no significant relationship between the perceived workload and perceived task priority based on the probe-questionnaire data.

127

5. CHAPTER 5: DISCUSSION

In this flight simulation study, the obtained questionnaire data indicated that perceived task importance and perceived task salience were significantly related to perceived task priority. Perceived task urgency was weakly correlated to the estimated task priorities. Recorded video and flight recorder revealed that task-related signal salience and expectancy factors will reduce the unnoticed time period of flight instrument problem signals. The number of observed concurrent tasks had a negative effect on the

SA stage and the task execution stage. Then, why did participants not always notice the task-related signals even when the perceived task priority was high? Why did participants not always execute the task even when the perceived task priority was high?

This chapter discusses the possible mechanisms of task prioritization behavior using both the significant factors from the main findings and additional possible factors raised by the above two questions. The first discussion point addresses why participants did not always notice the signals with high task priority. The second discussion point addresses why participants did not always execute tasks with high task priority. The third discussion point addresses why the perceived task performance status and workload did not relate to the perceived task priority. The fourth discussion point presents the integrated model of task prioritization behavior using the main-findings and possible additional factors. Finally, the assumptions and limitations of this study are discussed and future work is recommended.

128

WHY DID PARTICIPANTS NOT ALWAYS NOTICE THE PROBLEM SIGNALS EVEN WITH HIGH TASK PRIORITY

Comparing the obtained questionnaire data and observed data indicated that even when participants reported to allocate high task priorities (i.e., high attention resource), they had problems in noticing task-related signals. An impaired awareness of task- related signals could be a big problem in task-prioritization at the decision stage. In the following subsection, possible mechanisms are discussed for how participants might have experienced difficulty in noticing task-related signals.

Cockpit Display Design and Visual Sampling Frequency

A possible reason for inattention/change blindness under high task priority might be due to the many separated instruments in the cockpit instrument panel design. Figure

5.1 shows the screen shot of the flight instrument panel used in this experiment.

Participants needed to monitor six separate instruments for the Aviate task, one instrument for the Navigate task, and six instruments for Manage Systems tasks. When a pilot prioritizes Aviate task, they have to monitor and control the following six subtasks, each with its own instruments: altitude from ground, airspeed, attitude, altimeter, vertical speed, and turn coordinator indicators. Because these instruments are separated, participants need to read each instrument one by one sequentially.

129

Figure 5.1 Screen Shot of Analog Flight Instrument Panel of Cessna 172 RG

Diagram A in Figure 5.2 shows a conceptual Gantt chart of simplified sequential scanning tasks, where a participant executes each scanning task one by one. Because the participants sequentially focused on each individual instrument with a focal vision, a blind period would deterministically occur in the other tasks. For example, at the moment when attending to the monitoring and controlling of airspeed subtask, it would be difficult for a pilot to notice an abnormal signal in the altimeter or vertical speed indicator. Moray (2003) claims that a failure in monitoring occurs even at the eutectic visual sampling frequency (the optimal, or at least satisfactory minimal sampling frequency). In case a signal event occurs during the visual sampling interval, then the operator will miss it. According to Moray (2003),

130

“the only way to guarantee that all signals will be detected is to

monitor only one source… but in most real systems, attention must be

distributed over many sources” (176).

Moray claimed that it is necessary to equip cockpits with alarms and warning devices because it is difficult for operators to notice a signal at the required level even when they scan visually with optimal sampling frequency.

Figure 5.2 Time Chart of Sequential Scanning in Analog Instrument Panel, and Concurrent Scanning in Integrated Digital Instrument Panel

131

Another approach for solving this problem is to integrate the several instruments into a single display. For example, a digitized “glass cockpit” display that integrates several instruments including the horizontal indicator, airspeed indicator, VMI, altimeter, and vertical speed indicator, etc. Diagram B in Figure

5.2 shows a Gantt chart of simplified concurrent scanning task with glass cockpit display, where a participant executes multiple scanning tasks simultaneously.

Because a pilot can use ambient vision in addition to the focal vision, the blind period is reduced. This is a possible mechanism to explain why participants did not always notice the task related signals with high task priority.

132

Concurrent Multitasking Weakened Situation Awareness

According to the obtained data, inattention blindness or change blindness might occur stochastically during multitasking even when perceived priority of the tasks was high. In the main findings, Principal Component Analysis (PCA) indicated that participants may want to set high urgency on two or more tasks simultaneously.

Consider a scenario where there is only one task that needs to be executed in

Figure 5.3. Since people often prioritize a task when it is more urgent (i.e., short buffer time), the priority graph has a negative slope over the buffer time (Case-2 in Figure 5.3).

On the other hand, Case-1 in Figure 5.3 has a positive slope, which may indicate a wrong decision “urgency is not critical for task priority”. Thus, the negative slope of task priority is often ideal when there is a single urgent task. As a matter of fact, the participants reported that each task has a negative priority slope (Figure 5.4)

Figure 5.3 The relation between task Priority (Y-Axis) and task buffer time (X-axis)

133

Figure 5.4 Negative slope trend of task priority with task urgency in four tasks

Now, consider a scenario where multiple tasks need to be executed, and as reported by pilots all tasks have a negative slope in Figure 5.5. In this situation, pilots may execute concurrent multitasking while they may not be able to anticipate the possibility of inattention/change blindness.

Figure 5.5 Conceptual sketch of task priority and urgency relationships when multiple tasks have negative slopes simultaneously

134

Figure 5.6 shows how the awareness of the task-related signals was affected by both the perceived task priority score and the number of observed concurrent tasks in airspeed indicator malfunction problem. The size of each circle is proportional to the unnoticed time period of cockpit malfunctions. The X-axis indicates the mean number of tasks, and the Y axis indicates the perceived Aviate task priority score. In the graph, the large bubbles were observed even when the perceived task priority scores were high.

There was not enough evidence to support the hypothesis that the task priority score would shorten the unnoticed time period (P-Value=0.69) after taking individual difference into account in the airspeed indicator malfunction problem. On the other hand, when participants were engaged in a single task (Aviate task), the unnoticed time was constantly small. As the number of concurrent tasks increases, the inattention / change blindness time increased (P-Value=0.02) even after taking individual difference into account. This may indicate that even when participants prioritized the Aviate task, they might not be able to anticipate or be aware of all the Aviate task-related instrument problems during concurrent multitasking.

135

Figure 5.6 The effect of perceived Aviate task priority (Y-axis) and the mean number of tasks toward the problem-unnoticed time period (the bubble sizes).

136

Different Visual Scanning Pattern

When the participants were engaged in a single task, they could notice malfunction signals more easily in the following problems: zero speed on the airspeed indicator (Figure 5.6), wrong direction on the RMI (Figure 5.7), low fuel, low vacuum pump pressure (Figure 5.9), and low altitude indicator (Figure 5.10). This is consistent with Endsley (2000)’s situation awareness theory; situation awareness needs working memory, and if working memory is occupied by concurrent tasks, it takes more time for pilots to notice the occurrence of malfunctions, or pilots do not notice it at all.

137

Figure 5.7 Bubble Plot Proportional to the Unnoticed Time (Seconds) in Navigation Indicator Problem

138

Figure 5.8 Bubble Plot Proportional to the Unnoticed Time (Seconds) in Low Fuel Problem

139

Figure 5.9 Bubble size Proportional to the Unnoticed Time (Seconds) for Vacuum Pump Indicator Problem

140

Figure 5.10 Bubble Plot Proportional to the Unnoticed Time (Seconds) for Altimeter Problem

141

On the other hand, the unnoticed time distribution of the horizontal/attitude indicator had a peculiar pattern of unnoticed time distribution. For the artificial horizon problems, participants noticed malfunction signals in shorter time periods when they were engaged in more concurrent tasks (Figure 5.11). One possible mechanism is that the multiple tasks may force the participants to look around and therefore increase the chances of noticing unexpected events in bottom up processing. Thus, visual scanning pattern may also affect the awareness of task-related signal. This is another possible mechanism for why task priority decision alone may not explain why participants did not always notice malfunction signals.

142

Figure 5.11 Bubble Plot Proportional to Unnoticed Time (Seconds) in Attitude/Horizontal Indicator Problem

143

Required Situation Awareness Level for Noticing Instrument Malfunction Signals

Table 5.1 summarizes awareness levels of task-related signals under four situations: Hit, Miss, False Alarm, and Correct Rejection. When a malfunction problem occurred, there were two possible outcomes; which were categorized as “Hit” if a participant noticed it, or “Miss” if a participant did not notice it (i.e., awareness of task- related signals problem). Similarly, there were two outcomes when a problem did not happen, categorized as “Correct Rejection” when a participant did not report the problem, or “False Alarm” when a participant reported it. The parentheses under the Miss and

False Alarm columns in Table 5.1 indicate the odds ratios of the errors against the correct actions.

The first four problems (1 to 4) required situation awareness level-1 (Endsley,

1995a); a participant only needed to perceive a malfunctioning signal, e.g., an indicator needle pointing in an erroneous direction.

The middle three problems (5 to 7) required situation awareness level-2 (Endsley,

1995a); not only the signals of problems be noticed, but participants would also have to integrate several noticed signals to understand the existence of the problem. This process required consciously manipulating the noticed information in working memory.

For example, participants had to look out of the cockpit window to check if the artificial horizon indicator might be wrong. Thus, the odds ratios of Miss increased in problems 5 to 7.

144

The last two problems (8 and 9) were the most difficult level-2 situation awareness problems. Because there was no visual cue, participants would have to notice associated signals and integrate them. For example, there was no indicator of static port clog; therefore, a participant needed to notice the subset of signals; zero level of vertical speed, slow airspeed, and strange behavior of the altimeter. Thus, the types of problems that require high level of situation awareness may also explain why participants did not always notice the problem signals even with high task priority.

Table 5.1 Summary of Awareness of task-related signals Problems

Problems occurred Problems did not occur Flight Instrument Problem False (Related Task) Hit Miss Correct Alarm (Odds Ratio) Rejection (Odds Ratio) 1. Airspeed Indicator 68 11 45 1 (Aviate) (0.16) (0.02) 2. Navigation Indicator 15 1 107 2 (Navigate) (0.07) (0.02) 3. Low Fuel 31 14 80 0 (Manage Systems) (0.45) (0) 4. Low Vacuum pump pressure 11 5 104 5 (Manage Systems) (0.45) (0.05) 5. Altitude Indicator 16 0 109 0 (Aviate) (0) (0) 6. Horizontal or Attitude Indicator 13 3 109 0 (Aviate) (0.23) (0) 7. Vertical Speed 9 7 107 2 (Aviate) (0.78) (0.02) 8. Pitot tube clog 10 54 59 2 (Manage Systems) (5.4) (0.03) 9. Static port clog 6 10 104 5 (Manage Systems) (1.67) (0.05)

145

Interrupting Tasks

Even when the perceived Aviate task priority score was higher than that of the

Communicate task (mean Aviate task priority was 0.50 point while the mean estimated

Communicate task priority was 0.12 point), the Communicate task was always executed immediately every time (i.e., participants responded to ATC calls immediately).

Furthermore, deterioration in Aviate task performance was observed when participants were engaged in the Communicate task. The abrupt nature of ATC calls might pull a participant’s attention, and lead the participant to start concurrent multitasking of Aviate and Communicate tasks. This might generate cognitive resource interference between

Aviate and Communicate tasks. Data showed that every 10 seconds of listening to

ATC calls increased the time period of inappropriate awareness of task-related signals

(inattention / change blindness) by 1.0 second (Wald’s Test P-Value<0.01). However there was no significant interference from the participant’s speaking activity (Wald’s Test

P-Value=0.14).

This fact may indicate the difficulty of prioritizing attention between Aviate and

Communicate tasks; participants might have difficulty in controlling their attention resources if two tasks have different channels (e.g., visual and audio). Auditory stimuli capture attention more than visual ones (e.g., Banbury, Macken, Tremblay &Jones,

2001). Strayer, Drews and Johnston (2003) reported that auditory tasks with mobile phone usage generated inattention blindness in driving tasks. McCarley, Vais, Pringle,

Kramer, Irwin and Strayer (2004) reported that change-blindness might be the result of curtailed time available for perceptual analysis and saccade eye movement planning.

146

They collected saccade data along with participants’ reports of noticed changes in pictures as well as their response times. Participants’ eye movements had smaller fixation times and larger numbers of saccades during concurrent multitasking of a vigilance task and conversation. Therefore McCarley et al., (2004) considered that peripheral guidance of attention toward the target was impaired because participants had to increase the numbers of saccades during the conversation. Thus, interrupting Communicate task may also impair the perception of prioritized task signals.

147

WHY DID PARTICIPANTS NOT ALWAYS EXECUTE THE PRIORITIZED TASKS?

According to the obtained questionnaire data, participants reported the Aviate task as the most prioritized task at 95 out of 120 (79%) of the simulation-freeze points, followed by Navigate (13%), Communicate (5%), and Manage Systems (3%) tasks.

While participants executed the highest perceived priority task in 107 out of 120 (89.2%) of the simulation freeze points, participants were not observed to execute the high priority task in 13 (10.8%) cases. For example, participant No.13 reported Communicate task as the highest priority at situation-3, while he attended the Aviate task at the freeze moment in the simulation. Why didn’t the participant attend to the Communicate task, but instead attended to the Aviate task? In the following subsection, the possible mechanisms are discussed for the inconsistency between the perceived task priority and executed tasks.

Top-Down Habitual Task Priority

When the participants did not execute the highest reported priority task, they might have been following their habitual task priority order. One possible reason for such a behavior is the ANCS hierarchical control loop. Cummings, Bruni, Mercier, and

Mitchell (2007) modeled ANCS control loops for a single Unmanned Arial Vehicle. The inner-most loop “Pilot” and “Flight Controls” correspond to the Aviate task that involves motion control where operations focus on the short term control. Cummings et al.

(2007) claim this loop involves skill-based behaviors (Rasmussen, 1983). The second

148 inner loop “Navigation” corresponds to the Navigate task, and the outermost loop is the highest level control loop that involves knowledge based reasoning (e.g., judgment, and abstract reasoning). If pilots follow this loop, the Aviate task is always executed because the Aviate loop is included in the Navigate loop, and attendance to the Navigate task always involves attendance to Aviate task.

The ANCS hierarchical loop hypothesis is consistent with the obtained shorter buffer time for the Aviate task. Participants reported the mean buffer time of Aviate,

Navigate, Communicate, and Manage Systems tasks as 8.9, 21.5, 31.6, and 25.9 seconds, respectively. Thus, even when the Navigate task was prioritized they would return to attending the Aviate task.

The above evidence indicates that pilot’s task prioritization behaviors may follow

Bayesian decision theory; the prioritized task is the result of combining events/data- driven momentary intention and the ANCS habitual prior behavior (Figure 5.12). It is well known that when people rationally make decisions that maximize their utilities, they often use heuristics that approximate the Bayesian rule; people combine prior knowledge or experience from top-down information processing with perceived information from bottom-up information processing (e.g., Körding & Wolpert, 2006; Chater and Oaksford,

1999). In the study of Colvin Funk, and Braune (2005), ANCS habitual prior behavior may corresponds to the “procedure task prioritization criterion”.

149

Figure 5.12 Task Prioritization in Bayesian Manner

There might be several reasons why pilots chose the ANCS hierarchical loop in their task prioritization behaviors. Schutte and Trujillo (1996) reported an ANCS based monitoring strategy (i.e., awareness management of task-related signals) that provided a diversity of useful information in solving Manage Systems problems for pilots during their flight experiment; pilots were able to notice important signals when they cycled the monitoring of task-related signals in order: Aviate, Navigate, Communicate, then Manage

Systems. On the other hand, Schutte and Trujillo (1996) reported that ANCS based workload management (i.e., task execution management) was not effective in dealing with trouble-shooting in a Manage Systems task because Manage Systems has the lowest task priority in ANCS. Schutte and Trujillo (1996) reported that perceived severity- driven workload management (pilots prioritize the task according to the perceived most threatening problem) and event/interruption driven workload management (pilots prioritize tasks based on an event or interruption) outperformed other task management styles.

The above discussion leads to a hypothesis: “concurrent multitasking in time- pressured environments may also drive pilots to execute tasks in ANCS priority order”.

150

This is because multitasking occupies the pilot’s attentional resources and working memory (Tsang, & Vidulich, 2006). Without enough working memory, pilots may not be able to consider critical task prioritization factors. For example, in order to know the urgency of a task, pilots needs to know the estimates of required time to execute and deadline of the task (Wickens et al. 2013). Estimation of required time and deadline needs a projection of future status (Level-3 SA), which cannot be conducted without enough working memory capacity (Endslay, 1995a). The fixed task priority may be harmful to appropriate and dynamic attention control, because Kramer and Willis (2002) reported people with variable task-prioritization training outperformed those who trained with fixed task-priority in executive functions, particularly among senior people.

In summary, task prioritization execution behavior may follow a Bayesian model; actual task prioritization behavior may be a combination of both perceived task priority and prior habitual task priority. This might be the possible mechanism why participants did not always execute the estimated high priority task.

151

Forgetfulness and Prospective Memory

Participants failed to execute the checklist task (a subtask of Manage Systems) in

41 out of 80 times (51%) during the simulation. The following three possible reasons were considered from the task prioritization perspective. The first hypothesis is that participants might have intentionally deferred executing the checklist task. This is because the mean reported buffer time for the Manage Systems task was 2.4 times longer than the mean reported buffer time for the Aviate task. Participants attended to the

Aviate task first after postponing the checklist (a subtask of Manage Systems) task. This implies the conversion of concurrent multitasking to sequential multitasking. Figure 5.13 shows the percentage of participants who did not execute checklist tasks during the five experiment situations (situation 1,3,5, 7, and 8). As the number of concurrent tasks increased, the checklist task execution decreased. This might be because participants intentionally deferred the checklist task or they may have forgotten it.

Pilots need to execute many tasks during taxiing, climbing, cruising and landing.

The flight manuals, guidance and personal experiences may influence the pilot’s intention in executing those tasks. Those intentions should not be forgotten until the appropriate moment with a memory. Such a memory is called prospective memory (e.g., Guynn,

Mcdaniel, & Einstein, 1998). It is challenging for participants to remember task initiation intentions (“latent” task status in CTM) or intentions for resuming suspended tasks (“pending” task status in CTM) at the appropriate moment. The pilot participants in

Colvin et al. (2005)’s flight simulation study reported “resist forgetting” as another potential task prioritization factor. For example, Spanair Flight 5022 crashed in 2008

152 after taking off with the flaps and slats retracted because pilots omitted the “set and check the flap/slap lever and lights” item in the checklist (CIDEA, 2015). Nowinski,

Holbrook, and Dismukes (2003) reported that 74 out of 75 memory failure problems were prospective memory failures. They randomly sampled 20% of all air carrier reports contained in the Aviation Safety Reporting System (ASRS) for 12 month period.

Dismukes and Nowinski (2006) claim that prospective memory failures would lead to execution failure in the following tasks: 1. episodic tasks (participants suspend some tasks for later resumption, and this task is not a habitual task), 2. habitual task (highly habitual task for experienced pilots), 3. atypical actions substituted for habituated actions

(tasks deviated from a well-established procedural sequences), 4. interrupted tasks (those tasks needed to be remembered to resume them), 5. interleaving tasks . Thus, participants did not always execute the prioritized task because participants might forget to execute the prioritized tasks.

Figure 5.13 The relationship between the proportion of failure of checklist task execution and the mean number of concurrent multitask

153

WHY AREN’T THE PERCEIVED TASK PEROFRMANCE STATUS AND WORKLOAD RELATED TO THE PERCEIVED TASK PRIORITY?

In the result for RQ-1, why weren’t the following two factors related to the perceived task priority score: the perceived task performance status score, and perceived task workload score?

The first possible reason is that the flight simulation scenario might not have been challenging enough to force the participants to consider the performance status and workload in this experiment.

The second possible reason is the innate limitation of the retrospective sampling with probe-questions using questionnaires. Ideally, the five candidate factors should be manipulated as causal factors prior to observing the perceived task priority as the result factor. In this experiment, the perceived task priority and five candidate factors were obtained all together in a “screen-shot” manner at the simulation-freeze moment; the participants attempted to recall which task was prioritized and rate each five candidate factor based on their memory. Because of possible confounding factors (e.g., forgetfulness), it is speculative to make a casual inference of the cause-and-effect relationship (Ramsey & Schafer, 2012). In order to conquer this problem, it is recommended to design an experiment that controls the task performance status and task workload factors with assistance of aviation domain experts. In this experiment, the above methodology was not employed because of the limitation of calibrating the explanatory factors. For instance, even if an experiment is designed so that task A is

154 supposed to have a better task performance than task B, some participants may interpret this differently (e.g., task B has a better task performance than task A). In order to effectively calibrate the perceptions of all five candidate factors, a long and intense collaboration with domain experts (expert pilots) would be needed.

The third possible reason is that participants might prioritize tasks regardless of their perceived performance status or workload. The obtained experiment data indicated that the participants almost always prioritized the Aviate task regardless of the perceived task performance status and the perceived task workload. Furthermore, the time spent on each task was approximately proportional to the perceived task priority score (See

Appendix I). This result is consistent with Raby and Wickens’ (1994) report that the operator’s task priority affects the task execution time, but high task priority did not necessarily lead the operators to immediately switch to the high priority task. This may indicate that pilot’s task prioritization could be conducted in a Bayesian manner

(e.g.,Körding & Wolpert, 2006) and the prior knowledge and experience (i.e., ANCS priority order) may be more influential than the stimulus and event drive factor (i.e., the perceived task performance status or the perceived task workload).

155

INTEGRATED MODEL OF TASK PRIORITIZATION BEHAIVOR

In this subsection, the main findings and the above discussions are integrated to construct a model for task-prioritization behavior in a time-pressured multitasking environment. Figure 5.14 shows a conceptual diagram of task-prioritization behavior model that depicts how the information of Aviate and Communicate tasks are processed in the situation awareness, decision-making, and execution stages. The framework for this model is based on Endlsey’s situation-awareness model (Endsley 1995a; Endsley

2000) and SEEV model (Wickens, Goh, Helleberg, Horrey, & Talleur, 2003).

According to Endlsey’s (1995a) situation awareness model, pilots perceive signals in the Level-1 situation awareness stage. Jones and Endsley (1996) reported that most of the situation awareness errors occur in the perception stage (76.3% were from Level 1

SA), and that the perception of signals is critical part in task prioritization. For example, in the Eastern Air Lines Flight 401 accident, the pilots did not notice the low altitude before the crash (NTSB, 1973) which was regarded as a task prioritization error

(Colvin et al., 2005).

In this study, signal-salience and signal-expectation factors were confirmed as critical factors in noticing signals. When both factors were present, the participants were able to notice the flight instrument malfunction signals even when they were engaged in concurrent multitasking.

156

Figure 5.14 Task Prioritization Behavior Model

157

In Level-2 situation awareness (SA) stage, pilots attempt to understand the current situation by synthesizing the disjoint perceived information in the light of a perceived goal (Endsley, 2000). Understanding the current situation is important as Jones and Endsley (1996) reported that 20.3 % of SA problems occur in Level 2 SA. Endsley

(2000) claimed that the goal and its associated mental model will be used for synthesizing the perceived information. In this experiment, the task importance factor was used to determine which goal (i.e., Aviate-related goal, Navigate-related goal, Communicate- related goal, and Manage Systems-related goal) might be used in the Level-2 SA. For example, if a pilot perceived the Aviate task was important, then the perceived signals that relate to the Aviate task should be more likely to be synthesized and noticed. In fact the unnoticed time of task-related signals was shorter as the perceived task importance increased (see Appendix I), which is consistent with the above theory4. Thus, the task importance factor can be a good measurement of the goal and its associated mental model which is used for synthesizing the perceived information.

In Level-3 situation awareness stage, pilots attempt to project the future status of the elements in the environment (Endsley, 1995a). Jones and Endsley (1996) reported that 3.4% of SA errors occur in Level-3 situation awareness stage. In this experiment, task urgency factor was used to measure the pilot’s projection of the task status, and the

4 Here, it should be noted the limitation of this study. While the experiment result was consistent to the above theory, the perceived task importance information was retrospectively observed with probe questions. Because of possible confounding factors, statistical significance cannot be generalized. In the future study, task importance factor should be controlled by experimenter, and randomly assign the participants to the different level of task importance conditions in order to observe a reliable effect of task importance.

158 perceived task urgency had statistically significant linear relationship with the perceived task priority score. Thus, task-urgency factor is used for measuring the pilot’s situation awareness of the tasks at near future.

In this model, the number of concurrent tasks is regarded as the cost factor in task prioritization. In this experiment, adding one task increased the unnoticed time of malfunction signals by a factor of 2.7 times. This is consistent to many studies pertaining to the relationships between the situation awareness and workload (e.g., Tsang

& Vidulich, 2006; Wickens, 2002b).

159

The above model indicates that task prioritization is actually a progressive behavior in the following six processes. First, expectation and salience factors influence the perception of signals in the Level-1 SA (perception of elements in the current situation) stage. People first process the stimulus information sequentially from the most salient to the least salient before considering the importance of the criteria associated with the stimuli in a Bayesian manner (Wallsten & Barton , 1982). Here, the important point is that pilots are not always aware of what they do not notice. A failure to notice a critical signal may then beget a wrong understanding of the current situation that may also beget a wrong task-prioritization decision. In addition, the perception of signals may be affected by the cockpit display design (i.e., task design), visual sampling frequency, visual scanning pattern, and difficult problems that require high levels of SA.

In the hypothetical example ① in Figure 5.14, it takes a long time for a pilot to perceive the true altitude of 500 feet since 3000 feet was expected in the pilot’s long-term memory

(i.e., mental model). In the hypothetical example ② in Figure 5.14, the pilot immediately recognizes the ATC altitude clearance of “1000 feet” as it is both expected and salient in the Communicate task.

Second, in Level-2 situation awareness, the goal in mind and associated mental model in the pilot’s long-term memory is used to synthesize the perceived information in order to comprehend the current situation (Endsley, 1995a). In the CTM theory, “1.

Create initial agenda” corresponds to the goal and the associated mental model (Funk,

1991). Endsley (2000) claimed that pursuing a wrong or less important goal will lead to a failure to recognize critical information. An interrupting Communicate task may also force the pilot to switch to the other goal(s) for synthesizing the perceived information to

160 the Communicate task and as a result the pilot may have difficulty synthesizing the perceived elements that relate to the Aviate task. Thus, a wrong understanding of the given situation may lead to inappropriate task prioritization decision. In the hypothetical example ③ in Figure 5.14, a pilot successfully synthesizes the two pieces of information in the light of the goal “safe flight”: the information of perceived current altitude of 500 feet from Aviate task, and the perceived required altitude of 1000 feet.

Third, in order to appropriately understand the current situation pilots need to integrate goal-directed top-down and stimulus-driven bottom-up processing. In the CTM theory, “2.a access current situation”, and “2.c assess the status of active tasks” may correspond to this stage (Funk, 1991). Endsley (2000) claims that the failure to alternate bottom-up and top-down processing will negatively impact the Level-2 SA.

This may happen as the result of pilots either failing to choose an appropriate goal, or pilots failing to recognize the perceived elements. Richard, Wright, Ee, Prime, Shimizu, and Vavirik (2002) reported that in visual search performance, dual task (concurrent multitasking) had a significant effect on search performance degradation; top-down knowledge-based approach did not always have a significant effect on visual search performance improvement. Richard et al., (2002) claimed that the effect of concurrent multitasking eliminated the effect of top-down/knowledge assistance.

Fourth, pilots need to predict the future status of the tasks in Level 3 situation awareness. A failure of task prediction may lead to a wrong task prioritization decision.

Here, the pilot may not be able to anticipate the possibility of inattention/change blindness with divided attention if they conduct concurrent multitasking. In the example ④ in Figure 5.14, a pilot may predict that an aircraft will crash in 1 minute.

161

Fifth, pilots need to execute the prioritized task at the task execution stage shown as ⑤ in Figure 5.14. Pilots may not be able to execute the prioritized task when their prospective memory is impaired by interrupting ATC calls or high workload.

Sixth, a high level of workload may have a negative effect on all of the above progressive behaviors in the task-prioritization process. In the CTM theory, this may create problems in the “assess task resource requirements”. In the hypothetical example

⑥ in Figure 5.14, the pilot is unaware of the status of Navigate and Manage Systems tasks due to the lack of attention resource and working memory.

The above task-prioritization behavior model is conceptually consistent with other cognitive models. In particular, the SEEV visual attention model developed by Wickens et al.,(2003, 2008) has four similar factors: cue Salience of the task, Expectation of a signal, Effort (cost of moving attention), and Value of the information. The SEEV model predicts where participant’s eyes focus on within the visual workspace. In this study, awareness of task-related signals was successfully predicted by using the four factors:

Salience, Expectancy, Effort (i.e., the number of concurrent tasks) and Priority, which is referred to here as the SEEP model. There are several distinct differences between SEEP and SEEV. The primary difference is the response variable. SEEP models how long it takes to notice the task-related signal, while SEEV models how much time a participant’s focus dwells on a particular area of interest (AOI). Because SEEV provides a prediction of visual scanning, it may be useful for predicting the visual attention allocation which is important for predicting vulnerability to missing roadway hazards (Horrey, Wickens, andConsalus, 2006), or designing optimal display layout where sequential visual

162 scanning of instruments requires that they should be placed close together (Wickens,

Goh, Helleberg, Horrey, & Talleur, 2003). On the other hand, SEEP may be useful for estimating the hazard probability for designing a multitasking environment that requires an operator to maintain good situation awareness while conducting the main tasks.

163

ASSUMPTIONS AND LIMITATIONS OF THIS STUDY

There were some limitations on this study. In order to freeze the flight for collecting information on the perceived task priority and the perceived task prioritization criteria at each situation, experiments could not be conducted in a real airplane. Still, the flight simulator realism was considered to be sufficient to capture data on the relationship between the perceived task priority score and the perceived task prioritization decision making factors. Despite the fact that data collection was not conducted in a real aircraft, data on multitasking behavior obtained from these flight simulations is considered to be sufficient for testing the research hypotheses.

Second, there was an innate limitation of retrospective data sampling. Because a questionnaire (recall-based memory probes) was used to assess the perceived task priority at the simulation-frozen moment, it was an observational and retrospective study; participants looked backwards from the moment of the frozen situation and answered the questionnaire questions about perceived task priority, urgency, performance, salience and workload. Here, memory interference (i.e., forgetfulness), hindsight bias, and anchoring bias may have affected the questionnaire data. In retrospective data collection, we do not know the real cause-and-effect relationship (e.g., Ramsey, & Schafer, 2012).

Finally, it must be assumed that participants honestly answered the questions based on the memory of the flight simulation at the frozen moment, yet some confounding factors might exist. Thirteen out of 120 (11%) were not consistent between the highest perceived prioritized task and observed task execution. Scrutinizing these 13 incidents of video record revealed the following possible confounding factors: 1.

164 forgetfulness, and 2. inexplicit task prioritization. For example, participant No.3 reported that the Navigate task was the highest priority at the situation-three freeze moment. However, he was not executing the Navigate task at the simulation-frozen moment. Scrutinizing the recorded video revealed that he was actually executing the

Navigate task at 30 seconds prior to the frozen point. Thus, participant No.3 might have been confused about whether he was still executing the Navigate task at the frozen moment. Furthermore, the participants might simply have kept thinking about the highest priority task without actually executing it. For example, participant No.13 reported the Communicate task as the highest priority at situation-3, while he attended the

Aviate task at the frozen moment. Scrutinizing the video revealed that there were many

ATC calls prior to the frozen point; he might have been thinking of the next incoming

ATC call. There was also a possibility that participants might interpret that the “task priority”, “task importance”, and “task urgency” were similar notions. However, the questionnaire clearly stated that the task priority is the priority of attention, while the task importance is the task importance and task urgency is about the buffer time for the task.

Overall, the obtained perceived task priority is regarded as reasonable estimate of the perceived task priority among participants since at 107 out of 120 simulation frozen points (89%), the perceived task priority and the observed executed tasks matched.

165

FUTURE STUDY

In a future study, a basic laboratory experiment could be designed with randomized task prioritization criteria as the cause factors for observing their effects on the perceived task priority (e.g., Multiple Task Battery simulator; Comstock & Arnegard,

1992). A careful calibration of the cause factors (e.g., the perceived task importance, the perceived task buffer time, and the perceived task salience) would be needed among all pilot participants for future research endeavors.

166

6. CHAPTER 6: CONCLUSIONS AND RECOMMENDATIONS

CONCLUSIONS

This research tested the hypotheses of Colvin et al. (2005) which state that perceived task priority could be related to the five candidate factors: perceived importance, urgency, performance status, salience and workload of tasks. Data was collected from sixteen participant using probe questions at eight frozen points during each simulated flight. Furthermore, it investigated if task execution time, task performance, and awareness of task-related signals could all be explained by perceived task priority.

The following are the main findings. Perceived task importance, perceived task urgency, and perceived salience were significantly related to the perceived task priority.

This result is consistent with Shakeri and Funk (2007), Colvin, Funk, and Braune (2005), and Wallsten and Barton (1982). The obtained data indicated that the participants were more likely to execute the tasks and notice malfunction signals within a shorter time when the task was highly prioritized. The increased number of concurrent tasks had a negative effect on both bottom-up processing (i.e., noticing the malfunction signal) and top-down processing (executing the checklist task). Participants did not always notice the task-related malfunction signals even when the task priority was high. If either signal salience or signal expectation were increased, each factor had an effect of enhancing the

167 pilot’s awareness of signals, but this effect disappeared as the number of tasks increased.

Only when the signal was both expected and salient could the participants robustly notice the malfunction signals during multitasking. Furthermore, even in cases when they were observed to have enough time participants did not execute the checklist tasks as the number of concurrent tasks increased. These findings are consistent with other multitasking studies: concurrent multitasking eliminates the benefits that result from alternating and integrating stimulus-driven bottom-up and goal-driven top-down processing, which is regarded as critical in task-prioritization.

RECOMMENDATIONS

This study provides the following recommendations to pilots. First, a pilot should recognize that we are not aware of what we do not notice during multitasking. In this study, the participants were not aware of signals of prioritized tasks especially under high workload condition. Thus, it is advisable to avoid a large number of concurrent tasks. Rescheduling the tasks may equalize the workload. Choosing integrated cockpit displays (e.g., glass cockpit displays) may also reduce the visual scanning workload by improving the visual sampling frequency and visual scanning pattern.

Second, the task-related salience factor and the task-related expectancy factor increase the chances of pilots noticing abnormal signals during the flight. This study also showed that it took longer times for pilots to notice the flight instrument problems that required a high level of situation awareness. Proper training with case studies of accidents, as well as understanding aircraft structure and mechanisms may improve the

168 right expectancy factor (i.e., mental model). Installing and deploying alarm functions

(e.g., stall, low fuel, etc.) may also improve the signal salience factor.

Third, solutions for both the salience factor (e.g., alarm installation) and the expectation factor (pilot training) should be implemented together because only when the right expectancy and salient signal conditions are met may pilots be able to notice the abnormal signals in a multitasking environment.

Fourth, in order to achieve good task prioritization, a pilot should be able to alternate goal-driven task priority (top-down) and stimuli-driven task-priority (bottom- up) in an appropriate way. Either missing an important signal or using inappropriate goal(s) (and associated mental model) may lead to a wrong understanding of the current situation which may then lead to a wrong task prioritization decision. Richard et al.

(2002) reported that the effect of concurrent multitasking eliminates top-down

/knowledge assistance. Pilots should consider the variable-priority training which was advocated by Bherer, Kramer, Peterson, Colcombe, Erickson, and Becic (2005). This could help pilots to improve their multitasking performance.

Finally, the SEEP attention model may be a useful tool for pilots and human factors engineers to estimate the hazard probability in multitasking environments.

169

BIBLIOGRAPHY

Abelson, R. P. (1981). Psychological status of the script concept. American psychologist,

36(7), 715.

Allport, D. A., Styles, E. A., & Hsieh, S. (1994). Shifting intentional set: Exploring the

dynamic control of tasks.

Altmann, E. M., & Trafton, J. G. (2002). Memory for goals: An activation-based model.

Cognitive science, 26(1), 39-83.

Anderson, J. R., & Lebiere, C. (Eds.). (1998). The atomic components of thought.

Psychology Press.

Baddeley, A. D. (2002). Is working memory still working?. European psychologist, 7(2),

85-97.

Baddeley, A. D. (2006). Working memory: An overview. Working memory and

education, 1-31.

Banich, M. T. (2009). Executive function the search for an integrated account. Current

Directions in Psychological Science, 18(2), 89-94.

Banbury, S.P., Macken, W. J., Tremblay, S., & Jones, D.M. (2001). Auditory distraction

and short-term memory: Phenomena and practical implications. Human Factors:

The Journal of the Human Factors and Ergonomics Society, 43(1), 12-29.

Bartlett, F. C., & Bartlett, F. C. (1995). Remembering: A study in experimental and social

170

psychology (Vol. 14). Cambridge University Press.

Bherer, L., Kramer, A. F., Peterson, M. S., Colcombe, S., Erickson, K., & Becic, E.

(2005). Training effects on dual-task performance: are there age-related

differences in plasticity of attentional control? . Psychology and aging , 20(4),

695.

Bishara, S., & Funk, K. (2002). Training pilots to prioritize tasks. In

Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol.

46, No. 1, pp. 96-100). SAGE Publications.

Brown, T. L., Lee, J. D., & McGehee, D. V. (2001). Human performance models and

rear-end collision avoidance algorithms. Human Factors: The Journal of the

Human Factors and Ergonomics Society, 43(3), 462-482.

Chater, N., & Oaksford, M. (1999). Ten years of the rational analysis of cognition.

Trends in Cognitive Sciences, 3(2), 57-65.

Chou, C. C., Madhavan, D., & Funk, K. (1996). Studies of cockpit task management

errors. The International Journal of Aviation Psychology, 6(4), 307-320.

CIDEA (2015). Report A-032/2008, Accident involving a McDonnel Douglas DC-9-

82(MD_82) aircraft, registration EC-HFP, operated by Spanair, at Madrid-

Barajas Airport on 20 August 2008, COMISION DE INVESTIGACION DE

ACCIDENTES E INCIDENTES DE AVIACION CIVIL

Clemen, R. T. (1996). Making hard decisions: an introduction to decision analysis.

Colvin, K., Funk, K., & Braune, R. (2005). Task Prioritization Factors: Two Part-Task

171

Simulator Studies. The International Journal of Aviation Psychology,

15(4), 321-338.

Comstock Jr, J. R., & Arnegard, R. J. (1992). The multi-attribute task battery for human

operator workload and strategic behavior research.

Cummings, M. L., Bruni, S., Mercier, S., & Mitchell, P. J. (2007). Automation

architecture for single operator, multiple UAV command and control.

MASSACHUSETTS INST OF TECH CAMBRIDGE.

Dismukes, R. K., & Nowinski, J. L. (2006). Prospective memory, concurrent task

management, and pilot error. Attention: From theory to practice, 225-236.

Dougherty, M. R., & Hunter, J. (2003a). Probability judgment and subadditivity: The role

of working memory capacity and constraining retrieval. Memory &

Cognition, 31(6), 968-982.

Dougherty, M. R., & Hunter, J. E. (2003b). Hypothesis generation, probability judgment,

and individual differences in working memory capacity. Acta psychologica,

113(3), 263-282.

Endsley, M. R. (1995a). Toward a theory of situation awareness in dynamic systems.

Human Factors: The Journal of the Human Factors and Ergonomics Society,

37(1), 32-64.

Endsley, M. R. (1995b). Measurement of situation awareness in dynamic systems.

Human Factors: The Journal of the Human Factors and Ergonomics Society,

37(1), 65-84.

172

Endsley, M. R., Farley, T. C., Jones, W. M., Midkiff, A. H., & Hansman, R. J. (1998).

Situation awareness information requirements for commercial airline pilots.

International Center for Air Transportation.

Endsley, M. R. (2000). Theoretical underpinnings of situation awareness: A critical

review. Situation awareness analysis and measurement, 3-32.

Falissard, B. (2011). Analysis of questionnaire data with R. CRC Press.

Fennema, M. G., & Kleinmuntz, D. N. (1995). Anticipations of effort and accuracy in

multiattribute choice. Organizational behavior and human decision processes,

63(1), 21-32.

Funk, K. (1991). Cockpit task management: Preliminary definitions, normative theory,

error taxonomy, and design recommendations. The International Journal of

Aviation Psychology, 1(4), 271-285.

Funk, K., Lyall, B., Wilson, J., Vint, R., Niemczyk, M., Suroteguh, C., & Owen, G.

(1999). Flight deck automation issues. The International Journal of Aviation

Psychology, 9(2), 109-123.

Gugerty, L. (2011). Situation awareness in driving. Handbook for driving simulation in

engineering, medicine and psychology.

Guynn, M. J., Mcdaniel, M. A., & Einstein, G. O. (1998). Prospective memory: When

reminders fail. Memory & Cognition, 26(2), 287-298.

Hambrick, D. Z., Oswald, F. L., Darowski, E. S., Rench, T. A., & Brou, R. (2010).

173

Predictors of multitasking performance in a synthetic work paradigm. Applied

Cognitive Psychology, 24(8), 1149-1167.

Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index):

Results of empirical and theoretical research. Advances in psychology, 52, 139-

183.

Horrey, W. J., Wickens, C. D., & Consalus, K. P. (2006). Modeling drivers' visual

attention allocation while interacting with in-vehicle technologies. Journal of

Experimental Psychology: Applied, 12(2), 67.

Iani, C., & Wickens, C. D. (2007). Factors affecting task management in aviation. Human

Factors: The Journal of the Human Factors and Ergonomics Society, 49(1), 16-24.

Johnson, J. W. (2004). Factors affecting relative weights: The influence of sampling and

measurement error. Organizational Research Methods, 7(3), 283-299.

Johnson, R. A., & Wichern, D. W. (2007). Applied multivariate statistical analysis.

Pearson Education Limited.

Jones, D.G., & Endsley, M.R. (1996). Sources of situation awareness errors in aviation.

Aviation, Space, and Environmental Medicine.

Kabacoff (2011). R in Action. Manning Publications

Kahneman, D. (2003). Maps of bounded rationality: Psychology for behavioral

economics. American economic review, 1449-1475.

Keinan, G. (1987). Decision making under stress: scanning of alternatives under

174

controllable and uncontrollable threats. Journal of personality and social

psychology, 52(3), 639.

Klein, G. A. (1993). A recognition-primed decision (RPD) model of rapid decision

making. Decision making in action: Models and methods, 5(4), 138-147.

Körding, K. P., & Wolpert, D. M. (2006). Bayesian decision theory in sensorimotor

control. Trends in cognitive sciences, 10(7), 319-326.

Konig, C. J., Buhner, M., & Murling, G. (2005). Working memory, fluid intelligence, and

attention are predictors of multitasking performance, but polychronicity and

extraversion are not. Human Performance, 18(3), 243-266.

Kramer, A. F., & Willis, S. L. (2002). Enhancing the cognitive vitality of older adults.

Current Directions in Psychological Science, 11(5), 173-177.

Kushleyeva, Yelena, Dario D. Salvucci, and Frank J. Lee. "Deciding when to switch

tasks in time-critical multitasking." Cognitive Systems Research 6.1 (2005): 41-

49.

Lee, J. D., McGehee, D. V., Brown, T. L., & Reyes, M. L. (2002). Collision warning

timing, driver distraction, and driver response to imminent rear-end collisions in a

high-fidelity driving simulator. Human Factors: The Journal of the Human

Factors and Ergonomics Society, 44(2), 314-334.

Levy, J., & Pashler, H. (2008). Task prioritization in multitasking during driving:

175

Opportunity to abort a concurrent task does not insulate braking responses from

dual‐task slowing. Applied Cognitive Psychology, 22(4), 507-525.

Lipshitz, R., & Strauss, O. (1997). Coping with uncertainty: A naturalistic decision-

making analysis. Organizational Behavior and Human Decision Processes, 69(2),

149-163.

McCarley, J.S., Vais, M. J., Pringle, H., Kramer, A.F., Irwin, D.E., & Strayer, D.L.

(2004). Conversation disrupts change detection in complex traffic scenes. Human

Factors: The journal of the Human Factors and Ergonomics Society, 46(3), 424-

436.

Ma, R., & Kaber, D. B. (2005). Situation awareness and workload in driving while using

adaptive cruise control and a cell phone. International Journal of Industrial

Ergonomics, 35(10), 939-953.

Mehle, T. (1982). Hypothesis generation in an automobile malfunction inference task.

Acta Psychologica, 52(1), 87-106.

Moray, N. (2003). Monitoring, complacency, skepticism and eutectic behavior.

International Journal of Industrial Ergonomics, 31(3), 175-178.

National Institute of Standards and Technology. (1993). Integration Definition for

Function Modeling (IDEFØ). Federal Information Processing Standards

Publication 183, Gaithersburg, Maryland.

National Safety Council White Paper. (2010). Understanding the distracted brain: Why

driving while using hands-free cell phones is risky behavior. (on-line publication)

176

Nowinski, J.L., Holbrook, J.B., & Dismukes, R. K. (2003). Human memory and cockpit

operation: An ASRS study. In Proceedings of the 12th international Symposium

on Aviation Psychology (pp.888-893).

NTSB. (1973). Aircraft accident report. Eastern Air Lines, Incorporated, L-1011,

N310EA, Miami, Florida, December 29, 1972. Report No. NTSB-AAR-73-14.

Washington, DC: National Transportation Safety Board

NTSB (1987). Aircraft accident report. Northwest Airlines, Incorporated, MD-82,

Detroit, Michigan, August 16, 1987. Report No. Washington, DC: National

Transportation Safety Board

Pashler, H. (1984). Processing stages in overlapping tasks: evidence for a central

bottleneck. Journal of Experimental Psychology: Human Perception and

Performance, 10(3), 358.

Pashler, H., & Johnston, J. C. (1989). Chronometric evidence for central postponement in

temporally overlapping tasks. The Quarterly Journal of Experimental Psychology,

41(1), 19-45.

Raby, M., & Wickens, C. D. (1994). Strategic workload management and decision biases

in aviation. The International Journal of Aviation Psychology, 4(3), 211-240.

Ramsey, F., & Schafer, D. (2012). The statistical sleuth: a course in methods of data

analysis. Cengage Learning.

Rasmussen, J. (1983). Skills, rules, and knowledge; signals, signs, and symbols and other

177

distractions in human performance models. Systems, Man and Cybernetics, IEEE

Transaction on, (3), 257-266.

Reason, J. (1990). Human error. Cambridge university press.

Richard, C.M., Wright, R. D., Ee, C., Prime, S.L., Shimizu, Y., & Vavrik, J. (2002).

Effect of a concurrent auditory task on visual seach performance in a driving-

related image-flicker task. Human Factors: The journal of the Human Factors and

Ergonomics Society, 44(1), 108-119.

Robert, G., & Hockey, J. (1997). Compensatory control in the regulation of human

performance under stress and high workload: A cognitive-energetical

framework. Biological psychology, 45(1), 73-93.

Saaty, T. L. (1990). How to make a decision: the analytic hierarchy process. European

journal of operational research, 48(1), 9-26.

Salvucci, D. D., Taatgen, N. A., & Borst, J. P. (2009). Toward a unified theory of the

multitasking continuum: From concurrent performance to task switching,

interruption, and resumption.

Salvucci, D. D., & Taatgen, N. A. (2010). The multitasking mind. Oxford University

Press.

Schank, R. C. (1980). Language and Memory*. Cognitive Science, 4(3), 243-284.

Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human

information processing: II. Perceptual learning, automatic attending and a general

178

theory. Psychological review, 84(2), 127.

Shakeri, S., & Funk, K. (2007). A comparison of human and near-optimal task

management behavior. Human Factors: The Journal of the Human Factors and

Ergonomics Society, 49(3), 400-416.

Schutte, P. C., & Trujillo, A. C. (1996, October). Flight crew task management in non-

normal situations. In Proceedings of the Human Factors and Ergonomics Society

Annual Meeting (Vol. 40, No. 4, pp. 244-248). SAGE Publications.

Shakeri, S., & Funk, K. (2007). A comparison of human and near-optimal task

management behavior. Human Factors: The Journal of the Human Factors and

Ergonomics Society, 49(3), 400-416.

Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information

processing: II. Perceptual learning, automatic attending and a general theory.

Psychological review, 84(2), 127.

Simon, H. A. (1979). Rational decision making in business organizations. The American

economic review, 493-513.

Simons, D. J., & Chabris, C. F. (1999). Gorillas in our midst: Sustained inattentional

blindness for dynamic events. Perception-London, 28(9), 1059-1074.

Simons, D. J., & Levin, D. T. (1998). Failure to detect changes to people during a real-

world interaction. Psychonomic Bulletin & Review, 5(4), 644-649.

Stanton, N. A., & Baber, C. (2008). Modelling of human alarm handling response times:

179

a case study of the Ladbroke Grove rail accident in the UK. Ergonomics, 51(4),

423-440.

Strayer, D. L., Drews, F. A., & Johnston, W. A. (2003). Cell phone-induced failures of

visual attention during simulated driving. Journal of experimental psychology:

Applied, 9(1), 23.

Strayer, D. L., Watson, J. M., & Drews, F. A. (2011). 2 Cognitive Distraction While

Multitasking in the Automobile. Psychology of Learning and Motivation-

Advances in Research and Theory, 54, 29.

Strayer, D. L., Drews, F. A., & Johnston, W. A. (2003). Cell phone-induced failures of

visual attention during simulated driving. Journal of experimental psychology:

Applied, 9(1), 23.

Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of

experimental psychology, 18(6), 643.

Tsang, P. S., & Vidulich, M. A. (2006). Mental workload and situation awareness.

Handbook of Human Factors and Ergonomics, Third Edition, 243-268.

Vidulich, M.A., and Tsang, P.S. (1987), “Absolute Magnitude Estimation and Relative

Judgement Approaches to Subjective Workload Assessment”, in Proceedings of

the Human Factors Society 31st Annual meeting, Human Factors and Ergonomics

Society, Santa Monica, CA, pp.1057-1061.

Wallenius, J., Dyer, J. S., Fishburn, P. C., Steuer, R. E., Zionts, S., & Deb, K. (2008).

180

Multiple criteria decision making, multiattribute utility theory: Recent

accomplishments and what lies ahead. Management Science, 54(7), 1336-1349.

Wallsten, T.S., & Barton, C. (1982). Processing probabilistic multidimensional

information for decisions. Journal of Experimental Psychology: Learning,

Memory, and Cognition, 8(5), 361.

Winston, W. L., & Albright, S. C. (2010). Practical Management Science with Disk, 4th

edition. Duxbury

Wickens, C. D., & Kessel, C. (1980). Processing resource demands of failure detection in

dynamic systems. Journal of Experimental Psychology: Human Perception and

Performance, 6(3), 564.

Wickens, C., Kramer, A., Vanasse, L., & Donchin, E. (1983). Performance of concurrent

tasks: A psychophysiological analysis of the reciprocity of information-processing

resources. Science.

Wickens, C. D. (2002a). Multiple resources and performance prediction. Theoretical

issues in ergonomics science, 3(2), 159-177.

Wickens, C. D. (2002b). Situation awareness and workload in aviation. Current directions

in psychological science, 11(4), 128-133.

Wickens, C. D., Goh, J., Helleberg, J., Horrey, W. J., & Talleur, D. A. (2003).

Attentional models of multitask pilot performance using advanced display

technology. Human Factors: The Journal of the Human Factors and Ergonomics

Society, 45(3), 360-380.

181

Wickens, C. D. (2008). Multiple resources and mental workload. Human Factors: The

Journal of the Human Factors and Ergonomics Society, 50(3), 449-455.

Wickens, C. D., Hollands, J. G., S. Banbury, and R. Parasuraman (2013). Engineering

psychology and human performance 4e: Prentice Hall New Jersey.

Wilson, J. R. (1998). The effect of automation on the frequency of task prioritization

errors on commercial aircraft flight decks: An ASRS incident report study.

Zumel, N., & Mount, J. (2014). Practical Data Science with R. Manning Publications

182

APPENDICIES

183

APPENDIX A: GLOSSARY

184

Table A-1: Terms and Meaning / Definitions

Terms Meaning / Definitions

Aviate task Those activities related to control of the aircraft’s motions, particularly altitude movement.

Cockpit Task Management Funk (1991)’s model that addresses managing concurrent multiple tasks in three stages: situation awareness, response selection and response execution

Communicate task Those activities related to transmitting information to or receiving information from another human such as ATC

Expectancy of a signal that An internal information of stimuli stored in the pilot’s relates to a task long-term memory (i.e., mental model) that relates to a specific task

Navigate task Those activities related to control the lateral motion from the present location to arrive at an intended location.

Number of concurrent tasks A parameter used for the number of tasks which were per second executed concurrently. This parameter was used to show the observed workload of the participants during the onset of flight instrument problems.

Manage Systems task Those activities related to the operating the aircraft’s secondary equipment, such as its electrical system, hydraulic system, etc.

Task Importance Weight of contribution of task to the ultimate goal “safe flight to the destination”

Task Performance Status How successful in accomplishing the goal of the task

Task Priority Allocation of policy of dividing resources among tasks

Task Salience The degree of attention-catching stimuli that relates to the task

Task Urgency The time difference (i.e., buffer time) between the deadline time of a task and the time required to finish that task

185

Table A-2: Definitions of Explanatory Factors and Response Variables

Explanatory Factors Definitions 1. Reported Reported importance of a task is the reported relative importance of a task contribution of task to the ultimate goal “fly safe to the ( I ) destination” obtainable in a pairwise comparison method (Saaty, 1990).

i  1 0  iTask 1 and  Task . All _ Task 2. Perceived urgency Perceived urgency of a task is the perceived time difference of a task (U ) between the time required to finish the task and its deadline time (unit: seconds). The shorter this interval, the greater the urgency.

(Wickens et al., 2013, p337). 1 uTask  300 (seconds) 3. Perceived Status of a task is the reported satisfactory level of a task performance status performance against the target performance threshold of the task. of a task ( P ) It is an integer value between 1 (least satisfactory performance)

and 9 (the most satisfactory). 1 pTask  9 . 4. Reported salient It is the perceived degree of attention-catching stimuli that stimuli of a task relates to each task. It is integer value between 1 (least salient) ( S ) and 9 (the most salient), and 1 sTask  9. 5. Reported workload It is the perceived degree of workload of a task measured by of a task (W ) NASA Task Load Index (Hart & Staveland, 1988).

1  wTask  100 Response Variable Definitions It is perceived allocation of divided cognitive resources to a task Reported priority of measured by pairwise comparison among tasks in questionnaire. a task (Y ) y  1 0  yTask  1 and  Task . ALL _ Task Functional attendance on a task based on flight recorder data and Observed priority of recorded video. a task ( Z )  1...(Attended ) zTask  0....(Otherwise)

186

Table A-3: Definition of Explanatory Variables Explanatory Definitions Variables 1. Mean number of Mean number of concurrent tasks per second is the fraction rate concurrent tasks of total number of concurrent multitask during the problem- (Continuous number) per second unnoticed time period divided by the time period of unnoticed time set T .

 mt pt tT uMultitasks|Pr oblem   pt tT

Where mt  Tasks is the total discrete number of active tasks t

at each of the discrete time t (1 second at each time) and pt is the binary indicator (0 or 1) value when the problem is unnoticed at time .

It should be noted that pt  1 for t T . For example, =1 during the problem is unnoticed during time period of t=(17,25)). However, would vary at each second. For

example, it could be mt  Tasks =1 during time period t

t (17,20) , and mt  Tasks=2 during the time period t t (21,25) . 2. Expectation of 1 problem (Binary) Expectation (Problem)=  0 If a pilot had an expectation of the problem, 1 is assigned. Otherwise, 0 is assigned. 3. Salience of the 1 signal related to the Salient Signal (Problem)=  problem 0 If there is a salient signal of the problem, 1 is assigned. Otherwise, 0 is assigned. Response Variable Definitions  1...(Noticed ) zPr oblem  Observed 0....(Unnoticed ) inattention blindness  ( Z ) A time period it took for a pilot to notice the malfunction Unnoticed time (T ) problem when a pilot notice it.

t problem  0 .

187

APPENDIX B: INFORMED CONSENT DOCUMENT FOR EXPERIMENT

188

Consent form

Project Title: Modeling Human Behaviors in a Time-Pressured Multitasking Environment Principal Investigator: Kenneth H. Funk II, PhD Student Researcher: Takeaki Toma Version Date: 03/11/2014

1. WHAT IS THE PURPOSE OF THIS FORM?

This form contains information you will need to help you decide whether to be in this study or not. Please read the form carefully and ask the study team member(s) questions about anything that is not clear.

2. WHY IS THIS STUDY BEING DONE?

The purpose of this study is to gain a better understanding of how human operators make decisions in time-pressured multitasking situations, like single-pilot flight operations.

The objective of this research is to collect data and test our hypotheses about factors that affect decision making by using statistical hypothesis testing analysis methods. The results of this study are expected to provide insights into human multitasking, which will help human factors engineers design better human-machine interfaces for cockpits as well as improve pilot training. A low-fidelity flight simulator is being used in order to collect data and to facilitate the generalization of our results. This study is being conducted by Takeaki Toma for the completion of his PhD thesis in Industrial Engineering. Up to 30 participants may be invited to take part in this study.

3. WHY AM I BEING INVITED TO TAKE PART IN THIS STUDY?

You are being invited to take part in this study because we are seeking licensed pilots with flight knowledge and experience capable of performing flight simulations.

189

4. WHAT WILL HAPPEN IF I TAKE PART IN THIS RESEARCH STUDY?

The study activities include: filling out a short questionnaire, training on our simulator (the training session of the flight simulation is flying to the Corvallis Municipal airport on the flight simulator in the lab), performing a flight simulation, and, afterwards, answering some questions about the simulated flight.

Study duration: The expected time for this study is 90 minutes, and it will not exceed 120 minutes. You will only visit the lab once and will not be asked to return for subsequent investigations.

Video and voice recording: You will be audio and video recorded during the simulated flight. This is because you will watch the recorded video to help you recall your decisions during the flight to answer questions about it. If you do not wish to be audio or video recorded, you have the choice not to participate.

5. WHAT ARE THE RISKS AND POSSIBLE DISCOMFORTS OF THIS STUDY?

There are no anticipated health or safety risks for individuals participating in this study.

6. WHAT ARE THE BENEFITS OF THIS STUDY?

You are expected to benefit in two ways. The first benefit is increased awareness of safety and risky and dangerous flight situations. The second benefit is that you will learn about modern human factors research concerning aviation safety (you will be informed of our research findings). After data analysis and dissertation is completed, the summary of main findings will be sent to the participant as a written document. This study is expected to contribute to society by providing insights about pilot performance. The study results will help human factor engineers develop better human-machine interfaces and training.

7. WILL I BE PAID FOR BEING IN THIS STUDY?

You will be compensated with a $25 gift card upon completion of all study activities. You will not be compensated if you do not complete all study activities.

8. WHO WILL SEE THE INFORMATION I GIVE?

190

The information you provide during this research study will be kept confidential to the extent permitted by law. Research records will be stored securely and only the Principle Investigator and student researcher will have access to the stored records. Federal regulatory agencies and the Oregon State University Institutional Review Board (a committee that reviews and approves research studies) may inspect and copy records pertaining to this research.

Collected questionnaire data and flight performance data will be encrypted with a 15- digit password. The encrypted data, recorded voice data, and video data will be stored on a single USB drive and stored in a locked cabinet in the PI’s office. The student researcher will make two copies of the encrypted data files on his computer for data analysis and thesis writing, to be deleted from his computer when the research is completed. The student researcher will be permitted to show the data on his computer to his PhD committee members (Dr. Eseonu, Dr. Funk, Dr. Lien, Dr. Emerson, and Dr. Veltri) for research consultation purposes only during his data-analysis and thesis-writing phases. Study related materials, including signed consent forms, flight data, voice recordings, and video recordings will be stored in a locked cabinet in the office of the PI. The above material will be retained for a minimum of three years post-study termination.

9. WHAT OTHER CHOICES DO I HAVE IF I DO NOT TAKE PART IN THIS STUDY?

Participation in this study is voluntary. If you decide to participate, you are free to withdraw at any time without penalty. You will not be treated differently if you decide to stop taking part in the study, except that you will not receive the gift card. If you choose to withdraw from this project before it ends, the researchers will not keep information collected about you.

Participation terminated by investigator: If you do not follow instructions during the simulation, the student investigator may elect to terminate your participation in the study.

10. WHO DO I CONTACT IF I HAVE QUESTIONS?

If you have any questions about this research project, please contact: Kenneth H. Funk II at (541) 737-2357, or by email at [email protected].

If you have questions about your rights or welfare as a participant, please contact the Oregon State University Institutional Review Board (IRB) Office, at (541) 737-8008 or by email at [email protected]

191

11. WHAT DOES MY SIGNATURE ON THIS CONSENT FORM MEAN?

Your signature indicates that this study has been explained to you, that your questions have been answered, and that you agree to take part in this study. You will receive a copy of this form.

Participant's Name (printed): ______

______(Signature of Participant) (Date)

______(Signature of Person Obtaining Consent) (Date)

192

APPENDIX C: IDEF0 DIAGRAM FOR MODELING COCKPIT TASK MANAGEMENT

193

194

195

196

197

198

APPENDIX D: EXPLANATION OF FLIGHT SIMULATION TO PARTICIPANTS

199

Good evening, my name is Takeaki Toma and I am a graduate student here at Oregon State

University. My area of interest is human factors engineering with special interest in multi-tasking in the aircraft cockpit environment.

First, thank you for taking part in this study; you have special skills and experience very relevant to this investigation. My objective is to develop information about how pilots multitask to improve training and equipment in aviation and possibly other critical workload environments.

Your participation in this study is entirely voluntary and confidential. No one will have access to your participation and flight data without your specific written consent. This includes any government or private sector organization. Other than relevance to the study data, you will not be evaluated or graded in any way. There are no right or wrong responses or answers. I encourage you to have some fun with the flight.

We are looking for some serious information, but in as stress free and open environment as possible. The only stress will be created by the situations developed to gain information on how pilots multi task and how they deal with distractions, communication, operational, weather and systems problems. Feel free to be as creative in your responses as you wish. Lastly, you may elect to discontinue the flight at any time, however data collected to that point may be used in the investigation.

The flight will be on an instrument flight plan in VFR, marginal VFR and IFR conditions. The aircraft simulated is the Cessna 172 RG and the simulator program is the popular X Plane application.

The simulator is set up to mimic the Cessna and you will recognize the controls and displays. This includes side views and sound. You will have up to 20 minutes to get acquainted with the flight controls and displays with actual flight practice. Please note that the HSI is not used in this flight. The heading information on the HSI is, however correct. All other instruments and displays may or may not

200 be working, depending the problem being evaluated. All I can say is expect the unexpected. Also feel free to ask any questions you may have, they will be answered to the point at which the answer may influence you response to the simulation and contaminate the data.

The basic flight plan will be from KEUG Eugene Mahlon Sweet airport to Roberts Filed Airport

(RDM) at Redmond near Bend Oregon on V269 and V536 with a navigation change at Mante intersection.

201

1. Flight simulator equipment  X-plane flight simulator is used for Cessna 172 RG aircraft.

Figure D-1: Flight simulator equipment

202

Figure D-2: Cockpit Instrument Panel

203

Figure D-3: Transponder

204

Figure D-4: Yoke Panel

2. The flight plan  You will fly from Eugene airport (KEUG) to Robert Field airport (BDN) through V-269 and V536 routes with Cessna 128RG at 8pm, June 25th (See the attached flight plan sheet)  The departure part is omitted, and you will start the flight simulation from climbing from KEUG until arriving at KRDM.  The flight is divided into 8 situations. After few minutes of the flight, the simulation will be suspended and you will answer to the questions.  The approach and landing to BDN is guided by vector guiding approach by synthesized voice of ATC call.

205

Figure D-5: Used Flight Path in this Experiment

3. Air Traffic Control (ATC) communication protocol

 In order to collect human multitasking data in a uniform environment, recorded synthesis voice will be played.  Because ATC controller’s script is predetermined, the only allowed communication is to repeat the ATC’s order for confirmation, and answer to the ATC request  If you miss the ATC communication, you can ask to repeat the sentences

4. Available tools  SkyVector’s information of Eugene and Bend airports.

206

 Aircraft checklist (Please read loud and confirm with the panels)  Flight Chart  Pen, note and kneeboard.

5. If any trouble occurs during the flight, take following actions

 Because the purpose of this flight simulation is to observe human multitasking (instead of super realistic flight simulation), even if there is a serious problem, please do not try to land on a nearest airport; just keep flying to the destination.

Table D-1: Instructions for Troubles in Flight Simulation Trouble categories Actions Display, mechanical troubles -Point finger to the problem instrument -Try to switch off/on 3 times -Report the problem to ATC Unclear ATC voice, missed -Ask ATC to repeat ATC communication If communication frequency -Confirm the information (e.g., communication frequency does not match (i.e., no ATC etc) on the flight chart, or SkyVector information reply)

207

6. Task categories used in the questionnaire

Figure D-6: Task Categories in the Cockpit Instrument Panel

Table D-2: Task Explanation Tasks Explanation Details

208

Those tasks related to 1.1 Control/ monitor vertical profile (altitude) Aviate control of the aircraft’s 1.2 Control/Monitor speed altitude, attitude 1.3 Maintain clearness and restrictions 1.4 Maintain separation with traffic, terrain

Those tasks related to 2.1 Control/monitor lateral profile Navigate determining the present 2.2 Maintain awareness of temporal profile location and how to arrive at 2.3 Modify route for weather, traffic, hazards an intended location 2.4 Set navigational radios

Those tasks related to 3.1 Communicate with ATC Communicate transmitting information to 3.2 Tune communication radios or receiving information 3.3 Receive ATIS from another human, such as ATC Those tasks related to the 4.1 Manage/control system faults Manage operating the aircraft’s 4.2 Monitor aircr4aft subsystems Systems secondary equipment, such as its electrical system, hydraulic system, etc.

209

Appendix E: ATC COMMUNICATION SCRIPT

210

Table E-1: ATC Communication Scripts

Situation No ATC The Pilot

Start Experiment during airborne 7000 feet

Cessna zero-one-four, Contact Eugene departure now one- (1) one-nine-point-six.

Eugene Tower asks (1) Cessna 014, one-one-nine-point-six. Cessna to take contact with Eugene Departure, Cessna 014 Oscar-Sierra, is with you at departure 119.6 seven thousand.

Cessna 014, Eugene Departure, radar contact.

Join Victor-two-six-nine on course to Mante, climb and (2) maintain nine thousand feet.

Contact Eugene (3) Report level at 9000 departure (2)Cessna 014, Join Victor-two-six-nine on course to Mante, Report when established on the Victor-two-six-nine radial. (4) climb and maintain nine thousand feet.

(3) Report when established on the Victor-two-six-nine.

Leaving 7000 for 9000……

(4.5) Eugene departure, Cessna 014, established on Victor-

269.

Eugene Departure (5) Cessna 014, contact Seattle Center one-two-five-point-eight.

(3.5) Seattle center, Cessna 014, 8000 passing altitude for

9000

(5) Roger, Cessna 014, contact Seattle Center one-two-five-

point-eight.

Seattle Center 125.8 (6) Cessna 014, Seattle Center, Ident .

(6 ) Cessna014,- Ident.

Seattle Center 125.8 Cessna 014, radar contact.

Freeze and collect questionnaire data for the situation-1

Seattle Center 125.8 (7) Cessna 014, Climb and maintain eleven thousand. (7) Cessna 014, leaving nine thousand, for eleven thousand,

(8) Report level at eleven-thousands. (8) Cessna Zero-One-Four, level at eleven thousand.

Thank you.

Freeze and collect questionnaire data for the situation-2

211

Situation No ATC The Pilot

Seattle Center, Cessna 014, Report crossing Mante Seattle Center 125.8 (9) intersection.

Cessna 014, report crossing Mante intersection.

(10) Seattle Center 125.8 Cessna 014, can you report cloud tops?

(10) Zero-One-Four, tops at 10,000

Cessna 014 , Thank you.

Cessna 014, contact Seattle center now on one three one point

(11) two five.

Seattle Center 125.8 (11 ) Roger, Seattle center one-three-one-point-two-five.

(He/she changes the frequency)

Seattle center, Cessna Zero-One-Four-Oscar-Sierra- is with

you at eleven thousand.

(12) Cessna 014, Seattle Center, Ident.

(12) Cessna Zero-One-Four,Ident.

Cessna Zero-One-Four , radar contact lost. Recycle

(13) transponder, Ident Seattle Center 131.25 (13) Roger, Recycling transponder, Ident.

Cessna Zero-One-Four , reset transponder, Squawk one,

(14) seven, one, four.

(14) one seven one four, Cessna Zero-One-Four.

Cessna Zero-One-Four, Radar contact!

(7 ) Cessna Zero-One-Four, crossing Mante intersection.

Seattle Center Cessna-014 , contact Seattle center now, one, two, eight, (15) 131.25 point, one, five.

(15) Seattle Center, one-two-eight-poiint-one-five.

Pilot tunes to 128.15 for another Seattle Center.

Seattle center, Cessna 014 is with you at eleven thousand.

Seattle Center Cessna Zero-One-Four, radar contact. 128.15

Seattle Center (16) Cessna-014 , if you are ready, we can give you ATIS 128.15 information for Roberts Field.

212

Situation No ATC The Pilot

Seattle Center Roberts Field Airport information Romeo. 128.15

Seattle Center Time-One six zero zero, (16) Roger Ready to copy ATIS 128.15

ceiling three thousand broken .

Visibility eight miles.

wind two six zero at four.

Seattle Center Temperature two six, 128.15

Altimeter three-zero-eight-zero.

ILS runway two, two in use

Advise you have information Romeo.

Freeze and collect data for the situation-3

No communication during situation 4

Freeze and collect data for the situation4

Seattle Center Cessna Zero-One-Four, Seattle center, contact Roberts (17) Cessna 014, switching Roberts Approach 128.15 128.15 (17) approach one-two-eight-point-one-five.

pilot tunes 256.8,

Roberts approach, Cessna 014 is at 11,000.

Roberts approach Cessna 014, Radar contact. 128.15

Fly direct to Deschutes VOR, then outbound radial Roberts approach 020. Expect radar vectors for the runway two-two ASR 128.15 (18) approach.

(18) Roger, Cessna Zero-One-Four, fly direct Deschutes

VOR, then outbound 020.

Roberts approach Cessna 014, descend and maintain 9000 feet. 128.15 (19)

(19) Cessna 014, it's 11000, descending 9000.

Roberts approach Report level at 9000. 128.15 (20)

213

Situation No ATC The Pilot

(20 ) Report level at 9000.

Roberts approach (20.5 ) Cessna Zero-One-Four, level at nine thousand. 128.15

Cessna Zero-One-Four , thank you.

Freeze and collect data for the situation-5

Cessna014, Roberts Fields approach, fly outbound Deschutes

(21) radial 020.

(21)Cessna014, Roberts Fields approach, fly outbound

Deschutes radial 020.

Roberts approach 128.15 (22) Cessna 014, descend and maintain 7000.

(22) Roger, leaving 9000 for 7000, Cessna 014.

(23) Cessna 014, turn right heading 1, 0, 6.

(23) Turning right 1, 0, 6, Cessna 014.

Freeze and collect data for the situation-6

Roberts approach Cessna 014, Roberts approach, Maintain heading 1,0,6,

128.15 (24) descend and maintain 5000.

(24) Cessna Zero-One-Four, maintain heading 106, leaving

7000 for 5000.

Roberts approach Cessna Zero-One-Four, contact Roberts Field approach one-

128.15 (25) two-five-point-four

(25) one-two-four-point-five, Cessna Zero-One-Four.

(25) Roberts field approach, Cessna 014, maintain 5000.

Roberts approach Cessna014, Roberts field approach, radar contact. This will (26)Cessna014, Roberts field approach, radar contact. This 124.5 (26) be a visual approach, runway two-two. will be a visual approach, runway two-two.

Missed approach instructions, fly heading 2, 2, 0 and climb to (27)Missed approach instructions, fly heading 2, 2, 0 and (27) 6000. climb to 6000.

(28) Cessna 014, turn right heading 2, 0, 0. Roberts approach 124.5 (28) Turn right heading 2, 0, 0, Cessna 014.

Freeze and collect data for the situation-7

214

Situation No ATC The Pilot

Cessna Zero-One-Four, Roberts approach, fly heading, two, (29) zero, zero to intercept final approach course two, two, five. Roberts approach Confirm the airport in sight. 124.5 (29) Fly heading two-zero-zero, intercept final approach

course two-two-five.

(30) Cessna 014, turn right final approach course two, two, five. (30) Turn right two-two-five,

Roberts approach 124.5 (31) Report airport in sight. (31) Airport in sight, Cessna 014.

Cessna 014, Roberts approach, cleared to land runway Two, (32) Two. Altimeter two, niner, niner, four.

Roberts approach (32) Cessna014, cleared to land runway two-two. Wind is

124.5 calm. Altimeter two-niner-niner-four.

Freeze and collect data for the situation-8

215

APPENDIX F: QUESTIONNAIRE

216

Questions about your aviation experience and background

1. Please fill in your pilot Information

FAA Certificate Type ______Date Issued ______Rating(s) ______

Last Flight Review Date ______

1. Have you ever been involved in any aircraft accidents or incidents?

Y ___ N ___ (Summary: )

3. How many total flying hours do you have?

______Hours (____hours of single pilot time)

4. How many hours did you fly?

2000~2014: ____ hours

5. How old are you?

_____ Years old

6. Have you ever piloted a Cessna airplane (model 172 or 172 RG)?

217

_____Yes ____ NO Type (______)

7. What aircraft do you most frequently fly?

______

8. Do you ever fly between Bend/Redmond (Robert Field Airport) and Eugene (Mahlon Sweet Field Airport)?

____ Yes (______hours total)

____ No

9. Have you ever used a computer flight simulator? If so, which one did you use and for how many hours?

____ X-plane® (_____ hours) ____ Microsoft Flight Simulator ® (_____ hours) ____ Others (______hours)

10. After the experiment will be finished, please provide any comment, advice, or suggestion for this study.

218

Question: Which task did you prioritize at this moment?

No.1 Aviate Navigate absolutely very much moremuch more slightly equally slightly much more very much moreabsolutely

Aviate is absolutely more attended than Navigate 11

No.2 Aviate Communicate absolutely very much moremuch more slightly equally slightly much more very much moreabsolutely

Aviate is absolutely more attended than Communicate 21

No.3 Aviate Manage System absolutely very much moremuch more slightly equally slightly much more very much moreabsolutely

Aviate is absolutely more attended than Manage System 31

No.4 Navigate Communicate absolutely very much moremuch more slightly equally slightly much more very much moreabsolutely

Communicate is absolutely more attended than Navigate 49

No.5 Navigate Manage System absolutely very much moremuch more slightly equally slightly much more very much moreabsolutely

Manage System is much more attended than Navigate 57 how to segrate navigate vs managesystem,or how to segrate aviate vs navigate

No.6 Communicate Manage System absolutely very much moremuch more slightly equally slightly much more very much moreabsolutely

Communicate is absolutely more attended than Manage System 61

219

220

221

Question: Based on your comprehension of the current situation, rate the performance of task, how successful you believe in accomplishing the goal of the task set by yourself?

1. Aviate Task Absolutely Extremely Very Slightly Neither Slightly Very Extremely Absolutely

Unsatisfactory ← →Satisfactory The task performance of Aviate is neither unsatisfactory/satisfactory.

2. Navigate Task Absolutely Extremely Very Slightly Neither Slightly Very Extremely Absolutely

Unsatisfactory ← →Satisfactory The task performance of Navigate is neither unsatisfactory/satisfactory.

3. Communicate Task Absolutely Extremely Very Slightly Neither Slightly Very Extremely Absolutely

Unsatisfactory ← →Satisfactory The task performance of Communicate is neither unsatisfactory/satisfactory.

4. Manage System Task Absolutely Extremely Very Slightly Neither Slightly Very Extremely Absolutely

Unsatisfactory ← →Satisfactory The task performance of Manage System is neither unsatisfactory/satisfactory.

222

223

224

225

226

227

Question: Based on your perception in the current situation, did you notice any equipment malfunction? Please click a checkbox(s) if you noticed any equipment failure.

Communi panel VOR panel

Airspeed Indicator Attitude Indicator Altimater

Transponder

Remaining fuel Turn Coordinator Vertical Speed Nav/Com- Radio(RMI)

Engine rpm Flap

Oil Pressure Vacuum Pump Fuel Tank Vent Block Wrong communication Frequency

Oil Temperature Cylinder head temperature Pitch Control Landing Gear Position

Engine Failure Static-port blockage

Pitot-tube blockage

Landing Gears

Cowl Flaps Landing Gear

228

APPENDIX G : CODING VARIABLES AND HYPOTHESIS TESTING METHODOLOGIES

229

In our study, we statistically test our research hypotheses. To do so, all explanatory variables and response variables must be numerically coded. In this section, it is explained about how to code these variables: reported importance of tasks, perceived urgency of tasks, reported status of tasks, reported salience of tasks, reported workload of tasks, reported priority of tasks, and observed priority of tasks. Finally, hypotheses testing methodologies are stated.

G-1. Coding importance of tasks (Factor 1)

The first encoding variable is the importance of task. Here, the definition of

“importance of task” is the pilot’s perceived weight of contribution of a task to the ultimate goal “safe flight to the destination”. The goal of coding of importance of tasks is to obtain a weight vector of tasks such as,

I  i1,i2 ,i3,i4   0.4 0.1 0.3 0.2 (1)

where i1 , i2 ,i3 , and i4 are the importance of aviate, navigate, communicate, and Manage

Systems tasks respectively.

The first step is to obtain the perception of the importance of each task by the pairwise comparison methodology (Saaty, 1991). Similar to the Situation Awareness

Global Assessment Technique (SAGAT, Endsley 1995b), the importance of tasks is collected by asking the following probe question, “ Based on your comprehension of the current situation, which task is more important ?”. Suppose the focal task “aviate” is compared to the task “ Navigation”. The participant moved the slider so that the relative

230 importance of task was chosen (See the following figure). In this example, the participant believes “Aviate is very much more important than Navigate”.

FIGURE G-1: An example of pairwise comparison

After moving the sliding bar, a sentence appeared such as, “Aviate [task i ] is much more important than Navigate [task j ]”. Then, the corresponding numerical value

ii aij  (here, ii is the importance of task i , and i j is the importance of task j ) was i j obtained from the following table.

231

Table G-1: Interpretation of relative importance between task and task

Interpretation Value of aij 1/9 Task j is absolutely more important than task i . 1/7 Task is very much more important than task . 1/5 Task is much more important than task . 1/3 Task is slightly more important than task . 1 Task i and j are equally important 3 Task slightly more important than task . 5 Task much more important than task . 7 Task very much more important than task . 9 Task absolutely more important than task .

The next step is to compute the relative weight of each task by using pairwise comparison matrix. The following (4 x 4) symmetric matrix was constructed; here we indicate subscript 1 indicates aviate task 1, subscript 2 indicates navigate, subscript 3 indicates communicate, and subscript 4 indicate Manage Systems task.

a11 a12 a13 a14   1 a12 a13 a14  a a a a  1/ a 1 a a  A   21 22 23 24   12 23 24 (2) a31 a32 a33 a34 1/ a13 1/ a23 1 a34     a41 a42 a43 a44 1/ a14 1/ a24 1/ a34 1 

T According to Saaty (1991), the weight vector I  i1 i2 i3 i4  for the importance of tasks can be estimated by a normalized eigenvector of matrix A. This is because there is a relations such that

A I  I (3)

T Where  and I  i1 i2 i3 i4  are the eigenvalue and normalized eigenvector of the matrix A . In our study, eigenvector was approximated by using the method of Winston and Albright (2010).

232

In the above case, the eigenvalue  corresponds to 4 as,

i1 i1 i1 i1  i i i i   1 2 3 4  i i i i i1  4i1  i1   2 2 2 2          i 4i i i1 i2 i3 i4  2   2   2       4 (4) i3 i3 i3 i3 i  4i  i    3 3 3 i i i i        1 2 3 4  i4  4i4  i4  i4 i4 i4 i4    i1 i2 i3 i4 

(4) holds only when all the pairwise comparisons are perfectly consistent in the matrix

A , that is , the following conditions holds at any combination of comparisons.

aij a jk  aik (5)

However, a perfect consistency may not occur in the real world human judgment, thus

Saaty (1991) proposed the Consistency Index (CI),

  n CI  max (6) n 1

If the consistency index of the obtained comparison matrix A is less than the Random

Consistency Index (RCI) shown in the following table, the matrix is regarded to be consistent (i.e., reliable).

Table G-2: Random Indices for Consistency Check for the AHP n 3 4 5 6 7 8 RCI 0.58 0.90 1.12 1.24 1.32 1.41

Thus, in our study, we accepted participant data satisfyingC.I  0.9 .

233

G-2. Coding perceived urgency of tasks (Factor 2)

The second encoding variable is the participant’s perceived urgency of task ( Aviate, Navigate, Communicate, or Manage Systems) at time t . Following Wickens et al.’s (2013) urgency definition, task urgency is defined as the time difference (i.e., buffer

time) between the deadline time of a task and the time required to finish that task (See the following figure). Thus, when the available buffer time approaches zero, the task is regarded as more urgent. On the other hand, when the available buffer time increases, the task is regarded as less urgent.

Figure G-2: Coding the urgency in the model

The goal of coding of the perceived urgency of task is to obtain an urgency vector of tasks U such as,

U  u1,u2 ,u3,u4   11 120 30 300 (7)

Which means that the buffer time for the aviation task ( u1 ) is 7 seconds, the time buffer

for navigation task ( u2 ) is 120 seconds, and so on.

234

The perceived urgency of each task was directly collected by asking the following probe question, “Based on your projection of the future status, rate the urgency of each task by its "buffer time"; the amount of time you could delay the task before it requires your attention to maintain safe flight.” For example, the participant believes “Buffer time of Aviate task is 11 seconds” in the following figure. In this example, the perceived

urgency of aviate task is coded as “ u1 =11”. By repeating this procedure, the urgency vector U was obtained.

Figure G-3: An example of rating urgency for Aviate task

235

G-3. Coding perceived task performance status of tasks (Factor 3)

The third encoding factor is the perceived task performance status. The goal of coding of the perceived urgency of task is to obtain a task performance vector of tasks P such as,

P  p1, p2 , p3, p4   7 7 5 5 (8)

The perceived performance status of each task is collected by asking the following probe question, “Based on your comprehension of the current situation, rate the performance of task, how successful you believe in accomplishing the goal of the task set by yourself? For example, the participant rated such as “Aviate task is very satisfactory” in the following figure.

Figure G-4: An example of rating the performance status of Aviate task

236

The corresponding coding value was retrieved from the following table. In this example, the participant chose the slide bar “Aviate task is very satisfactory”, the task

performance status is coded as p1  7 from the following table. By repeating this procedure, the performance status vector P was obtained.

Table G-3: Coding of performance status of task X

Coding Interpretation of the performance of task X 1 Task X is absolutely unsatisfactory. 2 Task X is extremely unsatisfactory. 3 Task X is very unsatisfactory. 4 Task X is slightly unsatisfactory. 5 Task X is neither unsatisfactory nor satisfactory. 6 Task X is slightly satisfactory. 7 Task X is very satisfactory. 8 Task X is extremely satisfactory. 9 Task X is absolutely satisfactory.

237

G-4. Coding salient stimuli of tasks (Factor-4) The task salience is defined as the degree of attention-catching stimuli that relates to each task. The goal of coding of the perceived urgency of task is to obtain an urgency vector of tasks S such as,

S  s1,s2 ,s3,s4   9 5 3 5 (9)

The reported salience of task is collected by asking the following probe question to the participant, “Based on your current perception, which task is more salient and draws your attention at the moment?” The participant moved the slide bar that matches his or her reported salience for each task (See the following figure).

Figure G-5: A participant moved the slide-bar that matches the reported salience

The corresponding code was retrieved from Table B-3. In the example of the above figure, “Aviate is very much more salient than Navigate”.

238

G-5. Coding workload of tasks (Factor 5)

The fifth factor is the workload of each task. The performance status of a task was defined after NASA’s Task Load Index (TLX), which was developed as a multidimensional rating procedure for estimating workload based on the six criteria: mental demands, physical demands, temporal demands, own performance, effort, and frustration.

The goal of coding of workload of tasks was to obtain a workload vector of tasks such as,  W  WA ,WN ,WC ,WSM   80 50 60 70 (10)

where W1 , W2 ,W3 , andW4 are the workload of Aviate, Navigate, Communicate, and

Manage Systems tasks respectively.

The workload of Aviate task was defined as,

6 WA  wi ai (11) i1

The workload of Navigate task was defined as,

6 WN  wi ni (12) i1

The workload of Communicate task was defined as,

6 WC  wi ci (13) i1

The workload of Manage Systems task was defined as,

6 WS  wi si (14) i1

239

where w1,w2 ,...,w6 were the weights of the six criteria (mental demand, physical demand, temporal demand, performance, effort and frustration) of a task that satisfies the following relation.

6 wi  1 (15) i1

On the other hand, for each criterion i 1,...6, the row scores (1 ai 100 ,

1 ni 100 , 1 ci 100 , 1 si 100 ) were obtained for each task ai ,ni ,ci and si .

The computation of formula (11) to (14) are summarized in the following table.

Table G-4: The workload computation table

Criteria Weight Aviate Navigate Communicate Manage Systems Task Of Criteria Task Task Task

1. Mental w1 a1 n1 c1 s1 Demand

2. Physical w2 a2 n2 c2 s2 Demand

3. Temporal w3 a3 n3 c3 s3 Demand

4. Performance w4 a4 n4 c4 s4

5. Effort w5 a5 n5 c5 s5

6. Frustration w6 a6 n6 c6 s6

6 N/A N/A N/A N/A

wi 1 i1 Total

240

Criteria Weight Aviate Navigate Communicate Manage Systems Task Of Criteria Task Task Task

Workload 6 6 6 6 wA  ai wi wN  ni wi wc  ci wi ws  si wi i1 i1 i1 i1 Of each e.g., e.g., e.g., e.g.,

Task wA  80 wN  50 wC  60 wS  70

Practically, the reported workload data was collected as the following two steps.

The first step was to collect the weight of each criterion. As same as data collection of importance of tasks, this process was conducted by comparing the combination of

6 C2 15 criterion. The relative criteria weights for workload of a task X, were collected by asking the following probe question to the participant, “Please choose which side of the paired criteria contributed more to the workload of the Aviate task”. The participant moved the sliding bar shown in the following figure.

241

….

Figure G-6: Sliding bar for obtaining the weight of each workload criteria

The second step was to collect the row score of each workload criterion. The row scores of each workload criterion task X, were collected by asking the following probe question to the participant, “Please move the bar to rate the following criteria” (See the following Figure).

….

Figure G-7: The sliding bar for rating the row score of workload of task X

242

The reported task priorities are defined as “allocation policy of dividing resources among tasks”. The goal of coding of the reported priorities of tasks is to obtain a priority vector of tasks such as,

Y  y1, y2 , y3, y4   0.6 0.1 0.2 0.1 (16)

where y1 , y2 , y3 , and y4 are the normalized reported priority of aviate, navigate, communicate, and Manage Systems tasks, respectively.

The perceived task priorities was collected by asking the following probe question to the participant, “Which task did you prioritize at this moment?” (See the following

figure). The participant compared 4 C2  6combinations of tasks for his/her perceived priority. The priority vector Y was coded by the pairwise comparison method (the same methodology used in coding the importance of tasks).

No.1 Aviate Navigate absolutely very much moremuch more slightly equally slightly much more very much moreabsolutely

Figure G-8: The sliding bar to obtain the estimated priority of tasks in pairwise comparison

243

G-6. Observed Priority of tasks

The observed priorities of tasks were defined as “functional attendance to tasks”.

The goal of coding of the observed prioritized tasks was to obtain an observed task vector such as,

Z  z1,z2 ,z3,z4   1 0 1 0 (17)

where z1 , z2 , z3 , and z4 are the binary value of attending aviate, navigate, communicate, and Manage Systems tasks, respectively. When a task was observed to be attended, then a binary number of “1” was assigned (otherwise “0” is assigned). The functional attendance of aviate task was estimated by the flight data output from the flight simulator.

The functional attendance of navigate, communicate and Manage Systems were obtained from video records of the flight simulation (See Table B-5).

244

Table G-5: Definition of Each Task

Major Task Descriptions How to detect the functional attendance Any observation of the following subtasks from the flight recorder of X-plane ® * and video record was judged as the execution of aviate task. Aviate Those activities related Task to control of the 1.1 Control aircraft configuration aircraft’s motions, 1.2 Control altitude particularly altitude 1.3 Control lateral profile movement 1.4 Control speed 1.5 Control vertical profile 1.6 Maintain clearness and restrictions 1.7 Maintain separation with traffic, terrain Those activities related Any observation of the follow subtasks from the video to determining the record was judged as the execution of navigate task. Navigate present location and Task how to arrive at an 2.1 Maintain awareness of temporal profile intended location 2.2 Modify or adjust route 2.3 Set navigational radios Those activities related Any observation of the follow subtasks from the video to transmitting record was judged as the execution of communicate task. Communicate information to or Task receiving information 3.1 Communicate with ATC from another human, 3.2 Tune communication radios such as ATC 3.3 Receive ATIS 3.4 Push Indent switch Those activities related Any observation of the follow subtasks from the video Manage to the operating the record was judged as the execution of Manage Systems Systems aircraft’s secondary task. Task equipment, such as its electrical system, 4.1 Conduct Checklist hydraulic system, etc. 4.2 Any other activities that do not belong to Aviate, Navigate, and Communicate tasks 4.3 Control aircraft subsystems

245

G-7. Hypotheses Testing of Five Factors regarding to Perceived Priorities of Tasks

For each factor i 1,2,...,5, the following null and alternative hypotheses were constructed for the perceived priority of each task j  1,2,3,4 .

Null Hypothesis (H0):

There is no significant effect of factor i to the intentional attendance to task j

(i.e., ij  0 )

Alternative Hypothesis (H1):

There is a significant effect of factor i to the intentional attendance to task

(i.e., ij  0 with  level of significance)

Statistical model and hypothesis testing

In order to test the above hypotheses, regression models was constructed to fit the

perceived attendance of each task y j . A fitting model was,

5 4 (18) y j  0  ij X ij i1 j1

In order to test each factor’s effect (i.e., ij  0 ), the Wald’s test was used for the importance of tasks ( i 1), the urgency of tasks (i  2), the salience of tasks (i  4), and workload of tasks (i  5) for each task j  1,2,3,4 .

246

APPENDIX H: PARTICIPANTS RESPONSES TO TASK PRIORITY QUESTIONNAIRES

247

Perceived Perceived Perceived Perceived Concinstency Perceived Perceived Perceived Perceived Importance Perceived Perceived Perceived Perceived Perceived Perceived Perceived Perceived Perceived Perceived Perceived Perceived Consistency Perceived Perceived Perceived Perceived Aviate Navigate Communicate SM Index of Aviate Navigate Communicate SM Concistancy Aviate Navigate Communicate Sys Mgmt Aviate Navigate Communicate Sys Mgmnt Aviate Navigate Communicate Sys Mgmnt Index of Aviate Navigate Communicate Sys Mgmnt Subjects Situations Priority Priority Priority Priority Priority Importance Importance Importance Importance Index Buffer Time BufferTime Buffer Time Buffer Time Performance Performance Performance Performance Salience Salience Salience Salience Salience Workload Workload Workload Workload 1 1 0.58 0.25 0.05 0.13 0.43 0.62 0.23 0.05 0.10 0.19 10 30 50 30 6 5 5 6 0.59 0.23 0.05 0.13 0.28 6.20 2.53 3.80 3.40 1 2 0.64 0.23 0.04 0.09 0.22 0.66 0.19 0.04 0.11 0.21 20 30 30 30 6 5 5 6 0.58 0.25 0.05 0.12 0.28 5.87 2.87 4.20 5.60 1 3 0.61 0.22 0.05 0.12 0.28 0.54 0.26 0.06 0.14 0.27 20 20 30 30 6 5 4 6 0.56 0.28 0.08 0.08 0.13 6.33 4.73 5.87 5.60 1 4 0.60 0.24 0.05 0.11 0.28 0.60 0.15 0.17 0.09 0.50 10 30 30 30 6 6 6 5 0.55 0.16 0.18 0.10 0.32 5.73 5.73 5.33 5.53 1 5 0.59 0.26 0.03 0.12 0.58 0.60 0.24 0.12 0.05 0.35 20 20 30 20 6 4 7 5 0.54 0.26 0.06 0.14 0.27 5.60 5.73 5.40 6.07 1 6 0.58 0.23 0.05 0.14 0.30 0.56 0.24 0.06 0.14 0.34 20 20 40 40 4 4 5 5 0.54 0.21 0.06 0.19 0.10 5.60 5.47 5.47 5.20 1 7 0.56 0.24 0.14 0.06 0.34 0.58 0.18 0.06 0.18 0.13 24 30 36 30 4 4 5 4 0.56 0.23 0.06 0.15 0.19 5.33 5.07 5.33 5.07 1 8 0.54 0.26 0.06 0.14 0.27 0.54 0.26 0.06 0.14 0.27 20 30 40 37 6 6 6 6 0.50 0.29 0.06 0.14 0.20 5.53 5.47 5.27 5.93 2 1 0.19 0.64 0.03 0.13 0.28 0.29 0.54 0.06 0.11 0.20 10 15 40 50 5 6 4 5 0.40 0.40 0.05 0.14 0.00 7.33 5.87 3.87 3.60 2 2 0.59 0.25 0.12 0.04 0.51 0.57 0.26 0.04 0.13 0.38 10 30 50 60 7 6 5 5 0.28 0.58 0.09 0.05 0.19 8.93 7.33 2.53 2.67 2 3 0.58 0.08 0.04 0.30 0.10 0.66 0.04 0.14 0.17 0.25 1 40 15 20 2 6 6 4 0.68 0.04 0.09 0.19 0.16 6.80 6.80 4.60 7.33 2 4 0.56 0.27 0.04 0.13 0.25 0.61 0.25 0.05 0.09 0.25 5 30 60 60 3 6 5 5 0.66 0.21 0.05 0.08 0.15 8.73 3.33 1.73 5.07 2 5 0.54 0.32 0.04 0.10 0.07 0.53 0.27 0.07 0.13 0.07 10 20 60 40 4 6 5 5 0.53 0.28 0.05 0.13 0.19 8.27 3.47 5.33 5.40 2 6 0.66 0.11 0.11 0.12 0.40 0.57 0.24 0.12 0.07 0.13 15 20 30 40 6 6 6 5 0.60 0.05 0.24 0.10 0.14 7.67 4.53 5.93 5.20 2 7 0.54 0.06 0.29 0.11 0.09 0.59 0.20 0.15 0.06 0.15 15 30 30 40 4 4 3 6 0.58 0.06 0.25 0.11 0.05 7.07 3.80 7.27 5.33 2 8 0.62 0.22 0.10 0.06 0.15 0.65 0.19 0.06 0.10 0.24 5 30 45 30 6 4 5 5 0.64 0.21 0.10 0.05 0.10 7.00 5.73 6.53 5.33 3 1 0.28 0.35 0.06 0.31 1.61 0.29 0.33 0.22 0.16 1.71 5 10 15 31 3 1 6 5 0.19 0.64 0.09 0.08 0.08 3 2 0.52 0.29 0.06 0.12 0.09 0.38 0.23 0.09 0.30 0.18 3 20 30 15 7 6 7 7 0.41 0.31 0.07 0.21 0.04 3 3 0.34 0.32 0.21 0.12 0.92 0.42 0.21 0.20 0.17 0.77 5 20 30 20 7 7 8 8 0.27 0.15 0.50 0.08 0.38 3 4 0.30 0.13 0.03 0.53 0.42 0.29 0.26 0.03 0.42 0.31 11 10 30 10 7 2 5 6 0.13 0.44 0.05 0.39 0.10 3 5 0.57 0.22 0.04 0.17 0.28 0.60 0.25 0.04 0.12 0.24 15 10 30 15 7 7 8 7 0.26 0.57 0.04 0.14 0.23 3 6 0.20 0.64 0.12 0.05 0.26 0.27 0.56 0.13 0.05 0.21 15 15 30 30 8 8 7 7 0.25 0.62 0.06 0.06 0.17 3 7 0.15 0.64 0.14 0.07 0.30 0.15 0.22 0.35 0.28 0.28 11 10 15 30 7 7 7 7 0.06 0.28 0.50 0.16 0.28 3 8 0.61 0.25 0.04 0.10 0.50 0.59 0.24 0.04 0.12 0.35 1 10 40 30 1 1 5 5 0.36 0.34 0.26 0.04 2.48 4 1 0.63 0.14 0.04 0.19 0.39 0.66 0.14 0.04 0.15 0.10 1 46 60 20 7 7 6 6 0.50 0.18 0.05 0.28 0.20 6.40 4.27 4.53 5.00 4 2 0.49 0.19 0.04 0.27 0.24 0.68 0.15 0.04 0.13 0.15 10 60 60 15 9 7 7 6 0.68 0.15 0.04 0.13 0.15 5.00 3.87 4.93 5.80 4 3 0.67 0.06 0.06 0.21 0.09 0.67 0.06 0.06 0.21 0.09 10 50 60 20 7 6 6 5 0.65 0.06 0.06 0.23 0.05 6.93 5.67 5.00 6.60 4 4 0.68 0.18 0.06 0.07 0.12 0.64 0.20 0.04 0.11 0.28 1 20 60 17 9 7 7 6 0.66 0.12 0.07 0.15 0.42 7.47 6.07 4.60 6.60 4 5 0.68 0.15 0.04 0.13 0.15 0.68 0.11 0.04 0.16 0.20 10 30 60 20 9 7 7 6 0.67 0.18 0.09 0.06 0.24 6.33 5.53 5.40 6.40 4 6 0.72 0.14 0.06 0.08 0.13 0.70 0.18 0.06 0.06 0.10 10 20 40 30 9 9 8 5 0.70 0.18 0.06 0.06 0.10 6.87 5.87 3.87 3.73 4 7 0.70 0.18 0.06 0.06 0.10 0.70 0.18 0.06 0.06 0.10 10 20 40 20 9 8 6 5 0.72 0.14 0.06 0.08 0.13 6.13 4.93 3.67 5.87 4 8 0.68 0.05 0.12 0.15 0.07 0.66 0.04 0.13 0.17 0.15 10 30 20 20 9 7 7 7 0.66 0.04 0.15 0.15 0.14 7.40 4.73 5.40 6.53 5 1 0.61 0.12 0.05 0.23 0.10 0.55 0.12 0.05 0.27 0.17 5 30 46 18 7 6 6 4 0.55 0.12 0.05 0.27 0.17 5 2 0.52 0.12 0.08 0.28 0.13 0.60 0.10 0.05 0.24 0.14 4 20 45 20 6 6 7 6 0.60 0.10 0.05 0.24 0.14 5 3 0.53 0.13 0.07 0.27 0.07 0.56 0.12 0.06 0.26 0.04 5 60 30 30 6 6 6 6 0.54 0.23 0.07 0.16 0.13 5 4 0.62 0.11 0.05 0.22 0.09 0.57 0.13 0.07 0.24 0.05 6 60 60 20 7 7 7 6 0.62 0.11 0.05 0.22 0.09 5 5 0.58 0.14 0.06 0.22 0.14 0.61 0.13 0.06 0.20 0.09 6 21 30 30 7 7 6 6 0.63 0.19 0.06 0.12 0.13 5 6 0.25 0.50 0.10 0.15 0.13 0.46 0.27 0.17 0.10 0.12 6 11 14 30 7 7 6 6 0.45 0.31 0.15 0.09 0.19 5 7 0.46 0.27 0.10 0.17 0.12 0.46 0.27 0.10 0.17 0.12 10 30 30 30 6 6 6 6 0.54 0.23 0.16 0.07 0.13 5 8 0.06 0.20 0.61 0.13 0.09 0.07 0.22 0.53 0.18 0.19 47 33 5 20 6 6 6 6 0.07 0.24 0.57 0.12 0.13 5.00 6 1 0.61 0.25 0.09 0.05 0.21 0.60 0.24 0.11 0.05 0.21 10 15 30 20 7 7 5 5 0.63 0.24 0.09 0.04 0.17 5.13 5.20 4.60 5.80 6 2 0.59 0.23 0.13 0.05 0.45 0.61 0.25 0.07 0.07 0.08 5 5 25 32 7 7 6 6 0.57 0.28 0.10 0.06 0.09 7.20 6.40 5.80 5.87 6 3 0.45 0.26 0.14 0.15 0.53 0.44 0.34 0.14 0.08 0.19 7 9 25 25 7 7 5 6 0.57 0.25 0.11 0.06 0.26 5.27 4.07 4.20 4.33 6 4 0.51 0.30 0.07 0.12 0.12 0.51 0.30 0.07 0.12 0.12 5 8 25 40 7 7 6 6 0.44 0.41 0.09 0.06 0.09 4.93 6 5 0.61 0.28 0.06 0.05 0.11 0.33 0.33 0.31 0.03 0.96 6 5 3 30 7 7 7 5 0.26 0.14 0.54 0.06 0.27 6.20 5.87 4.13 4.33 7 1 0.55 0.11 0.06 0.27 0.19 0.42 0.08 0.08 0.42 0.00 5 20 20 11 7 3 3 5 0.66 0.17 0.06 0.11 0.12 7 2 0.64 0.24 0.06 0.06 0.15 0.70 0.10 0.10 0.10 0.00 5 20 30 30 7 3 5 5 0.69 0.17 0.06 0.09 0.21 7 3 0.63 0.13 0.13 0.13 0.00 0.63 0.17 0.09 0.11 0.06 10 26 30 30 5 5 7 6 0.63 0.13 0.13 0.13 0.00 7 4 0.67 0.17 0.05 0.12 0.11 0.68 0.14 0.08 0.10 0.06 5 20 30 30 5 5 5 5 0.67 0.11 0.11 0.11 0.01 7 5 0.68 0.10 0.14 0.08 0.06 0.68 0.08 0.10 0.14 0.06 5 21 30 30 7 4 6 6 0.72 0.12 0.07 0.09 0.06 7 6 0.51 0.04 0.15 0.30 0.14 0.56 0.05 0.18 0.20 0.06 8 30 15 15 3 5 3 3 0.61 0.05 0.17 0.17 0.08 7 7 0.70 0.10 0.10 0.10 0.00 0.70 0.10 0.10 0.10 0.00 6 30 30 30 6 5 6 6 0.70 0.10 0.10 0.10 0.00 7 8 0.67 0.17 0.07 0.10 0.13 0.65 0.21 0.07 0.07 0.10 5 7 19 27 5 5 5 5 0.63 0.23 0.07 0.07 0.13 8 1 0.52 0.27 0.05 0.15 0.46 0.59 0.24 0.04 0.14 0.29 6 20 32 20 8 8 8 7 0.56 0.21 0.04 0.18 0.31 7.50 5.20 5.13 5.67 8 2 0.62 0.23 0.04 0.11 0.23 0.61 0.17 0.04 0.18 0.16 8 25 30 11 9 8 8 8 0.65 0.15 0.04 0.16 0.07 7.07 6.40 6.80 6.73 8 3 0.68 0.15 0.04 0.13 0.15 0.61 0.18 0.04 0.18 0.09 20 30 50 20 8 7 7 7 0.63 0.20 0.04 0.13 0.21 5.07 4.13 5.73 5.67 8 4 0.37 0.11 0.34 0.18 0.10 0.25 0.25 0.25 0.25 0.00 20 40 40 30 8 8 8 8 0.25 0.25 0.25 0.25 0.00 4.20 5.00 5.00 5.27 8 5 0.58 0.13 0.13 0.15 0.01 0.61 0.17 0.10 0.12 0.06 20 30 30 30 8 8 7 7 0.61 0.15 0.09 0.14 0.09 5.33 5.20 4.67 4.87 8 6 0.42 0.29 0.11 0.17 0.07 0.44 0.23 0.16 0.17 0.10 20 20 30 20 5 5 5 5 0.56 0.23 0.06 0.15 0.19 5.13 4.47 5.13 5.13 8 7 0.64 0.20 0.04 0.11 0.16 0.65 0.16 0.04 0.15 0.13 6 15 30 11 9 9 9 9 0.65 0.16 0.04 0.15 0.13 6.20 4.60 5.73 6.07 8 8 0.61 0.25 0.04 0.10 0.50 0.61 0.24 0.03 0.12 0.58 8 20 30 15 7 7 7 7 0.55 0.24 0.03 0.18 0.26 5.33 4.87 5.07 5.07

248

Perceived Perceived Perceived Perceived Concinstency Perceived Perceived Perceived Perceived Importance Perceived Perceived Perceived Perceived Perceived Perceived Perceived Perceived Perceived Perceived Perceived Perceived Consistency Perceived Perceived Perceived Perceived Aviate Navigate Communicate SM Index of Aviate Navigate Communicate SM Concistancy Aviate Navigate Communicate Sys Mgmt Aviate Navigate Communicate Sys Mgmnt Aviate Navigate Communicate Sys Mgmnt Index of Aviate Navigate Communicate Sys Mgmnt Subjects Situations Priority Priority Priority Priority Priority Importance Importance Importance Importance Index Buffer Time BufferTime Buffer Time Buffer Time Performance Performance Performance Performance Salience Salience Salience Salience Salience Workload Workload Workload Workload 9 1 0.43 0.39 0.07 0.11 0.08 0.43 0.37 0.07 0.13 0.04 5 30 30 10 4 4 4 6 0.52 0.28 0.12 0.08 0.13 6.27 5.93 6.27 4.80 9 2 0.42 0.36 0.06 0.16 0.10 0.56 0.22 0.15 0.07 0.18 5 30 30 30 6 7 7 7 0.56 0.15 0.15 0.15 0.39 5.87 5.87 4.87 5.47 9 3 0.61 0.12 0.23 0.05 0.10 0.51 0.15 0.26 0.08 0.26 5 20 20 30 7 7 7 4 0.56 0.12 0.26 0.06 0.28 7.60 5.27 5.73 6.13 9 4 0.39 0.44 0.05 0.12 0.02 0.36 0.36 0.10 0.17 0.06 20 20 30 20 8 7 8 7 0.14 0.25 0.06 0.55 0.21 6.20 6.67 3.47 7.07 9 5 0.38 0.44 0.12 0.05 0.14 0.41 0.27 0.23 0.09 0.19 30 10 10 10 4 6 7 6 0.15 0.49 0.27 0.08 0.22 7.53 6.80 6.93 6.21 9 6 0.30 0.51 0.14 0.04 0.20 0.25 0.50 0.20 0.06 0.16 20 20 36 5 7 7 7 7 0.08 0.20 0.20 0.52 0.02 6.20 6.47 6.54 5.13 9 7 0.49 0.24 0.20 0.07 0.49 0.29 0.29 0.33 0.09 0.01 10 10 10 20 7 6 7 4 0.28 0.42 0.24 0.05 0.29 6.07 5.40 3.93 3.13 9 8 0.58 0.22 0.14 0.06 0.10 0.30 0.36 0.18 0.15 0.18 5 10 30 30 6 4 7 1 0.62 0.23 0.11 0.05 0.34 8.27 5.47 2.13 6.20 10 1 0.51 0.12 0.10 0.27 0.04 0.30 0.30 0.10 0.30 0.00 3 10 15 10 6 6 6 5 0.08 0.16 0.50 0.26 0.07 7.20 6.93 6.80 6.80 10 2 0.48 0.10 0.21 0.21 0.06 0.48 0.10 0.21 0.21 0.06 5 20 10 10 6 5 7 4 0.52 0.08 0.20 0.20 0.02 6.33 6.73 5.40 5.73 10 3 0.38 0.23 0.10 0.29 0.06 0.38 0.23 0.10 0.29 0.06 5 10 10 10 6 5 5 5 0.34 0.28 0.31 0.07 0.55 6.20 6.40 6.13 6.27 10 4 0.20 0.52 0.08 0.20 0.02 0.20 0.52 0.08 0.20 0.02 8 7 30 15 6 7 5 6 0.26 0.53 0.07 0.15 0.14 6.80 6.20 5.60 5.73 10 5 0.17 0.50 0.17 0.17 0.00 0.17 0.50 0.17 0.17 0.00 5 5 10 10 6 4 6 6 0.16 0.52 0.21 0.11 0.04 5.87 6.73 6.73 6.67 10 6 0.38 0.38 0.13 0.13 0.00 0.38 0.38 0.13 0.13 0.00 10 10 15 13 6 6 6 4 0.39 0.39 0.15 0.07 0.02 6.87 6.67 5.73 5.40 10 7 0.58 0.25 0.09 0.08 0.10 0.48 0.22 0.13 0.16 0.06 5 5 10 10 3 4 6 6 0.52 0.25 0.13 0.09 0.27 6.40 7.13 6.40 6.27 10 8 0.53 0.26 0.07 0.15 0.14 0.53 0.26 0.07 0.15 0.14 3 5 10 7 4 4 7 6 0.55 0.27 0.06 0.11 0.19 6.40 7.27 6.47 6.87 11 1 0.65 0.14 0.13 0.08 0.22 0.73 0.12 0.07 0.08 0.06 2 5 30 42 3 4 6 6 0.61 0.22 0.04 0.13 0.40 11 2 0.73 0.08 0.07 0.12 0.06 0.73 0.11 0.05 0.11 0.06 1 12 40 20 3 5 7 4 0.66 0.21 0.04 0.10 0.28 11 3 0.30 0.39 0.13 0.18 0.06 0.31 0.37 0.07 0.25 0.10 5 20 36 30 4 7 7 5 0.66 0.14 0.04 0.15 0.10 11 4 0.41 0.31 0.04 0.24 0.06 0.71 0.11 0.05 0.13 0.10 5 20 52 30 6 6 6 6 0.58 0.17 0.06 0.19 0.37 11 5 0.70 0.08 0.06 0.15 0.11 0.68 0.10 0.08 0.14 0.17 5 20 30 15 6 6 6 6 0.50 0.17 0.07 0.26 0.14 11 6 0.66 0.16 0.05 0.13 0.10 0.64 0.20 0.05 0.10 0.13 5 10 30 25 6 6 7 7 0.66 0.13 0.05 0.16 0.10 11 7 0.67 0.17 0.04 0.12 0.19 0.68 0.16 0.04 0.11 0.20 5 15 30 30 7 7 6 6 0.71 0.13 0.05 0.11 0.10 11 8 0.69 0.13 0.04 0.14 0.11 0.67 0.17 0.04 0.12 0.19 5 10 30 20 6 6 6 7 0.65 0.18 0.04 0.12 0.36 12 1 0.52 0.32 0.04 0.12 0.24 0.55 0.29 0.04 0.12 0.22 10 60 60 60 8 7 9 8 0.62 0.22 0.13 0.04 0.16 12 2 0.55 0.24 0.09 0.12 0.28 0.65 0.21 0.04 0.09 0.12 10 60 60 30 8 8 8 7 0.65 0.07 0.07 0.21 0.10 12 3 0.56 0.27 0.13 0.05 0.21 0.43 0.39 0.07 0.11 0.08 15 30 60 60 9 8 7 6 0.14 0.23 0.58 0.05 0.30 12 4 0.54 0.32 0.09 0.05 0.12 0.44 0.44 0.06 0.06 0.00 15 30 60 60 8 9 8 8 0.28 0.57 0.05 0.11 0.28 12 5 0.52 0.25 0.06 0.17 0.19 0.57 0.29 0.07 0.07 0.14 10 60 60 40 7 6 7 7 0.41 0.41 0.05 0.14 0.13 12 6 0.54 0.32 0.09 0.05 0.12 0.58 0.28 0.09 0.05 0.19 15 30 45 60 9 9 8 8 0.33 0.56 0.05 0.06 0.06 12 7 0.57 0.28 0.11 0.05 0.28 0.60 0.27 0.08 0.05 0.19 10 20 40 60 7 9 8 7 0.12 0.26 0.54 0.09 0.05 12 8 0.41 0.41 0.07 0.12 0.06 0.52 0.31 0.05 0.13 0.14 5 10 30 45 9 9 7 7 0.41 0.20 0.20 0.19 0.37 13 1 0.09 0.18 0.49 0.24 0.23 0.27 0.10 0.17 0.46 0.12 3 4 2 7 3 4 3 3 0.08 0.31 0.31 0.31 0.39 13 2 0.39 0.35 0.08 0.19 0.10 0.47 0.14 0.06 0.32 0.13 4 8 3 3 4 5 4 5 0.48 0.18 0.06 0.28 0.12 13 3 0.08 0.26 0.50 0.16 0.07 0.09 0.24 0.50 0.17 0.19 3 3 2 5 6 4 3 4 0.13 0.46 0.22 0.19 0.18 13 4 0.30 0.43 0.04 0.23 0.55 0.42 0.32 0.04 0.23 0.05 4 3 5 2 6 6 5 4 0.25 0.55 0.06 0.14 0.11 13 5 0.19 0.22 0.36 0.23 0.97 0.08 0.13 0.42 0.38 0.10 3 2 3 4 4 4 3 4 0.17 0.27 0.46 0.10 0.12 13 6 0.54 0.22 0.09 0.14 0.19 0.27 0.20 0.27 0.27 0.39 2 6 3 2 6 3 3 4 0.48 0.21 0.10 0.21 0.06 13 7 0.30 0.36 0.15 0.18 0.18 0.31 0.31 0.31 0.08 0.39 2 2 3 3 3 3 3 3 0.23 0.38 0.29 0.10 0.06 13 8 0.36 0.25 0.16 0.23 0.76 0.30 0.38 0.23 0.09 0.18 3 3 5 5 3 4 4 3 0.31 0.23 0.31 0.14 0.30 14 1 0.43 0.40 0.11 0.06 0.05 0.66 0.19 0.10 0.04 0.28 30 60 43 60 7 6 6 4 0.64 0.20 0.10 0.06 0.20 14 2 0.66 0.19 0.10 0.04 0.28 0.68 0.20 0.08 0.04 0.20 15 30 60 44 7 6 6 4 0.59 0.22 0.14 0.05 0.14 14 3 0.66 0.24 0.05 0.05 0.20 0.61 0.25 0.11 0.04 0.37 5 30 60 33 7 6 4 5 0.63 0.27 0.05 0.05 0.26 14 4 0.66 0.24 0.05 0.05 0.20 0.66 0.24 0.05 0.05 0.20 7 20 60 45 6 7 6 4 0.66 0.24 0.05 0.05 0.20 14 5 0.64 0.25 0.05 0.05 0.23 0.66 0.24 0.05 0.05 0.20 10 30 60 60 6 6 5 4 0.64 0.25 0.05 0.05 0.23 14 6 0.61 0.20 0.13 0.06 0.09 0.60 0.23 0.08 0.09 0.16 20 40 60 50 7 6 6 5 0.39 0.30 0.18 0.13 0.06 14 7 0.69 0.20 0.06 0.06 0.13 0.69 0.20 0.06 0.06 0.13 10 30 60 40 7 6 5 5 0.69 0.20 0.06 0.06 0.13 14 8 0.66 0.05 0.05 0.24 0.20 0.57 0.05 0.05 0.32 0.05 11 20 30 30 6 6 6 5 0.45 0.12 0.07 0.36 0.08 15 1 0.60 0.13 0.07 0.19 0.16 0.64 0.17 0.05 0.13 0.17 10 45 60 30 3 4 5 6 0.57 0.16 0.07 0.20 0.20 15 2 0.19 0.29 0.05 0.47 0.16 0.56 0.15 0.06 0.23 0.19 10 30 45 30 6 6 5 7 0.45 0.26 0.09 0.20 0.19 15 3 0.22 0.17 0.28 0.33 0.66 0.61 0.15 0.06 0.18 0.05 10 45 60 30 4 4 4 4 0.56 0.17 0.08 0.19 0.04 15 4 0.21 0.43 0.06 0.30 0.13 0.63 0.17 0.04 0.17 0.05 10 30 60 30 7 6 5 6 0.55 0.20 0.07 0.18 0.09 15 5 0.36 0.17 0.10 0.36 0.06 0.54 0.18 0.14 0.15 0.55 10 15 20 30 4 3 4 5 0.46 0.17 0.10 0.27 0.12 15 6 0.35 0.10 0.10 0.45 0.13 0.46 0.10 0.17 0.27 0.12 10 20 50 20 4 4 4 4 0.38 0.26 0.09 0.28 0.10 15 7 0.33 0.37 0.08 0.22 0.19 0.33 0.28 0.22 0.17 0.66 15 20 15 40 5 6 6 6 0.10 0.27 0.46 0.17 0.12 15 8 0.61 0.26 0.06 0.08 0.18 0.54 0.24 0.14 0.08 0.08 6 10 15 30 5 6 6 6 0.60 0.19 0.13 0.07 0.16 16 1 0.48 0.25 0.09 0.18 0.30 0.35 0.16 0.18 0.31 0.18 1 10 20 11 4 4 4 4 0.44 0.25 0.09 0.22 0.51 16 2 0.47 0.12 0.17 0.24 0.43 0.41 0.41 0.07 0.12 0.06 1 10 20 30 6 6 7 6 0.61 0.12 0.17 0.10 0.06 16 3 0.22 0.28 0.45 0.05 0.24 0.30 0.23 0.43 0.05 0.55 3 10 7 20 6 6 6 6 0.46 0.15 0.26 0.14 0.80 16 4 0.18 0.17 0.61 0.04 0.16 0.26 0.13 0.56 0.05 0.51 3 4 2 9 6 6 7 6 0.39 0.13 0.42 0.05 0.26 16 5 0.35 0.24 0.30 0.11 0.53 0.18 0.22 0.43 0.16 1.01 1 7 5 11 6 6 7 6 0.47 0.07 0.32 0.14 0.20 16 6 0.34 0.14 0.29 0.23 1.73 0.22 0.13 0.61 0.04 0.40 1 5 4 10 6 6 7 6 0.41 0.22 0.33 0.05 0.54 16 7 0.37 0.22 0.29 0.12 0.52 0.54 0.14 0.26 0.06 0.27 1 6 3 5 6 6 6 6 0.64 0.10 0.18 0.08 0.17 16 8 0.62 0.04 0.26 0.09 0.42 0.62 0.04 0.22 0.12 0.29 1 13 1 1 7 6 7 7 0.63 0.04 0.24 0.09 0.36

249

APPENDIX I: ADDITIONAL ANALYSIS

250

Figure I-1 shows the task priority scores of two malfunction problems that required deeper level-2 situation awareness but without any direct visual cue : Pitot tube clog problem (the left graph) and static port clog problem (the right graph). The perceived priority did not show a significant difference between the hit and miss situations.

Figure I-1: Comparison of perceived task priority with four situations (Hit, Miss, False Alarm, and Correct Rejection) for four problems (8. Pitot tube clog root cause, 9. Static port-clog root cause).

The above findings indicate two points about the perceived task priority scores and the inattention/change blindness problems. Participants made false alarms in relation to the flight instrument problem when the associated task priority was low, while participants missed the instrument panel malfunction problems even when the associated task priority was high.

251

The Perceived task priority and Time spent for the Task

An investigation was conducted to determine how much time was spent in executing each task during the each flight situation. In Figure I-2, the left graph shows the proportion of actual time spent on each task, and the right graph shows the priority score of each task. As can be seen in the graphs, the proportion of task execution time and the priority score of tasks both have a similar distribution; the Aviate task has the highest priority score (0.50), followed by Navigate (0.24), Communicate (0.12), and

Manage Systems tasks (0.15). The mean proportion of task time was 59.6% for the

Aviate task, 26.4% for the Navigate task, 12.2% for the Communicate task, and 1.8% for the Manage Systems task.

252

Figure I-2: Spent Time on each task and Perceived priority of each task

253

Relationships between the perceived task priority score and the actual spent time were investigated. Figure I-3 shows scatter plots of the proportion of time spent on each task (Y-axis) according to the perceived task priority score (X-axis). Blue dots indicate the observed data points. Linear models were constructed for each task, and the coefficients of the slopes were tested. There was a linear relationship between the time proportion of Aviate task execution and the perceived Aviate task priority score. The above model had a slope P-value=0.004 and sum of squares F-test P-Value=0.11. There was a quadratic relationship between the time proportion of Navigate task execution and the perceived Navigate task priority score. This model had sum of squares F-test P- value=0.11 for the linear term, and P-value=0.001 for the quadratic term. There was a linear relationship between the time proportion of executing Communicate task and the perceived Communicate task priority score, which had a Wald’s test P-Value=0.002, and sum of squares F-test P-Value=0.005. There was not a clear linear relationship between the time proportion of executing Manage Systems task and the perceived Manage

Systems task priority score. This had a Wald’s test slope P-value=0.17, Sum of Square

F-test P-value=0.18).

254

Figure I-3: Task Execution Time Proportion and Perceived task priority Scores

255

The subtasks of Communicate task had even more significant relationship than the parent (i.e., perceived Communicate task) task priority. As can be seen in Figure I-

4, the perceived task priority of Communicate task had a strong linear relationship with the actual time spent for Communicate subtasks. Figure I-5 shows that the time spent for checklists had a relationship with the perceived task priority for Manage Systems. But there was not a relationship between the execution time for looking at the map / writing memo subtask and the perceived task priority score of Manage Systems.

256

Figure I-4: The Communicate Subtask Execution Time and the Perceived Task

257

Figure I-5: Task Execution Time and Perceived task priority Score of Manage Systems Task