MITIGATING COGNITIVE BIASES IN RISK IDENTIFICATION: Practitioner Checklist for the AEROSPACE SECTOR Debra L. Emmons, Thomas A. Mazzuchi, Shahram Sarkani, and Curtis E. Larsen This research contributes an operational checklist for mitigating cogni- tive biases in the aerospace sector risk management process. The Risk Identification and Evaluation Bias Reduction Checklist includes steps for grounding the risk identification and evaluation activities in past project experiences through historical data, and emphasizes the importance of incorporating multiple methods and perspectives to guard against optimism and a singular project instantiation-focused view. The authors developed a survey to elicit subject matter expert judgment on the value of the check- list to support its use in government and industry as a risk management tool. The survey also provided insights on bias mitigation strategies and lessons learned. This checklist addresses the deficiency in the literature in providing operational steps for the practitioner to recognize and implement strategies for bias reduction in risk management in the aerospace sector. DOI: https://doi.org/10.22594/dau.16-770.25.01 Keywords: Risk Management, Optimism Bias, Planning Fallacy, Cognitive Bias Reduction Mitigating Cognitive Biases in Risk Identification http://www.dau.mil January 2018 This article and its accompanying research contribute an operational FIGURE 1. RESEARCH APPROACH Risk Identification and Evaluation Bias Reduction Checklist for cognitive bias mitigation in risk management for the aerospace sector. The checklist Cognitive Biases & Bias described herein offers a practical and implementable project management Enabling framework to help reduce biases in the aerospace sector and redress the Conditions cognitive limitations in the risk identification and analysis process. Detailed discussion focuses on the development of strategies targeted at reducing Key Biases: Optimism four cognitive biases and their influence on the risk identification process, Planning Fallacy Anchoring and to the development and validation of the practitioner checklist. Ambiguity Effect Open-ended Likert scale Questions Expert Judgment Expert Judgment Final Bias Initial Literature Survey: Survey: Reduction Background and Literature Bias Reduction Review Checklist Bias Reduction Checklist & Checklist The authors’ research began with a review of the literature, which Validation Strategies Strategies covered the areas of risk management, cognitive biases, and bias enabling conditions. The biases of optimism, planning fallacy, anchoring, and ambiguity Bias Risk Mitigation effect were deemed particularly influential to the risk identification and Management Approaches evaluation processes. The authors reviewed and synthesized the bias mit- igation literature and developed the initial bias reduction checklist. After the development of the initial checklist, they designed and administered The checklist presented herein is grounded in the academic literature the survey to seek feedback and validation of the checklist surrounding cognitive bias mitigation and, in particular, the Nobel-prize- as a risk management tool. A Likert-scale instrument was winning efforts of Kahneman and Tversky (1977) in reference class used for the survey, which is an appropriate instrument for forecasting. The review of the literature begins with a discussion of risk measuring attitudes and beliefs. The answers to the open-ended management in the aerospace sector and a description of the risk identi- questions of the survey provided insights, lessons learned, as well fication practices and challenges. Subsequently, the nature of cognitive as other measures that are used by practitioners to biases is described. These biases are persistent across industries, individual reduce biases. The survey design, data experts, and teams, and affect humans’ ability to impartially identify and collection, and analysis followed the assess risks. Bias enabling conditions in the project environment are also academic literature guidelines for garner- examined, and the characteristics common to both the transportation and ing attitudes and feedback on the effectiveness aerospace sectors are highlighted. Although the research is tailored to the of the checklist as a risk management tool. aerospace sector, important insights from the transportation sector are also Nonetheless, the authors recognize that like considered. Finally, the review of the literature concludes with a discussion any of the measurement methods in the science of the approaches to reduce the cognitive biases. disciplines, the social or attitude survey method is not error-free (Fowler, 2013). The authors Risk Management incorporated the feedback from both the Risk management includes a documented process, and both formal Likert survey and the open-ended ques- and informal practices applied in government programs and commer- tions into the final checklist. Finally, cial industries alike. The Department of Defense (DoD) and the National a discussion follows on the checklist Aeronautics and Space Administration (NASA) have established risk implementation and potential chal- management processes. For example, the Department of Defense Risk, Issue, lenges. The research approach is and Opportunity Management Guide for Defense Acquisition Programs highlighted in Figure 1. represents but one of numerous DoD policy and guidance documents that 54 Defense ARJ, January 2018, Vol. 25 No. 1 : 52–93 Defense ARJ, January 2018, Vol. 25 No. 1 : 52–93 55 Mitigating Cognitive Biases in Risk Identification http://www.dau.mil January 2018 focus on risk management. Figure 2 depicts an overview of the risk and issue Maytorena, Winch, Freeman, and Kiely (2007) highlight the “importance of management process, which is an organized and iterative decision-making the risk identification and analysis phases of the risk management process” technique designed to improve the probability of project success (DoD, (p. 315), since they can have a great influence on the correctness of the risk 2017). This five-step management process may be used for issues that assessment activity. Their research suggests that the role of experience in are nonprobabilistic in nature, or risks. This process is intended to be a this process is much less meaningful than it is regularly presumed to be. proactive and continuous approach that identifies discrete risks or issues, Alternately, “information search approach, education, and training in risk assesses the likelihood and consequence of these risks or consequences of management have a significant role in risk identification performance” the issues, develops mitigation options for all the identified risks, monitors (p. 315). As for the role of expertise in the risk identification and analysis progress to confirm that cumulative project risk is truly declining, and process, Freudenburg (1988) described the challenges amongst specialists communicates the risk status (DoD, 2017). The DoD risk-mitigation options that may lead to cognitive miscalculations in the risk estimation methods, include acceptance (and monitoring), avoidance, transfer, and control (DoD, including the failure to anticipate all the elements. Such miscalculations 2017). Similar continuous risk management processes have also been rep- may then lead to mistakes and bias in the estimating. The identification resented in the NASA guidance (NASA, 2007) and A Guide to the Project or risk discovery methods and tools typically include brainstorming, per- Management Body of Knowledge (Project Management Institute, 2013). A sonal knowledge and experience, questionnaires, lessons learned, and risk review of the literature also indicates that project risk management prac- management tools such as Failure Modes Effects and Analysis, Fault Tree tices differ across projects and are affected by project characteristics such Analysis, and Probability Risk Analysis. as scope, complexity, and category (Omidvar, Jaryani, Zafarghandi, Nasab, & Jam Shidi, 2011). Risk identification is a continuing process throughout the life cycle of the project; however, it is critically important in the early conceptual design FIGURE 2. RISK AND ISSUE MANAGEMENT PROCESS OVERVIEW and formulation phases to ensure the appropriate risk and programmatic posture is established. A study by the Jet Propulsion Laboratory noted Process “significant variability in risk identification and risk reporting in the early Planning conceptual design” (Hihn, Chattopadhyay, Hanna, Port, & Eggleston, 2010, What are the Identification p. 14). Some of this variability is attributable to the inherent vagueness of program’s risk and What has, can, or new system design at this early phase in the life cycle. Further exacerbating issue management will go wrong? processes? factors include the hectic concurrent engineering design team environment and the absence of an organized risk identification and ranking process that would potentially increase the level of evenness across risk-recording activities. This team concluded, “Generating risk checklists that can be used Communication for risk identification guidance during early concept studies would enable and Feedback more consistent risk reporting” (Hihn et al., 2010, p. 14). Monitoring Analysis What is the likelihood How has the risk Cognitive Biases of the risk and the or issue consequence of the The issue of bias based on human mental shortcuts (also called heuris- changed? risk or issue? tics) in subjective assessment and decision