MTR160163 MITRE TECHNICAL REPORT The Assessment of Biases in Cognition This document reports on work supported by the Office of the Development and Evaluation of an Director of National Intelligence Assessment Instrument for the Measurement (ODNI), Intelligence Advanced Research Projects Activity of Cognitive Bias (IARPA), via contract 2015- 14120200002-002, and is subject to the Rights in Data-General Clause 52-227.14, Alt. IV (DEC 2007). The views and conclusions contained herein are those of the Abigail Gertner, The MITRE Corporation authors and should not be interpreted as necessarily Franklin Zaromb, Educational Testing Service representing the official policies or Robert Schneider, Research & Assessment endorsements, either expressed or Solutions, Ltd. implied, of IARPA, ODNI, or the U.S. Government. The U.S. Richard D. Roberts, Professional Examination Government is authorized to Service reproduce and distribute reprints for Government purposes Gerald Matthews, University of Central Florida notwithstanding any copyright annotation thereon. ©2016 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 16-0956 Abstract The Assessment of Biases in Cognition (ABC) is a new standardized assessment of biases in judgment and decision-making behavior that was developed by the MITRE Corporation and the Educational Testing Service (ETS) for the Intelligence Advanced Research Projects Activity (IARPA) Sirius Program. The purpose of the IARPA Sirius Program is to create serious video games designed to train intelligence analysts to improve their explicit knowledge of, and ability to recognize, six well-known cognitive biases and to significantly mitigate the influence of those biases on behavior as a result of this training. The six biases are: (1) confirmation bias (CB), (2) fundamental attribution error (FAE), (3) bias blind spot (BBS), (4) anchoring bias (ANC), (5) representativeness bias (REP), and (6) projection bias (PRO). The first version of the ABC (ABC-1) was developed for the first phase of the Sirius Program to assess mitigation of CB, FAE, and BBS. The second version of the ABC (ABC-2) was developed for use in second phase of the Sirius Program to assess mitigation of ANC, REP, and PRO. The ABC-1 and the ABC-2 each include one recognition and discrimination (RD) scale and three behavioral elicitation (BE) scales, one for each bias. The RD scales consist primarily of multiple- choice items and are intended to assess declarative knowledge of the biases. The BE scales consist of a variety of innovative tasks intended to evaluate test-takers’ procedural knowledge regarding how to avoid committing the targeted biases in judgment and decision-making tasks specifically designed to give test-takers opportunities to commit those biases. Each version of the ABC is administered online using a customized test delivery platform developed by the MITRE Corporation and takes approximately 45 to 60 minutes to complete. The ABC-1 and ABC-2 both include three equated test forms. This facilitated evaluation of bias mitigation training outcomes by making it possible to compare test-takers’ performance on one form post-training with their pre-training performance on an alternate, equated ABC test form. This report summarizes the (1) test development process, (2) research conducted during the development and validity evaluation of the ABC, (3) validity argument for the ABC, and (4) suggestions for future research. iii This page intentionally left blank. iv Executive Summary The Assessment of Biases in Cognition (ABC) is a new standardized assessment of biases in judgment and decision-making behavior that was developed by the MITRE Corporation and the Educational Testing Service (ETS) for the Intelligence Advanced Research Projects Activity (IARPA) Sirius Program. The purpose of the IARPA Sirius Program is to create serious video games designed to train intelligence analysts to improve their explicit knowledge of, and ability to recognize, six well-known cognitive biases and to significantly mitigate the influence of those biases on behavior as a result of this training. The six biases are: (1) confirmation bias (CB), (2) fundamental attribution error (FAE), (3) bias blind spot (BBS), (4) anchoring bias (ANC), (5) representativeness bias (REP), and (6) projection bias (PRO). The Sirius Program was divided into two phases. Phase 1 encompassed biases 1–3 and took place between October 2011 and September 2013. The first version of the ABC (ABC-1) was developed for use in the Phase 1 Independent Validation and Verification (IV&V) study to assess mitigation of CB, FAE, and BBS. Phase 2 encompassed biases 4–6 and took place between September 2013 and November 2015. The second version of the ABC (ABC-2) was developed for use in the Phase 2 IV&V to assess mitigation of ANC, REP, and PRO. The ABC-1 and ABC- 2 are referred to, collectively, as the ABC. The ABC consists of two broad classes of items: recognition and discrimination (RD) and behavioral elicitation (BE). The ABC-1 and the ABC-2 each include one RD scale and three BE scales, one for each bias. The RD scales consist primarily of multiple-choice items and are intended to assess declarative knowledge of the biases. The BE scales consist of a variety of innovative tasks intended to evaluate test-takers’ procedural knowledge regarding how to avoid committing the targeted biases in judgment and decision-making tasks specifically designed to give test-takers opportunities to commit those biases. To the extent possible, the tasks were grounded in, and adapted to varying degrees from, extant paradigms relevant to each of the six biases. The BE tasks are complex scenario-based assessments that require test-takers to make decisions and solve problems presented in text, video, and/or voice-over audio formats, typically under conditions of uncertainty, time pressure, and/or rewards (and penalties). Each version of the ABC is administered online using a customized test delivery platform developed by the MITRE Corporation and takes approximately 45 to 60 minutes to complete. The ABC-1 and ABC-2 both include three equated test forms. This facilitated evaluation of bias mitigation training outcomes by making it possible to compare test-takers’ performance on one form post-training with their pre-training performance on an alternate, equated ABC test form. The purpose of this Executive Summary is to provide a relatively brief synopsis of the complete, and rather extensive, ABC technical report. In the sections that follow, we summarize the (1) test development process, (2) research conducted during the development and validity evaluation of the ABC, (3) validity argument for the ABC, and (4) suggestions for subsequent research based on the project described in this technical report. Test Development Development of the ABC-1 and ABC-2 included the following steps: • Construct Identification. This process included: v o reviewing literature relevant to the Sirius project biases, including bias description and elicitation, bias mitigation techniques, individual differences in bias susceptibility, correlates of the biases, and illustrations of how the biases relate to the work of intelligence analysts; o generating operational definitions of the bias constructs, including their facets, to help ensure the most complete possible coverage of each bias construct; and o periodically consulting with a technical advisory group (TAG), subject matter experts (SMEs), and the IV&V team (which included representatives from IARPA, Johns Hopkins University Applied Physics Lab [JHUAPL], and MITRE) in order to clarify the content and boundaries of each bias construct. • Development of Item Prototypes. We developed BE and RD item prototypes using the following sources: (1) operational definitions of each bias or bias facet; (2) our review of the literature; (2) case studies of intelligence analysis; (3) critical incidents adapted from in-depth interviews with several IC SMEs; and (4) input from the TAG and IV&V team. • Cognitive Laboratory Pilot Research. We conducted two rounds of cognitive lab studies of BE item prototypes with several dozen ETS employees to identify task elements that test-takers found to be unclear, distracting, or too demanding. In addition, we examined both concurrent think–aloud protocols and retrospective descriptions of test responses in order to enhance understanding of conscious decision making and problem solving strategies adopted by test-takers to improve the ABC. • Item Generation. Following the development and evaluation of item prototypes, we created a pool of over 600 BE and RD items during both phases of the project. The item pool included several dozen scripted scenarios that were filmed and edited by a professional video production company in Louisville, KY, and at the ETS Princeton, NJ, campus with local professional actors and ETS employees. • Item Review. Items were reviewed by assessment development specialists and SMEs, including the IV&V team and TAG, with respect to criteria such as (a) clarity, (b) lack of ambiguity and vagueness, (c) ensuring that the items do not assume knowledge specific to the intelligence analyst job, and (d) sensitivity to EEOC protected class (e.g., based on gender, race/ethnicity, age) bias and fairness issues. For items that had content specific to intelligence analysis work, additional reviews were performed by Intelligence Community SMEs at MITRE. • Pilot Testing. Because the constructs targeted