Evaluating Structured Analytic Techniques
Total Page:16
File Type:pdf, Size:1020Kb
IMPROVING THE ANALYSIS OF FOREIGN AFFAIRS: EVALUATING STRUCTURED ANALYTIC TECHNIQUES By Stephen J. Coulthart B.A. in Political Science, State University of New York at Oswego, 2006 M.A. in International Relations and Diplomacy, Seton Hall University, 2010 M.P.A in Public Administration, Seton Hall University, 2010 Submitted to the Graduate Faculty of the Graduate School of Public and International Affairs in partial fulfillment of the requirements for the degree of PhD in Public and International Affairs University of Pittsburgh 2015 UNIVERSITY OF PITTSBURGH GRADUATE SCHOOL OF PUBLIC AND INTERNATIONAL AFFAIRS This dissertation was presented By Stephen Coulthart It was defended on June 4, 2015 and approved by Shawn Bird, PhD, Division Chief, U.S. Department of State-Bureau of Intelligence and Research Michael Kenney, PhD, Associate Professor, Graduate School of Public and International Affairs, University of Pittsburgh Phil Williams, PhD, Professor, Graduate School of Public and International Affairs, University of Pittsburgh Dissertation Chair: William N. Dunn, PhD, Professor, Graduate School of Public and International Affairs, University of Pittsburgh ii Copyright © by Stephen J. Coulthart 2015 iii IMPROVING ANALYSIS OF FOREIGN AFFAIRS: EVALUATING STRUCTURED ANALYTIC TECHNIQUES Stephen Coulthart, M.P.A./M.P.A University of Pittsburgh, 2015 Research suggests that foreign affairs analysis is weak—even the best analysts are accurate less than 35 percent of the time (Tetlock 2005). To compensate for analytic weaknesses, some have called for the use of structured analytic techniques, that is, formalized intelligence analysis methods. This imperative was enshrined in the Intelligence Reform and Terrorism Prevention Act (2004), which mandates that analysts use these techniques. This research investigates how the techniques have been applied in the U.S. intelligence community (IC) while making a modest attempt to evaluate 12 core techniques. The investigation of how the techniques are applied is based on semi-structured interviews with 5 intelligence experts and a survey of 80 analysts at an IC agency, along with follow-up interviews with 15 analysts. Interestingly, 1 in 3 analysts reported never using the techniques. Two factors were related to the use of the techniques: analytic training (p=0.001, Cramer's V=0.41) and the perception of their value (p=.049, Cramér's V= 0.23). There was not a statistically significant relation between the time pressure under which analysts work and their use of the techniques (p=0.74). Questions about the effectiveness of the techniques were answered in part by employing a “systematic review,” a novel methodology for synthesizing a large body of research. A random sample of more than 2,000 studies, suggests that there is moderate to iv strong evidence affirming the efficacy of using three techniques: Analysis of Competing Hypotheses, Brainstorming, and Devil’s Advocacy. There were three main findings: face- to-face collaboration decreases creativity, evidence weighting appears to be more important than seeking disconfirming evidence, and conflict tends to improve the quality of analysis. This research also employed an experiment with 21 graduate intelligence studies students, which confirmed the first two findings of the systematic review. The findings of the dissertation represent a contribution to “evidence-based intelligence analysis,” the systematic effort to develop a robust evidence-base linking the use of specific analytic techniques to the improvement of analysis in foreign affairs. Future research might build on the evidence-base presented here to improve intelligence analysis, one of the most important areas of judgment in foreign affairs. v TABLE OF CONTENTS 1.0 CHAPTER 1: ADDRESSING THE LIMITS OF FOREIGN AFFAIRS ANALYSIS ................................................................................................................................... 1 1.1 THE LIMITATIONS OF EXPERTISE IN FOREIGN AFFAIRS ANALYSIS .... 2 1.2 SPANNING THE DIVIDE: IMPROVING DECISION AND ANALYSIS IN FOREIGN AFFAIRS .......................................................................................................... 4 1.2.1 The Theory-Application Gap- Evaluating the Intelligence Reform Act 6 1.3 DEVELOPING A THEORY OF STRUCTURED ANALYTIC TECHNIQUES 10 1.3.1 A Theory of Structured Analytic Techniques ........................................ 13 1.4 ADOPTING EVIDENCE-BASED INTELLIGENCE ANALYSIS TO ANSWER THE RESEARCH QUESTIONS ..................................................................................... 17 1.4.1 Research Question 1: The Implementation of Structured Analytic Techniques ................................................................................................................ 18 1.4.2 Research Question 2: The Effectiveness of Structured Analytic Techniques ................................................................................................................ 20 1.5 AT THE FRONTIER OF FOREIGN AFFAIRS ANALYSIS .............................. 24 2.0 CHAPTER 2: ASSESSING THE QUALITY OF INTELLIGENCE ANALYSIS: A SYNTHESIS AND REFORMULATION .............................................................................. 26 2.1 WHAT IS ‘QUALITY’ INTELLIGENCE ANALYSIS AND WHY IT’S SO HARD TO PRODUCE? ................................................................................................... 28 vi 2.1.1 Defining Intelligence Analysis Quality .................................................. 29 2.1.2 The Diminishing Returns of Expert Judgment ...................................... 34 2.2 REFRAMING INTELLIGENCE ANALYSIS AS AN “INEXACT SCIENCE” 38 2.2.1 Intelligence Analysis as an “Inexact Science” ....................................... 39 2.2.2 A Justification for Structured Analytic Techniques: System 1 and 2 Thinking .................................................................................................................... 44 2.2.3 A New Theory of Structured Analytic Techniques .............................. 46 2.3 IMPLEMENTING STRUCTURED ANALYTIC TECHNIQUES IN THE IC . 51 2.3.1 The Answer is in the Head, not the Numbers ........................................ 51 2.3.2 New Era, New Analysis .............................................................................. 54 2.3.3 Intelligence Reform and Terrorism Prevention Act of 2004 and Large- Scale Implementation .............................................................................................. 57 2.3.4 Factors Affecting the Implementation of Structured Analytic Techniques ................................................................................................................ 59 2.4 SUMMARIZING THE ARGUMENT .................................................................... 62 3.0 CHAPTER 3: RESEARCH DESIGN AND METHODOLOGY ................................. 65 3.1 FIELD STUDY: IMPLEMENTING STRUCTURED ANALYTIC TECHNIQUES 66 3.1.1 Semi-structured Interviews with Intelligence Analysis Experts ........ 67 3.1.2 Survey and Interviews at the Bureau of Intelligence and Research .. 68 3.2 EVALUATING STRUCTURED ANALYTIC TECHNIQUES: A SYSTEMATIC REVIEW AND FIELD EXPERIMENT .......................................................................... 76 vii 3.2.1 A Systematic Review of Structured Analytic Techniques................... 77 3.2.2 An Experiment of ACH and Indicators ................................................... 84 3.3 CONCLUSION ........................................................................................................... 94 4.0 CHAPTER 4: WHO USES STRUCTURED ANALYTIC TECHNIQUES AND WHY?: A SURVEY IN THE INTELLIGENCE COMMUNITY ........................................ 96 4.1 FRAMING THE STUDY: TESTING THE VARIABLES AT STATE DEPARTMENT INR ........................................................................................................ 97 4.1.1 Study Hypotheses ........................................................................................ 98 4.1.2 State Department’s Bureau of Intelligence and Research ................... 99 4.1.3 Gauging the Use of Structured Analytic Techniques at INR ........... 101 4.2 STUDY HYPOTHESIS 1: ANALYST TRAINING .................................. 102 4.3 PERCEPTIONS OF THE TECHNIQUES, TIME PRESSURE AND DEMOGRAPHICS ......................................................................................................... 106 4.3.1 Perception of Structured Analytic Techniques .................................... 107 4.3.2 Time Pressure ............................................................................................. 110 4.3.3 Demographics: Age ................................................................................... 112 4.4 SUMMARY OF RESULTS AND CONCLUSIONS ................................ 115 5.0 CHAPTER 5: “WHAT WORKS?” A SYSTEMATIC REVIEW OF STRUCTURED ANALYTIC TECHNIQUES ..................................................................... 117 5.1 OVERVIEW: “WHAT WORKS?” ............................................................... 120 5.1.1 Selecting and Describing the Evaluative Studies ............................... 121 5.1.2 Assessing the Credibility of Evidence................................................... 123 viii 5.2 UNDER WHAT CONDITIONS ARE THE TECHNIQUES EFFECTIVE? 127 5.2.1 Analysis of Competing Hypotheses Studies ........................................ 128 5.2.2 Analysis of Competing Hypotheses Discussion ................................. 130 5.2.3 Brainstorming Studies .............................................................................