WHEN IS ANALYSIS SUFFICIENT? A STUDY OF HOW PROFESSIONAL INTELLIGENCE ANALYSTS JUDGE RIGOR A Thesis Presented in Partial Fulfillment of the Requirements for the Degree Master of Science in the Graduate School of The Ohio State University By Daniel J. Zelik, B.S. * * * * * The Ohio State University 2007 Master’s Examination Committee: Approved by: Dr. David D. Woods, Advisor _________________________________ Dr. Emily S. Patterson Advisor Graduate Program in Industrial and Systems Engineering Copyright © by Daniel J. Zelik 2007 ABSTRACT The proliferation of data accessibility has exacerbated the risk of shallowness in information analysis, making it increasingly difficult to tell when analysis is sufficient for making decisions or changing plans, even as it becomes increasingly easy to find seemingly relevant data. In addressing the risk of shallow analysis, the concept of rigor emerges as an approach for coping with this fundamental uncertainty—motivating the need to better define and understand analytical rigor. The concept of rigor is explored in this thesis through a study that asks how professional analysts decide when there is sufficient rigor in an analytic process. Nine professional intelligence analysts participated in a scenario walkthrough in which they critiqued the analysis processes of two junior analysts—one representing a high- rigor analysis process and the other a low-rigor process. In the study, participants assumed the role of analyst supervisor, deciding if these analyses were of sufficient rigor to send to a decision maker—a fundamental judgment task characterized as the Supervisor's Dilemma. This study design validated and refined the Elicitation by Critiquing methodology, also developing the Liquified Natural Gas Scenario, based on security issues that challenge safety analyses, as a cognitive case for exploring themes in information analysis. This research identified three general findings on rigor in information analysis. First, it found that process insight influenced judgments of rigor. Second, it found that while similar cues were used in forming assessments of rigor, the way in which those cues were interpreted as indicating rigor tended to be more varied. Third, the results of the study suggest a revised definition of analytical rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation. This expanded understanding of rigor serves as an analytic broadening check to be leveraged against the risk of shallow analysis. ii To Allison & Paxton iii ACKNOWLEDGMENTS I thank my committee members, Advisor Dr. David D. Woods and Dr. Emily S. Patterson, for their guidance in preparing this thesis. I also thank the sponsor intelligence organization, as well as the individual professional analysts, both participatory and advisory, who were involved in the research effort. In particular, I thank Chris, Jeff, and Stoney for their inputs during the development of the study. My fellow C/S/E/L researchers, who both helped and hampered the realization of this work, are also acknowledged. Additionally, I would like to thank my family for all of their support throughout this latest academic journey: Paxton, who too was simultaneously a source of both inspiration and affable distraction; parents Howard and Regina; brothers Karl, Samuel, and particularly Jonpaul, who has accompanied me throughout much of the journey; family-in-law Daniel, Jolene, Dominic, Dawn, and Annie; and all others too numerous to list. I also thank Estelle, Charles, and their families for the quarters and comestibles they provided to me throughout the research process. I would especially like to thank Allison, who has aided me in myriad ways during the preparation of this thesis. In particular, I thank you for all of the love and patience you afford me throughout the pursuit of my dreams. iv VITA May 4, 1982.................................................... Born in Pittsburgh, Pennsylvania June 11, 2000.................................................. Diploma Upper St. Clair High School Pittsburgh, Pennsylvania May 7, 2005.................................................... Bachelor of Science Industrial Engineering Iowa State University of Science and Technology Ames, Iowa Bachelor of Science Liberal Arts and Sciences, Major in Psychology Iowa State University of Science and Technology Ames, Iowa PUBLICATIONS Dorn, B., Zelik, D., Vepadharmalingam, H., Ghosh, M., & Adams, S. K. (2004). Designing a user interface for a PDA-based campus navigation device. In Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting. Santa Monica, CA: Human Factors and Ergonomics Society. Peacock, J. B., Zelik, D., & Schaffer, A. (2003). Computer aided habitability assessment. In Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting. Santa Monica, CA: Human Factors and Ergonomics Society. v FIELDS OF STUDY Major Field: Industrial and Systems Engineering Specialization: Cognitive Systems Engineering vi TABLE OF CONTENTS Abstract........................................................................................................................................... ii Dedication...................................................................................................................................... iii Acknowledgments......................................................................................................................... iv Vita................................................................................................................................................... v List of Tables................................................................................................................................... ix List of Figures.................................................................................................................................. x Introduction.................................................................................................................................... 1 Chapters CHAPTER 1 On the Understanding of Rigor........................................................................... 8 Defining Rigor....................................................................................................... 8 A Brief History of Rigor........................................................................................ 9 CHAPTER 2 Researching Rigor in Information Analysis...................................................... 12 Intelligence Analysis as Information Analysis................................................... 12 Research Question.............................................................................................. 16 CHAPTER 3 A Study of the Professional Intelligence Analyst............................................ 17 Participants......................................................................................................... 17 LNG Scenario...................................................................................................... 19 vii Description............................................................................................. 19 Adaptation............................................................................................. 22 Selection................................................................................................. 23 Design.................................................................................................... 25 Supervisor's Dilemma......................................................................................... 29 Procedure............................................................................................................ 31 Data Analysis Methodology............................................................................... 36 Study Findings..................................................................................................... 39 Comparison of Briefing Reports Prompt.............................................. 39 Comparison of Process Documents Prompt......................................... 47 Rigor Assessment Prompt...................................................................... 50 Supervisor's Dilemma Prompt............................................................... 54 Participant Insights............................................................................................. 56 CHAPTER 4 Implications of the Findings on Rigor............................................................... 59 Methodological Contributions.......................................................................... 59 A Discussion of Findings..................................................................................... 61 Impact of Process Insight on Judgment of Analytic Rigor.................. 61 Cues for Inferring Rigor in Intelligence Analysis................................. 63 Informing the Design of Analysis Support Tools................................. 65 Extending the Findings Across Information Analysis Domains........... 66 Rigor in Intelligence Analysis............................................................................. 67 Limitations of Study Design............................................................................... 72 An Alternative Perspective on Rigor................................................................. 76 Future Work.......................................................................................................