Expert Political Judgment : How Good Is It? How Can We Know? / Philip E
Total Page:16
File Type:pdf, Size:1020Kb
Expert Political Judgment This page intentionally left blank Expert Political Judgment how good is it? how can we know? Philip E. Tetlock princeton university press princeton and oxford Copyright © 2005 by Princeton University Press Published by Princeton University Press, 41 William Street, Princeton, New Jersey 08540 In the United Kingdom: Princeton University Press, 3 Market Place, Woodstock, Oxfordshire OX20 1SY All Rights Reserved Sixth printing, and first paperback printing, 2006 Paperback ISBN-13: 978-0-691-12871-9 Paperback ISBN-10: 0-691-12871-5 THE LIBRARY OF CONGRESS HAS CATALOGED THE CLOTH EDITION OF THIS BOOK AS FOLLOWS Tetlock, Philip. Expert political judgment : how good is it? how can we know? / Philip E. Tetlock. p. cm. Includes bibliographical references and index. ISBN-13: 978-0-691-12302-8 (alk. paper) ISBN-10: 0-691-12302-0 (alk. paper) 1. Political psychology. 2. Ideology. I. Title. JA74.5.T38 2005 320'.01'9—dc22 2004061694 British Library Cataloging-in-Publication Data is available This book has been composed in Sabon Printed on acid-free paper. ∞ pup.princeton.edu Printed in the United States of America 109876 To Jenny, Paul, and Barb This page intentionally left blank Contents Acknowledgments ix Preface xi Chapter 1 Quantifying the Unquantifiable 1 Chapter 2 The Ego-deflating Challenge of Radical Skepticism 25 Chapter 3 Knowing the Limits of One’s Knowledge: Foxes Have Better Calibration and Discrimination Scores than Hedgehogs 67 Chapter 4 Honoring Reputational Bets: Foxes Are Better Bayesians than Hedgehogs 121 Chapter 5 Contemplating Counterfactuals: Foxes Are More Willing than Hedgehogs to Entertain Self-subversive Scenarios 144 Chapter 6 The Hedgehogs Strike Back 164 Chapter 7 Are We Open-minded Enough to Acknowledge the Limits of Open-mindedness? 189 Chapter 8 Exploring the Limits on Objectivity and Accountability 216 Methodological Appendix 239 Technical Appendix Phillip Rescober and Philip E. Tetlock 273 Index 313 This page intentionally left blank Acknowledgments The jury is out on just how much bad judgment I showed by undertaking the good-judgment project. The project dates back to the year I gained tenure and lost my generic excuse for postponing projects that I knew were worth doing, worthier than anything I was doing back then, but also knew would take a long time to come to fruition. As I write twenty years later, the data are still trickling in and the project now threatens to outlast not just my career but me. Some long-term forecasts that experts offered will not come due until 2026. But most of the data are tabulated, some surprising patterns have emerged, and I see no reason for delaying the write-up into my retirement. Of course, a project of this duration requires the cooperation of many people over many years. My greatest collective debt is to the thoughtful professionals who patiently worked through the often tedious batteries of questions on what could have been, what is, and what might yet be. I told them at the outset that I did not intend to write a book that named names, or that, by exploiting hindsight bias, incited readers to glorify those who got it right or ridicule those who got it wrong. I promised strict confidentiality. The book that would emerge from this effort would be variable-centered, not person-centered. The focus would be on the links between how people think and what they get right or wrong, at various junctures, in a kaleidoscopically shifting world. I realize that the resulting cognitive portrait of expert political judgment is not altogether flattering, but I hope that research participants, even the “hedgehogs” among them, do not feel shabbily treated. I level no charges of judgmen- tal flaws that do not also apply to me. Another great debt is to the many colleagues who offered method- ological and theoretical advice that saved me from making an even big- ger fool of myself than I may have already done. Barbara Mellers, Paul Tetlock, and Phillip Rescober offered invaluable guidance on how to de- sign measures of forecasting skill that were sensitive to the variety of in- genious objections that forecasters raised when either high probability events failed to materialize or low probability events did materialize. And colleagues from several disciplines—including psychology, political science, economics, history, and the hybrid field of intelligence analysis— made suggestions at various junctures in this long journey that, in my opinion at least, improved the final product. I cannot remember the source of every insightful observation at every stage of this project, but x • Acknowledgments this list should include in roughly chronological order from 1984 to 2004: Peter Suedfeld, Aaron Wildavsky, Alexander George, George Bres- lauer, Danny Kahneman, Robyn Dawes, Terry Busch, Yuen Foong Khong, John Mercer, Lynn Eden, Amos Tversky, Ward Edwards, Ron Howard, Arie Kruglanski, James March, Joel Mokyr, Richard Herrmann, Geoffrey Parker, Gary Klein, Steve Rieber, Yaacov Vertzberger, Jim Goldgeier, Erika Henik, Rose McDermott, Cass Sunnstein, and Hal Arkes. In the final phases of this project, Paul Sniderman and Bob Jervis played a particu- larly critical role in helping to sharpen the central arguments of the book. Needless to say, though, none of the aforementioned bears responsibility for those errors of fact or interpretation that have persisted despite their perceptive advice. I also owe many thanks to the many former and current students who have worked, in one capacity or another, on various components of this project. They include Charles McGuire, Kristen Hannum, Karl Dake, Jane Bernzweig, Richard Boettger, Dan Newman, Randall Peterson, Penny Visser, Orie Kristel, Beth Elson, Aaron Belkin, Megan Berkowitz, Sara Hohenbrink, Jeannette Porubin, Meaghan Quinn, Patrick Quinn, Brooke Curtiss, Rachel Szteiter, Elaine Willey, and Jason Mills. I also greatly ap- preciate the staff support of Deborah Houy and Carol Chapman. Turning to institutional sponsors, this project would have been impos- sible but for generous financial and administrative support from the fol- lowing: the John D. and Catherine T. MacArthur Foundation Program in International Security, the Institute of Personality and Social Research of the University of California, Berkeley, the Institute on Global Conflict and Cooperation at the University of California, the Center for Advanced Study in the Behavioral Sciences in Palo Alto, the Mershon Center of the Ohio State University, the Social Science Research Council, the National Science Foundation, the United States Institute of Peace, the Burtt En- dowed Chair in the Psychology Department at the Ohio State University, and the Mitchell Endowed Chair at the Haas School of Business at the University of California, Berkeley. Finally, I thank my family—especially, Barb, Jenny, and Paul—for their infinite forbearance with my workaholic ways. Preface Autobiographical exercises that explore why the researcher opted to go forward with one project rather than another have often struck me as self-dramatizing. What matters is the evidence, not why one collected it. Up to now, therefore, I have hewed to the just-the-facts conventions of my profession: state your puzzle, your methods, and your answers, and exit the stage. I could follow that formula again. I have long been puzzled by why so many political disagreements—be they on national security or trade or welfare policy—are so intractable. I have long been annoyed by how rarely partisans admit error even in the face of massive evidence that things did not work out as they once confidently declared. And I have long wondered what we might learn if we approached these disputes in a more aggressively scientific spirit—if, instead of passively watching warring partisans score their own performance and duly pronounce themselves victorious, we presumed to take on the role of epistemologi- cal referees: soliciting testable predictions, scoring accuracy ourselves, and checking whether partisans change their minds when they get it wrong. I initially implemented my research plan tentatively, in a trial-and-error fashion in small-scale forecasting exercises on the Soviet Union in the mid-1980s, and then gradually more boldly, in larger-scale exercises around the world over the next decade. My instinct was to adopt and, when necessary, adapt methods of keeping score from my home discipline of psychology: correspondence measures of how close political observers come to making accurate predictions and logical-process measures of the degree to which observers play fair with evidence and live up to reputa- tional bets that require them to update their beliefs. Without giving too much away, I can say that surprises are in store. We shall discover that the best forecasters and timeliest belief updaters shared a self-deprecating style of thinking that spared them some of the big mistakes to which their more ideologically exuberant colleagues were prone. There is often a curiously inverse relationship between how well forecasters thought they were doing and how well they did. I could now exit the stage. But the project makes more sense when traced to its origins: my first close-up contact with the ingenuity and de- termination that political elites display in rendering their positions im- pregnable to evidence. The natural starting point is a 1984 meeting at xii • Preface the National Research Council, the administrative branch of the Na- tional Academy of Sciences. I was a freshly tenured professor from Berke- ley and the most junior (nonmember of the academy) member of the committee. The committee had been convened—as academic committees often are—to midwife the birth of another committee. This new commit- tee would have an ambitious—critics said pretentious—mandate: to ex- plore the contributions of the social sciences, not to the humdrum, usual-suspect problems of early childhood education or affirmative ac- tion, but to rescuing civilization itself from nuclear incineration.