Rethinking Social Inquiry ................. 17811$ $$FM 06-28-10 14:26:41 PS PAGE i ................. 17811$ $$FM 06-28-10 14:26:41 PS PAGE ii Rethinking Social Inquiry Diverse Tools, Shared Standards Second Edition Edited by Henry E. Brady and David Collier ROWMAN & LITTLEFIELD PUBLISHERS, INC. Lanham • Boulder • New York • Toronto • Plymouth, UK ................. 17811$ $$FM 06-28-10 14:26:42 PS PAGE iii Published by Rowman & Littlefield Publishers, Inc. A wholly owned subsidiary of The Rowman & Littlefield Publishing Group, Inc. 4501 Forbes Boulevard, Suite 200, Lanham, Maryland 20706 http://www.rowmanlittlefield.com Estover Road, Plymouth PL6 7PY, United Kingdom Copyright ᭧ 2010 by Rowman & Littlefield Publishers, Inc. Chapters 6–9 from the previous edition are available as downloadable files from the Rowman & Littlefield website. For more information about these chapters and to obtain the required username and password visit http://www.rowmanlittlefield .com/RL/books/RSI2e/ or e-mail [email protected]. All rights reserved. No part of this book may be reproduced in any form or by any electronic or mechanical means, including information storage and retrieval systems, without written permission from the publisher, except by a reviewer who may quote passages in a review. British Library Cataloguing in Publication Information Available Library of Congress Cataloging-in-Publication Data Rethinking social inquiry : diverse tools, shared standards / edited by Henry E. Brady and David Collier.—2nd ed. p. cm. Includes bibliographical references and index. ISBN 978-1-4422-0343-3 (cloth : alk. paper) ISBN 978-1-4422-0344-0 (pbk. : alk. paper) 1. Social sciences—Research. 2. Social sciences—Methodology. I. Brady, Henry E. II. Collier, David. H62.R4646 2010 300.72—dc22 2010022477 ⅜ϱ ீThe paper used in this publication meets the minimum requirements of American National Standard for Information Sciences—Permanence of Paper for Printed Library Materials, ANSI/NISO Z39.48-1992. Printed in the United States of America ................. 17811$ $$FM 06-28-10 14:26:42 PS PAGE iv To the memory of David A. Freedman Brilliant statistician, Guardian against precarious statistical models, Champion of joining quantitative and qualitative analysis, Friend of remarkable wit and generosity And indeed, social science methodology is now catching up with him ................. 17811$ $$FM 06-28-10 14:26:42 PS PAGE v ................. 17811$ $$FM 06-28-10 14:26:43 PS PAGE vi Contents List of Figures and Tables xi Preface to the Second Edition xiii Preface to the First Edition xv Introduction to the Second Edition: A Sea Change in Political Methodology 1 David Collier, Henry E. Brady, and Jason Seawright Part I. A Debate on Methodology 11 A. Framing the Debate 13 1. Refocusing the Discussion of Methodology 15 Henry E. Brady, David Collier, and Jason Seawright 2. The Quest for Standards: King, Keohane, and Verba’s Designing Social Inquiry 33 David Collier, Jason Seawright, and Gerardo L. Munck B. Critiques of the Quantitative Template 65 3. Doing Good and Doing Better: How Far Does the Quantitative Template Get Us? 67 Henry E. Brady 4. Some Unfulfilled Promises of Quantitative Imperialism 83 Larry M. Bartels vii ................. 17811$ CNTS 06-28-10 14:26:45 PS PAGE vii viii Contents 5. How Inference in the Social (but Not the Physical) Sciences Neglects Theoretical Anomaly 89 Ronald Rogowski C. Linking the Quantitative and Qualitative Traditions 99 6. Bridging the Quantitative-Qualitative Divide 101 Sidney Tarrow 7. The Importance of Research Design 111 Gary King, Robert O. Keohane, and Sidney Verba D. Diverse Tools, Shared Standards 123 8. Critiques, Responses, and Trade-Offs: Drawing Together the Debate 135 David Collier, Henry E. Brady, and Jason Seawright 9. Sources of Leverage in Causal Inference: Toward an Alternative View of Methodology 161 David Collier, Henry E. Brady, and Jason Seawright Part II. Causal Inference: Old Dilemmas, New Tools 201 Introduction to Part II David Collier, Henry E. Brady, and Jason Seawright E. Qualitative Tools for Causal Inference 205 10. Process Tracing and Causal Inference 207 Andrew Bennett 11. On Types of Scientific Inquiry: The Role of Qualitative Reasoning 221 David A. Freedman 12. Data-Set Observations versus Causal-Process Observations: The 2000 U.S. Presidential Election 237 Henry E. Brady Addendum: Teaching Process Tracing 243 David Collier F. Quantitative Tools for Causal Inference 245 13. Regression-Based Inference: A Case Study in Failed Causal Assessment 247 Jason Seawright ................. 17811$ CNTS 06-28-10 14:26:45 PS PAGE viii Contents ix 14. Design-Based Inference: Beyond the Pitfalls of Regression Analysis? 273 Thad Dunning Glossary 313 Jason Seawright and David Collier Bibliography 361 Acknowledgment of Permission to Reprint Copyrighted Material 387 Subject Index 389 Name Index 397 About the Contributors 405 ................. 17811$ CNTS 06-28-10 14:26:45 PS PAGE ix ................. 17811$ CNTS 06-28-10 14:26:46 PS PAGE x Figures and Tables FIGURES 2.1 Steps in the Research Cycle: A Framework for Summarizing Designing Social Inquiry 50 8.1 Evaluating Necessary and/or Sufficient Causes 147 13.1 Mean Growth Rates by Level of Democracy 252 13.2 Stable Empirical Results Are Compatible with Three Alternative Causal Models 257 14.1 Plausibility of As-If Random Assignment 292 14.2 Credibility of Statistical Models 296 14.3 Substantive Relevance of Intervention 301 14.4 Typology of Natural Experiments 305 TABLES 2.1 Quantitative Tools Employed in Designing Social Inquiry 39 2.2 Anticipating the Debate on Designing Social Inquiry 58 6.1 Tools for Bridging the Qualitative-Quantitative Divide 104 9.1 Four Approaches to the Qualitative-Quantitative Distinction 179 9.2 Data-Set Observation versus Causal-Process Observation 185 9.3 Adding Different Forms of Data: Consequences for Causal Inference 192 10.1 Process Tracing: Four Tests for Causation 210 13.1 Economic Growth Regressed on Democracy and Lagged Income Level 253 xi ................. 17811$ FIGS 06-28-10 14:26:49 PS PAGE xi xii Figures and Tables 13.2 Economic Growth Regressed on Democracy, Lagged Income Level, and Primary Educational Attainment 255 13.3 Alternative Combinations of Control Variables 260 14.1 Examples of Natural Experiments, Including Regression Discontinuity (RD) and Instrumental Variable (IV) Designs 282 ................. 17811$ FIGS 06-28-10 14:26:49 PS PAGE xii Preface to the Second Edition Rethinking Social Inquiry seeks to redirect ongoing discussions of methodol- ogy in political and social science. This Preface presents our two goals in launching a second edition. The first goal (a central focus of Part I) is to sustain the debate with King, Keohane, and Verba’s (KKV)1 Designing Social Inquiry. Nine chapters from the first edition are included here to continue this exchange. Although published more than 15 years ago, KKV remains a fundamental point of reference in political science methodology and in controversies on methods—as we dis- cuss in the Introduction to the Second Edition. Through articulating the approach we call ‘‘mainstream quantitative methods,’’ KKV has wide impor- tance in the political science discipline—and, correspondingly, in graduate student training. While we admire aspects of the book’s contribution, our strong dissent from many of the arguments remains highly salient today. KKV has played a key role in narrowing attention to a particular set of quantitative tools, and the methodological horizon of political science has been shortened by the book’s continuing influence. Sustaining this debate in 2010 therefore remainsasnecessaryasitwaswhenourfirsteditionappearedin2004. The second goal is to open new avenues of discussion in methodology, both qualitative and quantitative. A number of chapters from the first edi- tion—in particular chapters 8 and 9—explore these wider themes. In addi- tion, a new set of chapters has been incorporated as Part II of the second edition. These chapters offer an innovative view of the crucial qualitative tools of process tracing and causal process observations, as well as an extended new discussion of the weaknesses and strengths of regression analysis and other quantitative tools. 1. To avoid personalizing the debate, we previously adopted the abbreviation DSI in referring to the book. However, the abbreviation KKV is now ubiquitous, and we have deferred to standard usage. In the present edition, DSI has been replaced by KKV. xiii ................. 17811$ PREF 06-28-10 14:26:51 PS PAGE xiii xiv Preface to the Second Edition A detailed overview of the new chapters is provided in the Introduction to Part II below. A central theme of these chapters is the importance of methodological pluralism and the value of multi-method research. Quali- tative analysis is strengthened when used in conjunction with quantitative research; and quantitative analysis, in turn, contributes more if it is built on a foundation of qualitative analysis and insight. Two distinctive features of the second edition must be underscored. The first is the online placement, on the Rowman & Littlefield website, of four chapters from the first edition that are not included here. The online chap- ters are part of the original debate with KKV, and they also extend the dis- cussion well beyond that debate.2 Thus, we are able to retain all the original chapters and accommodate the new chapters in Part II, with little change in the overall length of the printed book. These chapters are accessible by following the instructions on the copyright page of this volume. Second, with the goal of advancing the understanding of process tracing and improving the teaching of this method, the online resources include a set of exercises. These challenge readers to push further in examining the case study evidence provided in the chapters by Bennett, Freedman, and Brady. The exercises also focus on additional readings, including the Sher- lock Holmes story ‘‘The Adventure of Silver Blaze,’’ an excellent illustration of process tracing. We are grateful for the extensive help we have received in preparing the second edition.