American Political Science Review Vo1. 89, No. 2 Tune 1995 THE IMPORTANCE OF RESEARCH DESIGN IN POLITICAL SCIENCE GARY KING, ROBERT 0. KEOHANE, and SIDNEY VERBA Harvard University

eceiving five serious reviews in this syrnpo- improving research design to produce better data in sium is gratifymg and confirms our belief that the first place, which almost always improves infer- R research design should be a priority for our ences more than the necessarily after-the-fact statis- discipline. We are pleased that our five distinguished tical solutions. reviewers appear to agree with our unified approach This lack of focus on research design in social to the logic of inference in the social sciences, and science statistics is as surprising as it is disappointing, with our fundamental point: that good quantitative since some of the most historically important works and good qualitative research designs are based fun- in the more general field of statistics are devoted to damentally on the same logic of inference. The re- problems of research design (see, e.g., Fisher (1935) viewers also raised virtually no objections to the main The Design of Experiments). Experiments in the social practical contribution of our book--our many specific sciences are relatively uncommon, but we can still procedures for avoiding bias, getting the most out of have an enormous effect on the value of our aualita- qualitative data, and making reliable inferences. tive or quantitative information, even withoui statis- However, the reviews make clear that although our tical corrections, by improving the design of our book may be the latest word on research design in research. We hope our book will help move these political science, it is surely not the last. We are taxed fields toward studying innovations in research de- for failing to include important issues in our analysis sim." and for dealing inadequately with some of what we We culled much useful information from the social included. Before responding to the reviewers' most science statistics literatures and qualitative methods direct criticisms, let us explain what we emphasize in fields. But for our goal of explicating and unifying the Designing Social Inquiry and how it relates to some of logic of inference, both literatures had problems. the points raised by the reviewers. Social science statistics focuses too little on research design, and its language seems arcane if not impen- etrable. The numerous languages used to describe WHAT WE TRIED TO DO methods in qualitative research are diverse, inconsis- tent in jargon and methodological advice, and not Designing Social Inquiry grew out of our discussions always helpful to researchers. We agree with David while coteaching a graduate seminar on research Collier that asvects of our advice can be revhrased design, reflecting on job talks in our department, and into some of the languages used in the q;alitative reading the professional literature in our respective methods literature or that used by quantitative re- subfields. Although many of the students, job candi- searchers. We hope our unified logic and, as David dates, and authors were highly sophisticated qualita- Laitin puts it, our "common vocabulary" will help tive and quantitative data collectors, interviewers, foster communication about these imvortant issues soakers and pokers, theorists, philosophers, formal among all social scientists. But we bekeve that any modelers, and advanced statistical analysts, many coherent language could be used to convey the same nevertheless had trouble defining a research question ideas.---- - and designing the empirical research to answer it. We demonstrated that "the differences between The students proposed impossible fieldwork to an- the quantitative and qualitative traditions are only swer unanswerable questions. Even many active stylistic and are methodologically and substantively scholars had difficulty with the basic questions: What unimportant" (p. 4). Indeed, much of the best social do you want to find out? How are you going to find science research can combine quantitative and quali- it out? and, above all, How would you know if you tative data, precisely because there is no contradic- were right or wrong? tion between the fundamental processes of inference We found conventional statistical training to be involved in each. Sidney Tarrow asks whether we only marginally relevant to those with qualitative agree that "it is the combination of quantitative and data. We even found it inadequate for students with qualitative" approaches that we desire (p. 473). We projects amenable to quantitative analysis, since so- do. But to combine both types of data sources pro- cial science statistics texts do not frequently focus on ductively, researchers need to understand the funda- research design in observational settings. With a few mental logic of inference and the more specific rules important exceptions, the scholarly literatures in and procedures that follow from an explication of this quantitative political methodology and other social logic. science statistics fields treat existing data and their Social science, both quantitative and qualitative, problems as given. As a result, these literatures seeks to develop and evaluate theories. Our concern largely ignore research design and, instead, focus on is less with the development of theory than theory making valid inferences through statistical correc- evaluation-how to use the hard facts of empirical tions to data problems. This approach has led to some reality to form scientific opinions about the theories dramatic progress; but it slights the advantage of and generalizations that are the hoped for outcome of Review Symposium June 1995 our efforts. Our social scientist uses theory to gener- specify ways to increase the information in qualitative ate observable implications, then systematically applies studies that can be used to evaluate theories; we publicly known procedures to infer from evidence show how this can be accomplished without return- whether what the theory implied is correct. Some ing to the field for additional data collection. theories emerge from detailed observation, but they Throughout the book, we illustrate our propositions should be evaluated with new observations, prefera- not only with hypothetical examples but with refer- bly ones that had not been gathered when the theo- ence to some of the best contemporary research in ries were being formulated. Our logic of theory political science. evaluation stresses maximizing leverage-explaining This statement of our purposes and fundamental as much as possible with as little as possible. It also arguments should put some of the reviewers' com- stresses minimizing bias. Lastly, though it cannot plaints about omissions into context. Our book is eliminate uncertainty, it encourages researchers to about doing empirical research designed to evaluate report estimates of the uncertainty of their conclu- theories and learn about the world-to make infer- sions. ences-not about generating theories to evaluate. We Theory and empirical work, from this perspective, believe that researchers who understand how to cannot productively exist in isolation. We believe that evaluate a theory will generate better theories-the- it should become standard practice to demand clear ories that are not only more internally consistent but implications of theory and observations checking that also have more observable implications (are more those implications derived through a method that at risk of being wrong) and are more consistent with minimizes bias. We hope that Designing Social lnquiy prior evidence. If, as Laitin suggests, our singlemind- helps to "discipline political science" in this way, as edness in driving home this argument led us implic- David Laitin recommends; and we hope, along with itly to downgrade the importance of such matters as James Caporaso, that "improvements in measure- ment accuracy, theoretical specification, and research concept formation and theory creation in political should yield a smaller range of allowable outcomes science, this was not our intention. consistent with the predictions made" (p. 459). Designing Social Inquiry repeatedly emphasizes the Our book also contains much specific advice, some attributes of good theory. How else to avoid omitted of it new and some at least freshly stated. We explain variable bias, choose causal effects to estimate, or how to distinguish systematic from nonsystematic derive observable implications? We did not offer components of phenomena under study and focus much advice about what is often called the "irrational explicitly on trade-offs that may exist between the nature of discovery," and we leave it to individual goals of unbiasedness and efficiency (chap. 2). We researchers to decide what theories they feel are discuss causality in relation to counterfactual analysis worth evaluating. We do set forth some criteria for and what Paul Holland calls the "fundamental prob- choosing theories to evaluate-in terms of their im- lem of causal inference" and consider possible com- portance to social science and to the real world-but plications introduced by thinking about causal mech- our methodological advice about research design anisms and multiple causality (chap. 3). Our applies to any type of theory. We come neither to discussion of counterfactual reasoning is, we believe, praise nor to bury rational-choice theory, nor to make consistent with Donald Campbell's "quasi-experi- an argument in favor of deductive over inductive mental" emphasis; and we thank James Caporaso for theory. All we ask is that whatever theory is chosen clarifying this. ' be evaluated by the same standards of inference. We pay special attention in chapter 4 to issues of Ronald Rogowski's favorite physicist, Richard Feyn- what to observe: how to avoid confusion about what man, explains clearly how to evaluate a theory (which constitutes a "case" and, especially, how to avoid or he refers to as a "guess"): "If it disagrees with [the limit selection bias. We show that selection on values empirical evidence], it is wrong. In that simple state- of explanatory variables does not introduce bias but ment is the key to science. It does not make any that selection on values of dependent variables does difference how beautiful your guess is. It does not so; and we offer advice to researchers who cannot make any difference how smart you are, who made avoid selecting on dependent variables. the guess, or what his name is-if it disagrees with We go on in chapter 5 to show that while random [the empirical evidence] it is wrong. That is all there measurement error in dependent variables does not is to it" (1965, 156).' bias causal inferences (although it does reduce effi- One last point about our goal: we want to set a high ciency), measurement error in explanatory variables standard for research but not an impossible one. All biases results in predictable ways. We also develop interesting qualitative and quantitative research procedures for correcting these biases even when yields uncertain conclusions. We think that this fact measurement error is unavoidable. In that same ought not to be dispiriting to researchers but should chapter, we undertake a sustained analysis of en- rather caution us to be aware of this uncertainty, dogeneity (i.e., when a designated "dependent vari- remind us to make the best use of data possible, and able" turns out to be causing what you thought was energize us to continue the struggle to improve our your "explanatory variable") and omitted variable stock of valid inferences about the political world. We bias, as well as how to control research situations so show that uncertain inferences are every bit as scien- as to mitigate these problems. In the final chapter, we tific as more certain ones so long as they are accom- American Political Science Review Vol. 89, No. 2 panied by honest statements of the degree of uncer- Then-surprisingly from the perspective of pluralist tainty accompaning each conclusion. theory-he found considerable evidence from many levels of analysis that the Netherlands was an espe- cially stable and peaceful democratic nation. These OUR ALLEGED ERRORS OF OMISSION descriptive inferences were valuable contributions to social science and important in and of themselves, The major theme of what may seem to be the most but Lijphart also wished to study the broader causal serious criticism offered above is stated forcefully by question. Ronald Rogowski. He fears that "devout attention" In isolation, a single study of the Netherlands, to our criteria would "paralyze, rather than stimulate, conducted only at the level of the nation at one point scientific inquiry." One of Rogowski's arguments, in time, cannot produce a valid estimate of the causal echoed by Laitin, is that we are too obsessed with effect of cross-cutting cleavages on the degree of increasing the amount of information we can bring to social peace in a nation. But Lijphart was not working bear on a theory and therefore fail to understand the in isolation. As part of a community of scholars, he value of case studies. The other major argument, had the benefit of Truman and others having col- made by both Rogowski and Collier is that we are too lected many prior observations. By using this prior critical of the practice of selecting observations ac- work, Lijphart could and did make a valid inference. cording to values of the dependent variable and that Prior researchers had either focused only on coun- we would thereby denigrate major work that engages tries with the same value of the explanatory variable in this practice. We consider these arguments in turn. (many cross-cutting cleavages) or on the basis of values of the dependent variable (high social con- flict). Previous researchers therefore made invalid Science as a Collective Enterprise inferences. Lijphart measured social peace for the other value of the explanatory variable (few cross- Rogowski argues that we would reject several classic cutting cleavages) and, by using his data in combina- case studies in comparative politics. We think he tion with that which came before, made a valid misunderstands these studies and misses our distinc- inference. tion between a "single case" and a collection of Lijphart's classic study is consistent with our model observations. Consider two works that he mentions, of good research design. As he stressed repeatedly in The Politics of Accommodation, by Arend Lij phart (1968), his book, Lijphart was contributing to a large schol- and The Nazi Seizure of Power, by William Sheridan arly literature. As such, he was not trylng to estimate Allen (1965). Good research designs are rarely exe- a causal effect from a single observation; nor was he cuted by individual scholars isolated from prior re- selecting on his dependent variable. Harvesting rele- searchers. As we say in our book, "A single observa- vant information from others' data, although often tion can be useful for evaluating causal explanations overlooked, may often be the best way to obtain if it is part of a research program. If there are other relevant information. observations, perhaps gathered by other researchers, By ignoring the place of Lijphart's book in the against which it can be compared, it is no longer a literature to which it was contributing, Rogowski was single observation" (p. 211; see also secs. 1.2.1,4.4.4, unable to recognize the nature of its contribution. the latter devoted entirely to this point). Rogowski Rogowski's alternative explanation for the impor- may have overlooked these passages. If we did not tance of this book and the others he mentions-that emphasize the point suf€iciently, we are grateful for "(1) all of them tested, relied on, or proposed, clear the opportunity to stress it here. and precise theories; and (2) all focused on anomalies" (p. 469)-suggests one of many possible strategies for Lijphart: The Case Study that choosing topics to research; but it is of almost no help Broke the Pluralist Camel's Back with practical issues of research design or ascertain- ing whether a theory is right or wrong. Indeed, the What was once called pluralist theory by David Tru- only way to determine whether something is an man and others holds that divisions along religious anomaly in the first place is to follow a clear logic of and class lines make polities less able to resolve scientific inference and theory evaluation, such as political arguments via peaceful means through dem- that provided in Designing Social Inquiry. ocratic institutions. The specific causal hypothesis is that the existence of many cross-cutting cleavages Allen: Distinguishing History From Social Science increases the level of social peace and, thus of stable, legitimate democratic government. The Nazi Seizure of Power is an account of life in an In The Politics of Accommodation, Arend Lijphart ordinary German community during the Nazi seizure (1968) sought to estimate this causal effe~t.~In addi- of power. Allen is not a social scientist: In his book, tion to prior literature, he had evidence from only one he proposes no generalization, evaluates no theory, case, the Netherlands. He first found numerous and does not refer to the scholarly literatures on Nazi observable implications of his descriptive hypothesis Germany; rather, he zeroes in on the story of what that the Netherlands had deep class and religious happened in one small place at a crucial moment in cleavages, relatively few of which were cross-cutting. history, and he does so brilliantly. In our terms, he is Review Symposium June 1995 describing historical detail and occasionally also con- in the 1930s, creating the basis for postwar corporat- ducting very limited descriptive inference. We em- ism (chap. 4). phasize the importance of such work: "Particular Katzenstein seeks to test the first of these causal events such as the French Revolution or the Demo- claims by comparing economic openness in small and cratic Senate primary in Texas may be of intrinsic large states (1985, table 1, p. 86). To evaluate the interest: they pique our curiosity, and if they were second hypothesis, he compares cross-class compro- preconditions for subsequent events (such as the mise in six small European states characterized by Napoleonic Wars or Johnson's presidency) we may weak landed aristocracies and strong urban sectors, need to know about them to understand those later with the relative absence of such compromise in five events" (p. 36). large industrialized countries and Austria, which had In our view, social science must go further than different values on these explanatory variables. Much Allen. The social scientist must make descriptive or of his analysis follows the rules of scientific inference causal inferences, thus seeking explanation and gen- we discuss-selecting cases to vary the value of the eralization. Indeed, we think even Rogowski would explanatory variables, specifying the observable irn- not accept Allen's classic work of history as a disser- plications of theories, seeking to determine whether tation in political science. Allen's work is, however, the facts meet theoretical expectations. not irrelevant to the task of explanation and general- But Katzenstein fudges the issue of causal infer- ization that is of interest to us. In the hands of a good ence by disavowing claims to causal validity: "Anal- social scientist, who could place Allen's work within yses like this one cannot meet the exacting standards an intellectual tradition, it becomes a single case of a social science test that asks for a distinction study in the framework of many others. This, of between necessary and sufficient conditions, a course, suggests one traditional and important way weighting of the relative importance of variables, in which social scientists can increase the amount of and, if possible, a proof of causality" (p. 138). How- information they can bring to bear on a problem: read ever, estimating causal inferences does not require a the descriptive case study literature. "distinction between necessary and sufficient condi- tions, a weighting of the relative importance of vari- ables," or an absolute "proof" of anything. Katzen- The Perils of Avoiding Selection Bias stein thus unnecessarily avoids causal language and We agree with David Collier's observation that, if our explicit attention to the logic of inference which arguments concerning selection bias are sustained, results. As we explain in our book, "Avoiding causal then "a small improvement in methodological self- language when causality is the real subject of inves- awareness can yield a large improvement in scholar- tigation either renders the research irrelevant or per- ship" (p. 461). Indeed, because qualitative research- mits it to remain undisciplined by the rules of scien- ers generally have more control over the selection of tific inference" (p. 76). their observations than over most other features of Remaining inexplicit about causal inference makes their research designs, selection is an especially im- some of Katzenstein's clauns ambiguous or unsup- portant concern (a topic to which we devote most of ported. For example, his conclusion seems to argue our chap. 4).4 that small states' corporatist strategies are responsible Rogowski believes that we would criticize Peter for their postwar economic success. But because of Katzenstein's (1985) Small States in World Markets or the selection bias induced by his decision to study Robert Bates's (1981) Markets and States in Tropical only successful cases, Katzenstein cannot rule out an Afrrca as inadmissibly selecting on the dependent important alternative causal hypothesis-that any of variable. We address each book in turn. a variety of other factors accounts for this uniform pattern. For instance, the postwar international po- litical economy may have been benign for small, Katzenstein: Distinguishing Descriptive developed countries in Europe. If so, corporatist Inference from Causal Inference strategies may have been unrelated to the degree of success experienced by small European states. Peter Katzenstein's (1985) Small States in World Mar- In the absence of variation in the strategies of his kets makes some important descriptive inferences. states, valid causal inferences about their effects re- For example, Katzenstein shows that small European main elusive. Had Katzenstein been more attentive to states responded flexibly and effectively to the eco- the problems of causal inference that we discuss, he nomic challenges that they faced during the 40 years would have been able to claim causal validity in some after World War 11; and he distinguishes between limited instances, such as when he had variation in what he calls "liberal and social corporatism" as two his explanatory and dependent variables (as in the patterns of response. But many of Katzenstein's 1930s analysis). More importantly, he would also arguments also imply causal claims-that in Western have been able to improve his research design so that Europe "small size has facilitated economic openness valid causal inferences were also possible in many and democratic corporatism" (p. 80), and that in the other areas. small European states, weak landed aristocracies, Rogowski is not correct in inferring that we would relatively strong urban sectors, and strong links be- dismiss the significance of Small States in World Mar- tween country and city led to cross-class compromise kets. Its descriptions are rich and fascinating, it elab- American Political Science Review Vol. 89, No. 2 orates insightful concepts such as liberal and social reviews. We often suggest procedures that qualitative corporatism, and it provides some evidence for a few researchers can use to increase the amount of infor- causal inferences. It is a fine book, but we believe that mation they bring to bear on evaluating a theory. This more explicit attention to the logic of inference could is sometimes referred to as "increasing the number of have made it even better. observations." As all our reviewers recognize, we do not expect researchers to increase the number of full-blown case studies to conduct a large-n statistical Bates: How to Identify a Dependent Variable analysis: our point is not to make quantitative re- Rogowski claims that Robert Bates's purpose in Mar- searchers out of qualitative researchers. In fact, most hts and States was to explain economic failure in qualitative studies already contain a vast amount of tropical African states and that by choosing only information. Our point is that appropriately marshal- states with failed economies and low agricultural ing all the thick description and rich contextualization production, Bates biased his inferences. If agricul- in a typical qualitative study to evaluate a specific tural production were Bates's dependent variable, theory or hypothesis can produce a very powerful Rogowski would be correct, since (as we describe in research design. Our book demonstrates how to Designing Social lnquiy and as elaborated by Collier), design research in order to collect the most useful using-but not correcting for-this type of case selec- qualitative data and how to restructure it even after tion does bias inferences. However, low agricultural data collection is finished, to turn qualitative infor- production was, in fact, not Bates's dependent variable. mation into ways of evaluating a specific theory. We Bates's book makes plain his two dependent vari- explain how researchers can do this by collecting ables: (1) the variations in public policies promulgated more observations on their dependent variable, by by African states and (2) differences in the group relations observing the same variable in another context, or by between the farmer and the state in each country. observing another dependent variable that is an im- Both variables vary considerably across his cases. Bates plication of the same theory. We also show how one also proposed several explanatory variables, which he can design theories to produce more observable im- derived from his preliminary descriptive inferences. plications that then put the theory at risk of being These include (1) whether state marketing boards were wrong more often and easily. founded by the producers or by alliances between This brings us to Sidney Tarrow's suggestions for government and trading interests, (2) whether urban or using the comparative advantages of both qualitative rural interests dominated the first postcolonial govem- and quantitative researchers. Tarrow is interested ment, (3) the degree of governmental committrnent to specifically in how unsystematic and systematic vari- spending programs, (4) the availability of nonagri- ables and patterns interact, and seems to think that cultural sources for governmental funds, and (5) principles could be derived to determine what unsys- whether the crops produced were for food or export. tematic events to examine. We think that this is an These explanatory variables do vary, and they helped interesting question for any historically-sensitive account for the variations in public policy and state- work. Many unsystematic, nonrepeated events oc- farmer relations that Bates observed. cur, a few of which may alter the path of history in As such, Bates did not select his observations so significant ways; and it would be useful to have they had a constant value for his dependent variable. criteria to determine how these events interact with Moreover, he did not stop at the national level of systematic patterns. We expect that our discussions analysis, for which he had a small number of cases of scientific inference could help in identrfylng which and relatively little information. Instead, he offered apparently random, but critical, events to study in numerous observable implications of the effects of specific instances, and we are confident that our logic these explanatory variables at other levels of analyses of inference will help determine whether these infer- within each country. As with many qualitative stud- ences are correct; but Tarrow or others may be able to ies, Bates had a small number of cases but an im- use the insights from qualitative researchers to spec- mense amount of information. We believe one of the ify them more clearly. We would look forward to a reasons Bates's study is-and should be-so highly book or article that presented such criteria. regarded is that it is an excellent example of a Another major point made by Tarrow is that all qualitative study that conforms to the rules of scien- appropriate methods to study a question should be tific inference. In sum, Rogowski says that Bates had employed. We agree: a major theme of our book is an excellent book that we would reject. If the book that there is a single unified logic of inference. Hence were as Rogowski describes it, we very well might it is possible effectively to combine different methods. reject it. Since it is not--and indeed is a good example However, the issue of triangulation that Tarrow so of our logic of research design-we join Rogowski in effectively raises is not the use of different logics or applauding it.5 methods, as he argues, but the triangulation of di- verse data sources trained on the same problem. Tri- angulation involves data collected at different places, TRIANGULAR CONCLUSIONS sources, times, levels of analysis, or perspectives, data that might be quantitative, or might involve We conclude by emphasizing a point that is empha- intensive interviews or thick historical description. sized both in Designing Social Inquiy and in the The best method should be chosen for each data Review Symposium June 1995 source. But more data are better. Triangulation, then, Achen, Christopher H. 1986. The Statistical Analysis of Quasi- is another word for refering to the practice of increas- Experiments. Berkeley: University of California Press. Achen, Christopher H., and Duncan Snidal. 1989. "Rational ing the amount of information to bear on a theory or Deterrence Theory and Comparative Case Studies." World hypothesis, and that is what our book is about. Politics 41:143-69. Allen, William Sheridan. 1965. The Nazi Seizure of Power: The Experience of a Single German Town, 1930-1935. New York: Watts. Notes Almond, Gabriel. 1990. A Divided Discipline. Newbury Park, CA: Sage. The table of contents, preface, and chapter 1 of Designing Arendt, Hannah. 1958. The Origins of Totalitarianism. Cleve- Social Inquiry are available via Gopher from hdc-gopher.har- land, OH: World. vard.edu. Ayer, A. J. 1946. Language, Tmth, and Logic. 2d ed. London: 1. To clarify further, we note that the definition of an Gollancz. "experiment" is investigator control over the assignment of Bakhtin, M. M. 1986. Speech Genres and Other Late Essays. values of explanatory variables to subjects. Caporaso empha- Trans. Vern M. McGee. Austin: University of Texas Press. sized also the value of random assignment, which is desirable Bates, Robert H. 1981. Markets and States in Tropical Africa: The in some situations (but not in others, see pp. 124-8) and Political Basis of Agrarian Policies. Berkeley: University of sometimes achievable in experiments. (Random selection and California Press. a large number of units are also desirable and also necessary Bates, Robert H. 1983. Essays on the Political Economy of Rural for relatively automatic unbiased inferences, but experiment- Africa. New York: Cambridge University Press. ers are rarely able to accomplish either.) A "quasi4xperi- Bendix, Reinhard. 1963. "Concepts and Generalizations in ment" is an observational study with an exogenous explana- Comparative Sociological Studies." American Sociological Re- tory variable that the investigator does not control. Thus, it is view 28:532-39. not an experiment. Campbell's choice of the word "quasi- Berkson, Joseph. 1946. "Limitations of the Application of experiment" reflected his insight that observational studies Fourfold Table Analysis to Hospital Data." Biometries Bulle- follow the same logic of inference as experiments. Thus, we tin 2:47-53. obviously agree with Campbell's and Caporaso's emphases Blalock, Hubert M. 1964. Causal Inferences in Non-experimental and ideas and only pointed out that the word "quasi- Research. Chapel Hill: University of North Carolina Press. experiment" adds another word to our lexicon with no Blalock, Hubert M. 1984. Basic Dilemmas in the Social Sciences. additional content. Its a line idea, much of which we have Beverly Hills, CA: Sage. adopted; but it is an unnecessary category. Bunce, Valerie. 1981. Do New Leaders Make a Difference? Exclu- 2. Telling researchers to "choose better theories" is not sive Succession and Public Policy under Capitalism and Socialism. much different than telling them to choose the right answer: Princeton: Princeton University Press. it is correct but not helpful. Many believe that deriving rules Bourdieu, Peirre. 1984. Distinction: A Social Critique of the for theory creation is impossible (e.g., Popper, Feynman), but Judgement of Taste. Trans. Richard Nice. Cambridge: Har- we see no compelling justification for this absolutist claim. As vard University Press. David Laitin correctly emphasizes, "the development of for- Campbell, Donald T. 1969. "Reforms as Experiments." Amer- mal criteria for such an endeavor is consistent with the ican Psychologist 24409-29. authors' goals." Campbell, Donald T., and Julian C. Stanley. 1963. Experimen- 3. Lijphart also went to great lengths to clarify the precise tal and Quasi-Exuerimental Desims-. for Research. Boston: theory he was investigating, because it was widely recognized ~ou~htonMifflin. that the concept of pluralism was often used in conflicting Collier.~ ~ -- David. 1993. "The Comparative Method." In Political ways, none clear or concrete enough to be called a theory. Science: The State of the ~iscipkne11, ed. Ada W. Finifter. Ronald Rogowski's description of pluralism as a "powerful, Washington: American Political Science Association. deductive, internally consistent theory" (p. 10) is surely the Cook, Thomas D., and Donald T. Campbell. 1979. Quasi- first time it has received such accolades. Experimentation. Chicago: Rand-McNally. 4. Selection problems are easily misunderstood. For exam- Durkheim, Emile. 1938. The Rules of Sociological Method. Trans. ple, Caporaso claims that "if selection biases operate indepen- Sarah A. Solovay and John H. Mueller. New York: Free dently of one's hypothesized causal variable, it is a threat to Press. internal validity; if these same selection factors interact with Eckstein, Harry. 1975. "Case Study and Theory in Political the causal variable, it is a threat to external validity" (p. 9). To Science." In Handbook of Political Science, vol. 7, Strategies of see that this claim is false, note, as Collier reemphasizes, that Inquiry, ed. Fred I. Greenstein and Nelson Polsby. Reading, Caporaso's "selection factors" can also be seen as an omitted MA: Addison-Wesley. variable. But omitted variables cannot cause bias if they are Fearon, James D. 1990. "Counterfactuals and Hypothesis independent of your key causal variable. Thus, although the Testing in Political Science." World Politics 43:169-95. distinction between internal and external validity is often Feynman, Richard Phillips. 1965. The Character of Physical Law. useful, it is not relevant to selection bias in the way Caporaso Cambridge: MIT Press. describes. Fisher, Sir Ronald Aylmer. 1935. The Design of Experiments. 5. Subsequently, Bates pursued the same research pro- Edinburgh: Oliver & Boyd. gram. For example, in Essays on the Political Economy of Rural Foucault, Michel. 1972. The Archaeology of Knowledge. Trans. Africa he evaluated his thesis for two additional areas-- A. M. Sheridan Smith. New York: Pantheon Books. colonial Ghana and Kenya (1983, chap. 3). So Bates does Geddes, Barbara. 1990. "How the Cases You Choose Affect exactly what we recommend: having developed his theory in the Answers You Get: Selection Bias in Comparative Poli- one domain, he extracts its observable implications and tics." In Political Analysis, vol. 2, ed. James A. Stimson. Ann moves to other domains to see whether he observes what the Arbor: University of Michigan Press. theory would lead him to expect. Geertz, Clifford. 1973. "Thick Description: Toward an Inter- pretative Theory of Culture." In his Interpretation of Cultures. New York: Basic Books. George, Alexander, and Timothy J. McKeown. 1985. "Case Symposium References Studies and Theories of Organizational Decision Making." Advances in Information Processes in Organizations 221-58. Achen, Christopher H. 1982. Interpreting and Using Regression Gourevitch, Peter Alexis. 1978. "The International System Analysis. University Paper series on Quantitative Applica- and Regime Formation: A Critical Review of Anderson and tions in the Social Sciences, no. 29. Beverly Hills: Sage. Wallerstein." Comparative Politics 10:419-38. American Political Science Review Vol. 89, No. 2

Griffin, Larry J. 1992. "Temporality, Events, and Explanation International Encyclopedia of the Social Sciences, vol. 15; ed. in Historical Sociology: An Introduction." Sociological Meth- David L. Sills. New York: Macmillan. ods and Research 20:40>27. Perrot, Michelle. 1986. "On the Formation of the French Heberle, Rudolf. 1970. From Democracy to Nazism: A Regional Working Class." In Working Class Formation, ed. Ira Katznel- Case Study on Political Parties in Germany. New York: Grosset son and Aristide Zolberg. Princeton: Princeton University & Dunlap. Press. Heberle, Rudolf. 1963. Landbevolkerung und Nationalsozialis- Porter, Michael E. 1990. The Competitive Advantage of Nations. mus: Eine soziologische Untersuchung der politischen Willensbil- New York: Free Press. dung in Schleswig-Holstein 1918 bis 1932. Rev. ed. Stuttgart: Przeworski, Adam, and Fernando Limongi. 1992. "Selection, Deutsche VerlagsAnstalt. Counterfactuals, and Comparisons." University of Chi- Hempel, Carl, G. 1966. Philosophy of Natural Science. Engle- cago. Typescript. wood Cliffs, NJ: Prentice-Hall. Przeworski, Adam, and Fernando Limongi. 1993. "Political Jewis, Robert. 1989. "Rational Deterrence: Theory and Evi- Regimes and Economic Growth." Journal of Economic Per- dence." World Politics 41:18>207. spectives F51-69. Katzenstein, Peter J. 1985. Small States in World Markets: Przeworski, Adam, and Henry Teune. 1970. The Logic of Industrial Policy in Europe. Ithaca, NY: Cornell University Comparative Social Inquiry. New York: Wiley. Press. Putnam, Robert D. 1993. Making Democracy Work: Civic Tradi- Kendall, Maurice G., and Wiiam R. Buckland. 1960. A tions in Modern Italy. Princeton: Princeton University Press. Dictionary of Statistical Terms. 2d ed. New York: Hafner. Ragin, Charles C. 1987. The Comparative Method: Moving Be- King, Gary. 1989. Unifying Political Methodology: The Likelihood Theory of Statistical Inference. Cambridge: Cambridge Univer- yond Qualitative and Quantitative Strategies. Berkeley: Univer- sity Press. sity of California Press. King, Gary, Robert 0. Keohane, and Sidney Verba. 1994. Russell, Bertrand. 1969. The Autobiography of Bertrand Russell, Designing Social Inquiry: Scientific Inference in Qualitative Re- 1914-1944. New York: Bantam Books. search. Princeton: Princeton University Press. Sartori, Giovanni. 1970. "Concept Misformation in Compar- Kohli, Atul. 1987. The State and Poverty in India. New York: ative Politics." American Political Science Review 64:103>53. Cambridge University Press. Sartori, Giovanni. 1976. Parties and Party Systems: A Framework Kornhauser, William. 1959. The Politics of Mass Society. New for Analysis. Cambridge: Cambridge University Press. York: Free Press of Glencoe. Scuibba, Roberto, and Rossana Schiubba Pace. 1976. Le comu- Kriesi, Jan. 1994. New Social Movements in Western Europe. nita di base in Italia. 2 vols. Rome: Coines. Minneapolis: University of Minnesota Press. Skocpol, Theda. 1979. States and Social Revolutions: A Compar- Kubik, Jan. 1994. The Power of Symbols against the Symbols of ative Analysis of France, Russia, and China. New York: Cam- Power: The Rise of Solidarity and the Fall of State Socialism in bridge University Press. Poland. University Park: Pennsylvania State University. Skocpol, Theda, and Margaret Somers. 1980. "The Uses of Kuhn, Thomas. 1962. The Structure of Scientific Revolutions. Comparative History in Macrosocial Inquiry." Comparative Chicago: University of Chicago Press. Studies in Society and History 22174-97. Laba, Roman. 1991. The Roots of Solidarity: A Political Sociology Smelser, Neil J. 1976. Comparative Methods in the Social Sciences. of Poland Working-class Democratization. Princeton: Princeton Englewood Cliffs, NJ: Prentice-Hall. University Press. Stinchcombe, Arthur L. 1968. Constructing Social Theories. New Laitin, David. 1994. "The Tower of Babel as a Coordination York: Harcourt, Brace & World. Game." American Political Science Review 88:622-34. Stolzenberg, Ross M., and Daniel A. Relles. 1990. "Theory Lave, Charles, and James March. 1975. An Introduction to Testing in a World of Constrained Research Design: The Models in the Social Sciences. New York: Harper & Row. Significance of Heclunan's Censored Sampling Bias Correc- Lederer, Emil. 1940. State of the Masses. New York: Norton. tion for Nonexperimental Research." Sociological Methods Lijphart, Arend. 1971. "Comparative Politics and the Com- and Research 18:395-415. parative Method." American Political Science Review 65:682- Tarrow, Sidney. 1988a. "Old Movements in New Cycles of 93. Protest: The Career of an Italian Religious Community." In Lijphart, Arend. 1975. The Politics of Accommodation: Pluralism From Structure to Action, ed. B. Klandermans, et al. Interna- and Democracy in the Netherlands. Berkeley: University of tional Social Movement Research Series, no l. Greenwich, California Press. CT: JAI Lipton, Michael. 1976. Why Poor People Stay Poor: Urban Bias in Tarrow, Sidney. 1988b. Democracy and Disorder: Protest and World Development. Cambridge: Harvard University Press. Politics in Italy, 1965-1975. New York: Oxford University McAdam, Doug. 1981. Freedom Summer. New York: Oxford Press. University Press. Tarrow, Sidney. 1994. Power in Movement: Collective Action, Martin, Lisa L. 1992. Coercive Cooperation: Explaining Multilat- Social Movements, and Politics. New York: Cambridge Uni- eral Economic Sanctions. Princeton: Princeton University versity Press. Press. Tilly, Charles. 1990. Coercion, Capital, and European States, Martin, Lisa L., and Kathryn Sikkink. 1993. "U.S. Policy and 990-1990 A.D. Cambridge, MA: Blackwell. Human Rights in Argentina and Guatemala, 1973-1980." In Tilly, Charles. 1993. European Revolutions, 1492-1992. Oxford: Double-edged Diplomacy ed. Peter B. Evans, Harold K. Jacob- Blackwell. son, and Robert D. Putnam. Berkeley: University of Cali- Tilly, Charles. 1994. "State and Nationalism in Europe, 1492- fornia Press. 1992." Theory and Society 2313146. Meehl, Paul E. 1967. "Theory-Testing in Psychology and Truman, David Bicknell. 1951. The Governmental Process: Polit- Physics: A Methodological Paradox." Philosophy of Science, ical Interest and Public Opinion. New York: Knopf. June, pp. 103-15. Walker, Henry A., and Bernard P. Cohen. 1985. "Scope Mill, John Stuart. 1974. "Of the Four Methods of Experimental Statements: Imperatives for Evaluating Theory." American Inquiry." In his A System of Logic. Toronto: University of Sociological Review 50:288-301. Toronto Press. Wallerstein, Immanuel. 1974. The Modern World System. Vol. Moore, Barrington, Jr. 1967. Social Origins of Dictatorship and 1. New York: Academic. Democracy. Boston: Beacon. Weber, Marianne. 1988. Max Weber: A Biography. Trans. Harry Moses, Lincoln E. 1968. "Truncation and Censorship." In Zohn. New Brunswick: Transaction.