The Rules of Inference The Harvard community has made this article openly available. Please share how this access benefits you. Your story matters Citation Epstein, Lee, and Gary King. 2002. "The Rules of Inference." University of Chicago Law Review 69:1–93. Citable link http://nrs.harvard.edu/urn-3:HUL.InstRepos:13457908 Terms of Use This article was downloaded from Harvard University’s DASH repository, and is made available under the terms and conditions applicable to Other Posted Material, as set forth at http:// nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of- use#LAA The Rules of Inference Lee Epstein & Gary King* Although the term “empirical research” has become commonplace in legal scholarship over the past two decades, law professors have, in fact, been conducting research that is empirical—i.e., learning about the world using quantitative data or qualitative information—for almost as long as they have been conducting research. For just as long, however, they appear to have been proceeding with little awareness of, much less compliance with, many of the rules of inference, and without paying heed to the key lessons of the revolution in empirical analysis that has been taking place over the last century in other disciplines. The tradition of including some articles devoted exclusively to the methodology of empirical analysis, so well represented in journals in traditional academic fields, is virtually nonexistent in the nation’s law reviews. As a result, readers learn considerably less accurate information about the empirical world than the studies’ stridently stated, but often overly confident, conclusions suggest. To remedy this situation—both for the producers and consumers of empirical work—we adapt the rules of inference used in the natural and social sciences to the special needs, theories, and data in legal scholarship, and explicate them with extensive illustrations from existing research. We also offer suggestions for how the infrastructure of teaching and research at law schools might be reorganized so that it can better support the creation of first-rate empirical research without compromising other important objectives. Introduction Just as research in the social and natural sciences addresses a wide array of theoretical, methodological, and substantive concerns, so too does scholarship produced by legal academics. The law reviews are replete with articles ranging from the normative to the descriptive, from narrow doctrinal analyses to large-n statistical investigations. Some studies advocate legal reform; others intend solely to add to the store of academic knowledge. And, yet, in all this variation in approach, in all this diversity in purpose, effect, and even intended audience, many, if not most, of these studies evince a common characteristic: a concern, however implicit, with empiricism—basing conclusions on observation or experimentation—and inference1—using facts we know to learn about facts we do not know. This may seem a puzzling, even odd, statement to legal academics. After all, in this community the word “empirical” has come to take on a particularly narrow meaning—one associated purely with “statistical techniques and analyses,” or quantitative data.2 But empirical research, as natural and * Lee Epstein ([email protected], http://www.artsci.wustl.edu/~PoliSci/Epstein/) is the Edward Mallinckrodt Distinguished University Professor of Political Science and Professor of Law, Washington University in St. Louis. Gary King ([email protected], http://GKing.Harvard.edu) is Professor of Government, Harvard University and Senior Science Advisor, Evidence and Information for Policy Cluster, World Health Organization. We thank the National Science Foundation (SBR-9729884, SBR-9753126, and IIS-9874747) and the National Institutes of Aging (P01 AG17625-01) for research support; Micah Altman, Anne Joseph, Susan Appleton, Stuart Banner, Shari Diamond, Bill Eskridge, Barry Friedman, Brian Glenn, Jack Goldsmith, Andrew Holbrook, Dan Keating, Bob Keohane, Jack Knight, Bert Kritzer, Ron Levin, Joan Nix, Eric Posner, Dan Schneider, Nancy Staudt, Sid Verba, and Adrian Vermeule for helpful comments and discussions; Dana Ellison, Andrew Holbrook, and Jeff Staton for research assistance; and Scott Norberg for his data. 1 See Gary King, Robert O. Keohane, & Sidney Verba, Designing Social Inquiry: Scientific Inference in Qualitative Research (1994), at 46. 2 See, for example, Michael Heise, "The Importance of Being Empirical," 26 Pepperdine L Rev 807 (1999): “[W]hen I speak of empirical legal scholarship I refer only to the subset of empirical legal scholarship that uses statistical techniques and analyses. By statistical techniques and analyses I mean studies that employ data (including systematically coded judicial opinions) that facilitate descriptions of or inferences to a larger sample or population as well as replication by other scholars;” Craig A. Nard, "Empirical Legal Scholarship: Reestablishing a Dialogue between the Academy and 2 UNIVERSITY OF CHICAGO LAW REVIEW [Vol. XXX:1 social scientists recognize, is far broader than these associations suggest. The word “empirical” denotes evidence about the world based on observation or experience. That evidence can be numerical (quantitative) or non-numerical (qualitative);3 neither is any more “empirical” than the other. What makes research empirical is that it is based on observations of the world—i.e., data, which is just a term for facts about the world. These facts may be historical or contemporary, based on legislation or case law, the results of interviews or surveys, or the outcomes of secondary archival research or primary data collection. Data can be precise or vague, relatively certain or very uncertain, directly observed or indirect proxies, and they can be anthropological, interpretive, sociological, economic, legal, political, biological, physical, or natural. As long as the facts have something to do with the world, they are data, and as long as research involves data that is observed or desired, it is empirical. Under this definition of “empirical,” assertions that “the amount of theoretical and doctrinal scholarship…overwhelms the amount of empirical scholarship,”4 ring hollow. For, as even the most casual reader of the nation’s law reviews must acknowledge, a large fraction of legal scholarship makes at least some claims about the world based on observation or experience.5 In fact, in terms of legal scholarship, it is only the purely normative or theoretical that is not empirical. But even many articles whose main purpose is normative often invoke empirical arguments to shore up their normative points—such as offering the positive empirical implications of adopting their preferred policy. Staudt’s essay on the Federal Income Tax Code is typical.6 A largely normative piece exploring conventional wisdom among tax scholars that housework should not or cannot be taxed, it ends with several unambiguously empirical claims: This reform, together with the market-oriented reform, would go far toward changing society's views of the value of productive activities carried out both in the home and in the market, and more importantly, it would represent a critical step in achieving greater economic security for women.7 The same holds for most doctrinal work, including the many studies that take issue with a particular line of court decisions or the logic used in them. Such is Sklansky’s investigation of the U.S. Supreme Court’s use of “new originalism”8 to resolve Fourth Amendment search and seizure the Profession," 30 Wake Forest L Rev 347 (1995): Empirical “scholarship [is] based on a detailed statistical study and analysis from which one could draw conclusions and formulate or reformulate policy.” Even Peter H. Schuck, "Why Don't Law Professors Do More Empirical Research?," 39 J Legal Educ 323 (1989), at 323, who recognizes that many forms of legal scholarship are, in fact, “empirical,” defines empirical scholarship as “primarily” “statistical studies.” 3 Many scholars divide empirical research into two types or styles: quantitative, which uses numbers and statistical methods, and qualitative, which does not rely on numbers but on historical materials, intensive interviews, and the like. See, for example, Earl Babbie, The Basics of Social Research (1999); William J. Dixon, "Research on Research Revisited: Another Half Decade of Quantitative Research on International Organizations," 31 Intl Org 65 (1977). This distinction may be relevant for some purposes but not for ours. That is because the rules governing empirical research, to which we devote most of this Article, apply with equal force to qualitative and quantitative work. See King, Keohane, & Verba, Designing Social Inquiry , supra note 1. 4 Heise, "The Importance of Being Empirical," supra note 2. 5 We conducted a systematic, but still preliminary, analysis of articles published over the last five years in six top law reviews (Chicago, Columbia Harvard, NYU, Stanford, and Yale). Virtually all the articles we have examined thus far include at least some empirical analyses or empirical hypotheses. Just under half used empirical evidence to evaluate their empirical hypotheses, and about half of these are solely qualitative. 6 Nancy C. Staudt, "Taxing Housework," 84 Georgetown L J 1571 (1996). 7 Id., at 1647. 8 According to David A. Sklansky, "The Fourth Amendment and Common Law," 100 Colum L Rev 1739 (2000), under “new originalism” the Court, assesses whether searches and seizures are