The University of Chicago Law Review Volume 69 Winter 2002 Number 1 © 2002 by The University of Chicago The Rules of Inference Lee Epstein† Gary King†† Although the term “empirical research” has become commonplace in legal scholarship over the past two decades, law professors have in fact been conducting research that is empirical—that is, learning about the world using quantitative data or qualitative information—for almost as long as they have been conducting research. For just as long, however, they appear to have been pro- ceeding with little awareness of, much less compliance with, many of the rules of inference, and without paying heed to the key lessons of the revolution in empirical analysis that has been taking place over the last century in other disciplines. The tradition of including some articles devoted ex- clusively to the methodology of empirical analysis—so well represented in journals in traditional academic fields—is virtually nonexistent in the nation’s law reviews. As a result, readers learn con- siderably less accurate information about the empirical world than the studies’ stridently stated, but overly confident, conclusions suggest. To remedy this situation both for the producers and con- sumers of empirical work, this Article adapts the rules of inference used in the natural and social sciences to the special needs, theories, and data in legal scholarship, and explicates them with exten- sive illustrations from existing research. The Article also offers suggestions for how the infrastruc- ture of teaching and research at law schools might be reorganized so that it can better support the creation of first-rate empirical research without compromising other important objectives. † Edward Mallinckrodt Distinguished University Professor of Political Science and Pro- fessor of Law, Washington University in St. Louis; <http://artsci.wustl.edu/~polisci/epstein/>; [email protected]. †† Professor of Government, Harvard University and Senior Science Advisor, Evidence and Information for Policy Cluster, World Health Organization; <http://GKing.Harvard.edu>; [email protected]. We thank the National Science Foundation (SBR-9729884, SBR-9753126, and IIS-9874747) and the National Institutes of Aging (P01 AG17625-01) for research support; Micah Altman, Anne Joseph, Susan Appleton, Stuart Banner, Shari Diamond, Bill Eskridge, Malcolm Feeley, Barry Friedman, Brian Glenn, Jack Goldsmith, Andrew Holbrook, Dan Keating, Bob Keohane, Jack Knight, Bert Kritzer, Ron Levin, Joan Nix, Eric Posner, Dan Schneider, Nancy Staudt, Sid Verba, and Adrian Vermeule for helpful comments and discussions; Dana Ellison, Andrew Hol- brook, and Jeff Staton for research assistance; and Scott Norberg for his data. 1 2 The University of Chicago Law Review [69:1 INTRODUCTION Just as research in the social and natural sciences addresses a wide array of theoretical, methodological, and substantive concerns, so too does scholarship produced by legal academics. The law reviews are replete with articles ranging from the normative to the descriptive, from narrow doctrinal analyses to large-sample-size (large-n) statisti- cal investigations. Some studies advocate legal reform; others intend solely to add to the store of academic knowledge. And yet in all this variation in approach, in all this diversity in purpose, effect, and even intended audience, many, if not most, of these studies evince a com- mon characteristic: a concern, however implicit, with empiricism— basing conclusions on observation or experimentation—and infer- ence1—using facts we know to learn about facts we do not know. This may seem a puzzling, even odd, statement to legal academics. After all, in this community the word “empirical” has come to take on a particularly narrow meaning—one associated purely with “statistical techniques and analyses,” or quantitative data.2 But empirical re- search, as natural and social scientists recognize, is far broader than these associations suggest. The word “empirical” denotes evidence about the world based on observation or experience. That evidence can be numerical (quantitative) or nonnumerical (qualitative);3 nei- ther is any more “empirical” than the other. What makes research em- pirical is that it is based on observations of the world—in other words, 1 See Gary King, Robert O. Keohane, and Sidney Verba, Designing Social Inquiry: Scien- tific Inference in Qualitative Research 46 (Princeton 1994). 2 See, for example, Michael Heise, The Importance of Being Empirical, 26 Pepperdine L Rev 807, 810 (1999): [W]hen I speak of empirical legal scholarship I refer only to the subset of empirical legal scholarship that uses statistical techniques and analyses. By statistical techniques and analy- ses I mean studies that employ data (including systematically coded judicial opinions) that facilitate descriptions of or inferences to a larger sample or population as well as replication by other scholars. Craig A. Nard, Empirical Legal Scholarship: Reestablishing a Dialogue between the Academy and the Profession, 30 Wake Forest L Rev 347, 349 (1995) (“[Empirical] scholarship [is] based on a detailed statistical study and analysis from which one could draw conclusions and formulate or reformulate policy.”). Even Peter H. Schuck, Why Don't Law Professors Do More Empirical Re- search?, 39 J Legal Educ 323, 323 (1989), who recognizes that many forms of legal scholarship are, in fact, “empirical,” defines empirical scholarship as “primarily . statistical studies.” 3 Many scholars divide empirical research into two types or styles: quantitative, which uses numbers and statistical methods, and qualitative, which does not rely on numbers but on histori- cal materials, intensive interviews, and the like. See, for example, Earl Babbie, The Basics of So- cial Research 258 (Nelson Thomas Learning 1999); William J. Dixon, Research on Research Revis- ited: Another Half Decade of Quantitative and Field Research on International Organizations, 31 Intl Org 65, 79 (1977). This distinction may be relevant for some purposes but not for ours. That is because the rules governing empirical research, to which we devote most of this Article, apply with equal force to qualitative and quantitative work. See King, Keohane, and Verba, Designing Social Inquiry at 6 (cited in note 1). 2002] The Rules of Inference 3 data, which is just a term for facts about the world. These facts may be historical or contemporary, or based on legislation or case law, the re- sults of interviews or surveys, or the outcomes of secondary archival research or primary data collection. Data can be precise or vague, relatively certain or very uncertain, directly observed or indirect prox- ies, and they can be anthropological, interpretive, sociological, eco- nomic, legal, political, biological, physical, or natural. As long as the facts have something to do with the world, they are data, and as long as research involves data that is observed or desired, it is empirical. Under this definition of “empirical,” assertions that “the amount of theoretical and doctrinal scholarship . overwhelms the amount of empirical scholarship,”4 ring hollow. As even the most casual reader of the nation’s law reviews must acknowledge, a large fraction of legal scholarship makes at least some claims about the world based on ob- servation or experience.5 In fact, in terms of legal scholarship, it is only the purely norma- tive or theoretical that is not empirical. But even many articles whose main purpose is normative often invoke empirical arguments to shore up their normative points—such as offering the positive empirical im- plications of adopting their preferred policy. Staudt’s essay on the In- ternal Revenue Code is typical.6 A largely normative piece exploring conventional wisdom among tax scholars that housework should not or cannot be taxed, it ends with several unambiguously empirical claims: “This reform, together with the market-oriented reform, would go far toward changing society’s views of the value of productive ac- tivities carried out both in the home and in the market, and more im- portantly, it would represent a critical step in achieving greater eco- nomic security for women.”7 The same holds for most doctrinal work, including the many stud- ies that take issue with a particular line of court decisions or the logic used in them. Such is Sklansky’s investigation of the Supreme Court’s use of “new originalism”8 to resolve Fourth Amendment search and seizure cases. The author’s primary objective is to provide support for 4 Heise, 26 Pepperdine L Rev at 812 (cited in note 2). 5 We conducted a systematic, but still preliminary, analysis of articles published over the last five years in six top law reviews (Chicago, Columbia, Harvard, NYU, Stanford, and Yale). Virtually all the articles we have examined thus far include at least some empirical analyses or empirical hypotheses. Just under half used empirical evidence to evaluate their empirical hy- potheses, and about half of these were solely qualitative. 6 Nancy C. Staudt, Taxing Housework, 84 Georgetown L J 1571 (1996). 7 Id at 1647. 8 According to David A. Sklansky, The Fourth Amendment and Common Law, 100 Colum L Rev 1739, 1744 (2000), under “new originalism” the Court assesses whether searches and sei- zures are “unreasonable” in violation of the Fourth Amendment by determining whether eight- eenth-century common law allowed them. 4 The University of Chicago Law Review [69:1 the claim that this “new” approach “should be unattractive even to those generally