Nebraska Law Review Volume 81 | Issue 2 Article 5 2002 Evaluating Empirical Research Methods: Using Empirical Research in Law and Policy Jennifer K. Robbennolt University of Illinois at Urbana-Champaign, [email protected] Follow this and additional works at: https://digitalcommons.unl.edu/nlr Recommended Citation Jennifer K. Robbennolt, Evaluating Empirical Research Methods: Using Empirical Research in Law and Policy, 81 Neb. L. Rev. (2002) Available at: https://digitalcommons.unl.edu/nlr/vol81/iss2/5 This Article is brought to you for free and open access by the Law, College of at DigitalCommons@University of Nebraska - Lincoln. It has been accepted for inclusion in Nebraska Law Review by an authorized administrator of DigitalCommons@University of Nebraska - Lincoln. Evaluating Empirical Research Methods: Using Empirical Research in Law and Policy Jennifer K Robbennolt* TABLE OF CONTENTS I. Introduction .......................................... 777 II. Social Science Methods and Legal Questions ........... 779 A. Archival Research - The Nebraska Death Penalty Study ............................................. 780 B. Experimental Research - Death Qualification and Penalty Phase Instructions ........................ 785 III. Obstacles to Evaluating Empirical Methodology ........ 789 A. Biased Assimilation ............................... 790 B. Methodological Background ........................ 795 IV. Strategies for Evaluating Empirical Research .......... 797 A. Explore the Tensions .............................. 798 B. Research in Context ............................... 799 C. Methodological Training ........................... 801 V . Conclusion ............................................ 804 I. INTRODUCTION David Baldus and his colleagues have conducted an extensive and useful study of capital punishment decisionmaking in the state of Ne- braska.1 Their research is a fine example of how empirical research can be relevant to questions of law and policy. In determining whether to have a system of capital punishment and, if so, how to structure such a system, it is extremely useful to have data about vari- ous aspects of how the existing system operates. The same can be said for innumerable legal and policy issues ranging from various aspects © Copyright held by the NEBRASKA LAW REVIEW. * Associate Professor of Law & Senior Fellow at the Center for the Study of Dispute Resolution, University of Missouri School of Law. B.A. 1991, Willamette Univer- sity; J.D. 1996, University of Nebraska-Lincoln; Ph.D. (Psychology) 1998, Univer- sity of Nebraska-Lincoln. I appreciate the comments of David Baldus, Chris Guthrie, Monica Johnson, Grant Robbennolt, Bob Schopp, and Rod Uphoff, the research assistance of Jason Moore, Mitch Berry, and Josie Pottebaum, and the financial support of the University of Missouri Law School Foundation. 1. David C. Baldus et al., Arbitrarinessand Discriminationin the Administrationof the Death Penalty: A Legal and EmpiricalAnalysis of the Nebraska Experience (1973-1999), 81 NEB. L. REV. 486 (2002) [hereinafter Nebraska Study]. NEBRASKA LAW REVIEW [Vol. 81:777 of jury decisionmaking in both criminal and civil cases, 2 to eyewitness identifications, 3 to inheritance policy,4 to affirmative action, 5 and to contract law, 6 to mention just a few. Legislators routinely consider issues that are informed by empirical research and judges are called upon to act as gatekeepers who regulate the admissibility of scientific evidence in court. 7 In any discussion of the relevance and usefulness of empirical re- search in the law, there is, however, persistent tension between the methods of social science and the theory, goals, and settings of law and policy. As David Faigman notes, A more vexing aspect of data collection for the law comes from the basic incon- gruity between what scientists study and what the law is interested in know- ing.... The law invariably relies on applied science, which typically involves extrapolating from controlled laboratory tests that are highly artificial or gen- eralizing from noncontrolled field tests that contain many confounding variables. 8 Using the study of capital punishment in Nebraska as a starting point, the purpose of this paper is to explore more broadly the tensions and tradeoffs at the intersection of social science methodology and the law from the perspective of legal and policy decisionmakers who are called upon to utilize empirical research. Those who use social scien- tific research to inform the law must do more than distinguish "good" research from "bad," they must also face the inevitable question of how to appropriately use well-done, but inherently imperfect research, for legal and policy purposes. 2. See generally Edith Greene et al., Jurors and Juries: A Review of the Field, in TAKING PSYCHOLOGY AND LAW INTO THE TWENTY-FIRST CENTURY, (J. Ogloff ed., 2002); Neil Vidmar, The Performance of the American Civil Jury: An Empirical Perspective, 40 ARiz. L. REV. 849 (1998). 3. See generally Gary L. Wells et al., Eyewitness Identification Procedures: Recom- mendations for Lineups and Photospreads, 22 LAW & HUM. BEHAV. 603 (1998); Gary L. Wells, et al. From the Lab to the Police Station: A Successful Application of Eyewitness Research, 55 AM. PSYCHOL. 581 (2000). 4. See, e.g., Mary Louise Fellows et al., Committed Partners and Inheritance: An Empirical Study, 16 LAW & INEQ. 1 (1998); Monica K. Johnson & Jennifer K. Robbennolt, Using Social Science to Inform the Law of Intestacy: The Case of Un- married Committed Partners,22 LAW & HUM. BEHAV. 479 (1998); Rita J. Simon et al., Public Versus Statutory Choice of Heirs:A Study of Public Attitudes About Property Distribution at Death, 58 Soc. FORCES 1263 (1980). 5. See, e.g., David L. Chambers et al., Michigan's Minority Graduates in Practice: The River Runs Through Law School, 25 LAW & Soc. INQUIRY 395 (2000). 6. See, e.g., Russell Korobkin, Inertia and Preference in Contract Negotiation: The Psychological Power of Default Rules and Form Terms, 51 VAND. L. REV. 1583 (1998); Dennis P. Stolle & Andrew J. Slain, Standard Form Contracts and Con- tract Schemas: A PreliminaryInvestigation of the Effects of Exculpatory Clauses on Consumers' Propensity to Sue, 15 BEHAV. Sc. & L. 83 (1997). 7. See Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993). 8. DAVID L. FAIGMAN, LEGAL ALCHEMY: THE USE AND MISUSE OF SCIENCE IN THE LAW 53 (1999). 2002] EVALUATING EMPIRICAL RESEARCH METHODS 779 Toward this end, Part II uses empirical research on capital punish- ment decisionmaking to illustrate the tension between methodological concerns and legal theory or settings. The first example is drawn from the recent study of capital punishment in Nebraska conducted by Baldus and his colleagues and demonstrates a tension between meth- odological concerns and legal theory. The second example draws on experimental research that examines the effects of death qualifying jurors and the comprehensibility of capital punishment penalty phase jury instructions. This example highlights the tradeoff between real- ism and experimental control, a tension inherent in social science re- search that has implications for how empirical research is used in the law. Part III identifies several obstacles to careful consideration of these tensions and tradeoffs, and suggests that cognitive bias and a lack of familiarity with the scientific method can interfere with evalu- ations of the methodology of empirical research. Part IV suggests ways of overcoming these obstacles in order to better achieve careful consideration of the tensions and tradeoffs that exist at the intersec- tion of law and methodology. II. SOCIAL SCIENCE METHODS AND LEGAL QUESTIONS There are any number of methodological approaches to addressing research questions that have relevance for law and policy, each with its own advantages and limitations. In considering the relevance of empirical research for answering legal and policy questions, deci- sionmakers ought to be concerned about several different aspects of the research. For example, those evaluating empirical research should be concerned that the measures used to quantify legal concepts are sufficient to capture the construct at issue (construct validity),9 the degree to which the research design allows inferences to be drawn about causal relationships between variables (internal validity),10 and the degree to which the research findings can be generalized to per- sons, times, and settings beyond those in which the research was con- ducted (external validity).ll The following sections illustrate some of the tensions between these methodological concerns and legal con- cepts or settings. 9. THOMAS D. COOK & DONALD T. CAMPBELL, QUASI-EXPERIMENTATION: DESIGN AND ANALYSIS FOR FIELD SErrINGs 39, 59-64 (1979); see also Lee Epstein & Gary King, The Rules of Inference, 69 U. CHI. L. REV. 1, 80-81 (2002) ("This means that our comparisons and, ultimately, our answers to research questions are only as valid as the measures we have developed. If those measures do not adequately mirror the concepts contained in our theories, the conclusions we draw will be faulty."). 10. CooK & CAMPBELL, supra note 9, at 38. 11. Id. at 39. NEBRASKA LAW REVIEW [Vol. 81:777 A. Archival Research - The Nebraska Death Penalty Study The study of capital punishment in Nebraska recently conducted by Baldus and his colleagues is an example of archival research in which the researchers measure variables present in actual cases and attempt to identify relationships between