Broken Experimentation, Sham Evidence-Based Policy
Total Page:16
File Type:pdf, Size:1020Kb
YALE LAW & POLICY REVIEW Broken Experimentation, Sham Evidence-Based Policy Kristen Underhill* Evidence-based policy is gaining attention, and legislation and agency regulation have been no exception to calls for greater uptake of research evidence. Indeed, current interest in “moneyball for government” is part of a long history of efforts to promote research-based decisions in government, from the U.S. Census to cost-benefit analysis. But although evidence-based policy-making (EBPM) is often both feasible and desirable, there are reasons to be skeptical of the capacity of EBPM in governmental decision-making. EBPM is itself bounded by limits on rationality, the capacity of science, the objectivity of science, and the authority we wish to give technocrats. Where values are highly contentious, efforts to produce and use evidence in legislative and regulatory decisions may go so far awry that they become “sham” versions of evidence-based choices. In this Article, I name several of these sham practices, including the distortion of evidence, the engagement in “terminal” experimentation that destabilizes governmental programs, and “ratcheting” actions that defund entire priorities rather than individual approaches. Broken experimentation is also common, with evaluations of government programming and policies neglecting or misusing opportunities to provide rigorous evidence. I argue that the stakes of these misuses are high, resulting in losses of welfare and efficiency, erosion of scientific legitimacy, and infringement on the dignity of human subjects. But where genuine engagement with empirical evidence is possible, the game is surely worth the candle. This Article proposes novel ways to promote responsible uses of empirical evidence in both legislation and agency regulation, including evaluation mandates, pre-registration of evaluation protocols and transparency of research reports, ex ante decision commitments, and more attentive uses of judicial review. INTRODUCTION ................................................................................................................. 151 I. THE CASE FOR (BOUNDED) EVIDENCE-BASED POLICY-MAKING ............................ 157 A. Judging Decisions by Their Own Lights ............................................................. 158 B. The Limits of EBPM ..................................................................................................... 162 150 BROKEN EXPERIMENTATION, SHAM EVIDENCE-BASED POLICY C. Bounded EBPM Rising: From the Census to Moneyball ............................ 168 1. Origins and Cost-Benefit Analysis ..................................................... 168 2. The Modern Era ....................................................................................... 171 3. “Moneyball for Government” .............................................................. 179 II. BROKEN EXPERIMENTATION, SHAM EBPM ............................................................. 180 A. Broken Experimentation .......................................................................................... 181 B. Sham EBPM ..................................................................................................................... 186 1. Technical Breakdowns .......................................................................... 186 2. Evidence Distortion ................................................................................ 187 3. Terminal Experimentation................................................................... 191 4. Ratcheting .................................................................................................. 193 III. THE COSTS (AND BENEFITS) OF SHAM EBPM ....................................................... 198 A. Welfare and Efficiency ............................................................................................... 198 B. Shallow Signaling and Scientific Legitimacy ................................................. 200 1. Free-Riding ................................................................................................ 201 2. Erosion of Scientific Legitimacy ......................................................... 202 C. Chilling Scientific Production of Knowledge .................................................. 203 D. Governmental Citizens as Human Subjects .................................................... 204 1. Research Lacking Practical Benefits ................................................. 205 2. The Government as Experimenter .................................................... 208 E. Positive Externalities .................................................................................................. 209 1. Collateral EBPM ....................................................................................... 210 2. The Hawthorne Effect ............................................................................ 211 3. Clarifying Values ...................................................................................... 212 IV. PROMOTING AND PRESERVING EVIDENCE-BASED EFFORTS ................................. 212 A. Terms of Engagement: The Limits of EBPM and Experimentation .... 213 B. Ex Ante Commitments ................................................................................................ 216 C. Evaluation Mandates .................................................................................................. 219 D. Evaluation Transparency, Ex Ante and Ex Post ........................................... 220 E. Teams of Rivals in Evidence Production ........................................................... 222 F. Judicial Review and Agency Oversight of Evidence-Based Decisions ........................................................................................................................ 223 CONCLUSION ...................................................................................................................... 226 INTRODUCTION In 2012, death penalty supporters and detractors eagerly awaited the publication of a report by the National Research Council (NRC), two years in the writing, that sought to resolve the question of whether the penalty in 151 YALE LAW & POLICY REVIEW 38 : 150 2019 fact deters homicide.1 The NRC’s prior effort to do so in 1978 had been inconclusive, finding that existing studies “provide[d] no useful evidence on the deterrent effect of capital punishment.”2 In the ensuing decades, however, a mass of new research had emerged with wildly divergent findings, ranging from showings of extreme deterrence to the suggestion of increased homicides where the death penalty is imposed.3 Rancorous disagreements flourished in both politics and the scientific literature. The new committee would weigh this body of evidence, appraise its methodology, and issue an updated conclusion. But when the report emerged, researchers and policy advocates alike were bewildered. Despite all of the intervening research, the committee had concluded that “all of the research about deterrence and the death penalty done in the past generation, including by some first-rank scholars at the most prestigious universities, should be ignored.”4 Evidence, therefore, could be no guide to state decisions on whether to retain or abolish the penalty—states were, as they long had been, left to resolve the question primarily on the basis of values rather than empirical evidence. Frustration about the weight and direction of empirical evidence across a range of issues—the deterrence effect of the death penalty,5 the extent to * Associate Professor of Law, Columbia Law School. I am grateful to Gillian Metzger, Susan Sturm, Jeff Fagan, Colleen Chien, Ian Ayres, Matt Spitzer, David Schwartz, Shari Diamond, Josh Kleinfeld, and participants at the Northwestern Law faculty workshop for valuable feedback and conversations that shaped this work. I am also grateful to Juliana Bennington for excellent research assistance. All errors herein are my own. 1. See NAT’L RESEARCH COUNCIL, DETERRENCE AND THE DEATH PENALTY (Daniel S. Nagin & John V. Pepper, eds., 2012). I am grateful to Josh Kleinfeld for suggesting this case study. 2. Id. at 1 (quoting the NRC report from 1978). 3. Id. at 1; see also John J. Donohue III & Justin Wolfers, Uses and Abuses of Empirical Evidence in the Death Penalty Debate, 58 STAN. L. REV. 791, 793 (2006). 4. Editorial Board, The Myth of Deterrence, N.Y. TIMES (Apr. 27, 2012), https://www.nytimes.com/2012/04/28/opinion/the-myth-of- deterrence.html [https://perma.cc/JWA9-79C5]. 5. NAT’L RESEARCH COUNCIL, supra note 1. 152 BROKEN EXPERIMENTATION, SHAM EVIDENCE-BASED POLICY which sanctions6 and walls7 deter migration, the extent to which supervised injection facilities reduce opioid overdoses,8 to name a few—has taken center stage because, by and large, we think evidence should matter in legislation and agency regulation. Indeed, evidence-based policy-making (EBPM) may be having a popular moment. EBPM refers to the systematic use of empirical research evidence by legislators and agency personnel when making government decisions, and it has long been the province of policy wonks and scholars eager for research evidence to play a larger role in decision-making. The process of EBPM entails both the generation and use of empirical evidence at several time points, suggesting that regulators and legislators should (1) consult past research evidence when making policy choices, including evidence that characterizes a public problem and the likely impacts of different policy