42705-crn_106-3 Sheet No. 4 Side A 03/25/2021 10:13:22 R R R R ..... 543 543 ...... 537 537 ...... CIENCES S OCIAL ITIGATION S L IVIL 529 ...... 545 545 ...... C RISIS IN THE † Edith Beerdsen C VIDENCE IN KNOWLEDGE CRISIS CRISIS KNOWLEDGE E ...... 530 530 ...... NOWLEDGE K Replication Crisis LITIGATION SCIENCE AFTER THE THE AFTER SCIENCE LITIGATION CIENTIFIC HE The knowledge crisis that has been causing turmoil in the that has been causing turmoil The knowledge crisis the broad implications of This Article is the first to address A. S T Acting Assistant Professor of Lawyering, New York University School of Acting Assistant Professor of Lawyering, social sciences for the past few years represents a fundamen- the past few years represents social sciences for to create understanding of what it takes tal shift in scientists’ reliable science. Replication This crisis, now known as the that the spotlight on research practices Crisis, has thrown that are of research results in ways undermine the reliability made visible and cannot be remedied or typically invisible after the fact. the defects is ample reason to believe that There Crisis are in the course of the Replication of method uncovered are in aca- in litigation science as they at least as pervasive demic science. has failed to recognize Yet the legal profession and address them. of scientific knowl- the Replication Crisis for the production edge in a civil-litigation context. from the Drawing on insights practice is simply in- Crisis, it argues that current procedural the information it needs to capable of providing a court with of the reliability of scientific make an accurate assessment evidence.a number of core principles, The Article identifies science to the Replica- drawn from the response of academic to the treatment of scientific tion Crisis, that can guide reforms evidence in civil litigation. that shoring up the courts’ It argues requires a rethinking of capacity to evaluate scientific evidence knowledge and a re- the entire chain of creation of scientific that chain. framing of the role of the court in I. II. † M K NTRODUCTION I \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 1 25-MAR-21 9:54 Law. Jason M. Chin, Vincent Daly, Jonah B. I am grateful to Edward K. Cheng, Mc- Gelbach, J. Benton Heath, Helen Hershkoff, Christopher B. Jaeger, Justin Crary, Erin E. Murphy, Dale A. Nance, Alexander A. Reinert, Daniel C. Richman, John Sexton, David Simson, James Steiner-Dillon, Jacob Victor, Maggie Wittlin, - and participants at the 2020 Evidence Summer Workshop and the NYU Lawyer ing Scholarship Colloquium for helpful discussions and suggestions. For diligent Wang. Thanks also to Lily A. Coad, research assistance, I thank Yan (C´eline) staff Victor Flores, John R. Mucciolo, Lachanda R. Reid, Michelle J. Zhu, and the of the Cornell Law Review for their conscientious and thoughtful editing. C Y 42705-crn_106-3 Sheet No. 4 Side A No. 4 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 4 Side B 03/25/2021 10:13:22 R R R R R R R R R R R R R R . . M K C Y SYCHOL SYCHOL . P , https:// ED OC , 21 P ...... 565 565 ...... 583 ...... & S [Vol. 106:529 106:529 [Vol...... 582 582 ...... Power Posing: Brief (Dec. 12, 2012, 3:06 PM), ERSONALITY LATE ...... 565 565 ...... , S ...... 555 555 ...... , 100 J. P Like Bem’s paper on extra- ...... 584 584 ...... 2 and even catapulted one of its and even catapulted one of its CIENCE 3 S The two studies had more in com- The two studies had more in 4 ...... 562 562 ...... 564 ...... NTRODUCTION I The Most Popular Talks of All Time, T ...... 558 558 ...... 585 585 ...... 572 572 ...... 568 568 ...... ITIGATION CORNELL CORNELL LAW REVIEW ...... 584 584 ...... L - (Jan. 5, 2011), https://www.nytimes.com/2011/01/06/sci Planning Planning Commitment Commitment Presentation Early Proceedings on Proposed Litigation Early Proceedings Science Requirements Updated Disclosure Analytical Flexibility Consideration of ...... 588 588 ...... IMES Replication Renaissance Replication 1. 2. 3. The Role of the Judge The Role of the Plaintiffs and The Balance Between Defendants Proposals Proposals 1. 2. 3. Reliability Problems in Litigation Science Litigation Science Problems in Reliability A year earlier, a team of psychologists had published A year earlier, a team of psychologists That Time I Tried to Be Wonder Woman (May 15, 2012, 1:30 PM), https://www.wired.com/2012/05/st-cuddy 1 ESSONS FOR MPLICATIONS B. A. B. I L B. A. Dana R. Carney, Amy J.C. Cuddy & Andy J. Yap, Dana R. Carney, Amy J.C. Cuddy & See, e.g., Danielle Venton, Power Postures Can Make You Feel More Power- In 2012, coauthor Amy Cuddy gave a presentation about power poses that, Feeling the Future: Experimental Evidence for Anomalous Daryl J. Bem, Feeling the Future: Experimental Evidence IRED Novelty catches attention. Daryl J. In 2011, psychologist 2 3 4 1 . 1363, 1366 (2010). III. IV. ONCLUSION CI a study demonstrating that adopting confident postures a study demonstrating that changes that reduce stress (“power poses”) causes hormonal and increase “feelings of power.” ]. ence/06esp.html [https://perma.cc/ZLW2-Y3XT - https://slate.com/human-interest/2012/12/amy-cuddy-and-power-poses-can ] body-language-affect-your-confidence.html [https://perma.cc/TGQ9-LJXL (describing the author’s experimentation with power poses). Nonverbal Displays Affect Neuroendocrine Levels and Risk Tolerance Nonverbal Displays Affect Neuroendocrine ful, W with more than fifty-five million views as of early 2020, is still the second-most watched TED talk in history. Retroactive Influences on Cognition and Affect Retroactive Influences on Cognition and sensory perception, the power-pose study received extensive sensory perception, the power-pose coverage by the popular press, coauthors to internet fame. Bem made the front page of the New York Times with a study Bem made the front page of the future”––thatshowing that people can “feel it is possible to a test by studying after taking improve one’s performance on the test. C \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown 530 Seq: 2 25-MAR-21 9:54 Journal’s Paper on ESP Expected to Prompt 407, 407 (2011); Benedict Carey, Journal’s Paper on ESP Expected to Outrage, N.Y. T S [https://perma.cc/GHF5-FH6F] (interviewing Cuddy about power poses); Katy Waldman, 42705-crn_106-3 Sheet No. 4 Side B No. 4 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 5 Side A 03/25/2021 10:13:22 R R R . 6 CI By 531 531 OPICS . S . 426, note 5, 11 . T HIL In con- SYCHOL SYCHOL 8 supra P . P , 29 P OC The findings, The findings, 9 & S infra subpart II.A and IRECTIONS And in each case, case, in each And D The results were so The results 5 7 —one large that drew 10 ERSONALITY . 95, 96 (2011) (examining the URRENT CI . S , 20 C , 100 J. P SYCHOL , 22 P note 1 (noting that copies of Bem’s article “have note 1 (noting that copies of Bem’s LITIGATION SCIENCE SCIENCE LITIGATION Eric-Jan Wagenmakers, Ruud Wetzels, Denny Borsboom & Han L. J. Eric-Jan Wagenmakers, Ruud Wetzels, Larry Laudan, Epistemic Crises and Justification Rules Powerful Postures Versus Powerful Roles: Which Is the Proximate Corre- What Are Emotion Expressions for? See, e.g., Li Huang, Adam D. Galinsky, Deborah H. Gruenfeld & Lucia E. See See infra subpart II.A. , Wagenmakers, Wetzels, Borsboom & van der Maas, See, e.g., Wagenmakers, Wetzels, Borsboom & study, see For critiques immediately following Bem’s See, e.g., Carey, supra See Inside the field of psychology, the reception of the two stud- of the two the reception field of psychology, Inside the 9 6 7 8 5 M K 10 11 trast, psychology researchers cited the power-pose study researchers cited the trast, psychology questioning its conclusions. largely without at 427 (When results are reported selectively, “we simply do not know how many at 427 (When results are reported selectively, only to come up short . . . [or] how other factors were taken into consideration and discarded.”). many other hypotheses were . . . tested Guillory, late of Thought and Behavior? notes 76–80. researchers . . . and have generated a circulated widely among psychological mixture of amusement and scorn”). while novel, were not as bewildering as those found by Bem, not as bewildering as those while novel, were scrutiny. not attract the same immediate and therefore did The to a searching in- would not be subjected power-pose paper field of psychology after publication, when the quiry until years of an epistemic crisis was in the midst the flaws were virtually undetectable in the published papers. in the published undetectable were virtually the flaws hard to believe that many researchers assumed that the meth- assumed that researchers believe that many hard to flawed. produced them had to be odology that had 2017, both studies had been discredited, along with scores of 2017, both studies had been ies was vastly different, at least initially. vastly different, ies was was Bem’s study apart immediately. and picked scrutinized 395, 398 (2011) (examining nonverbal emotional expression). 271, 273 (2001) (“[A]n epistemic crisis occurs when a group or community . . . 271, 273 (2001) (“[A]n epistemic crisis occurs when a group or community it finds itself with reasons to question the correctness of the rules and structures has been using for fixing beliefs.”). [https://perma.cc/ www.ted.com/playlists/171/the_most_popular_talks_of_all 3TTZ-99L7] (last visited Nov. 20, 2020). \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 3 25-MAR-21 9:54 swaths of research findings into question and that would radi- findings into question and swaths of research done in the social sciences. cally alter the way research is effects of body posture on behavior and thought); Azim F. Shariff & Jessica L. effects of body posture on behavior and thought); Azim F. Shariff & Jessica Tracy, 2021] 2021] power. viral and novelty than their mon suffered studies Both flaws. methodological severe the same from 427 (2011) (arguing that Bem obtained his result through a “fishing expedition” 427 (2011) (arguing that Bem obtained only the successful ones); Letter from using a variety of analyses and reporting Berkeley Dep’t of Psychology, https:// Dana R. Carney, Assoc. Professor, UC / faculty.haas.berkeley.edu/dana_carney [https://perma.cc/5BFK- pdf_my%20position%20on%20power%20poses.pdf in a letter by the power-pose study’s NXLL] (last visited Nov. 20, 2020) (conceding, or generated by attempting numer- co-author, that the results were “p-hacked,” “the ones that ‘worked’”).ous different analyses and reporting only Why Psychologists Must Change the Way They Analyze Their Data: van der Maas, Why Psychologists Must Change the Way The Case of Psi: Comment on Bem (2011) C Y 42705-crn_106-3 Sheet No. 5 Side A No. 5 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 5 Side B 03/25/2021 10:13:22 R R R R M K A C Y , 13 , 26 15 . 511, 514–15 The Replica- [Vol. 106:529 106:529 [Vol. 16 SYCHOL . P Cancer Research Is Bro- note 15, at 512 (methods EV . 687, 690 (2017) (conclud- throwing into doubt throwing into . R CI 14 NN . S e0202121, at 2 (2018), https:// 12 notes 84 and 89 (reporting failed NE , 69 A S O SYCHOL Evaluating Replicability of Laboratory Evaluating Replicability of Laboratory O - 1433, 1434 (2016) (reporting failed repli , 28 P , 13 PL CIENCE was sparked by a series of developments of developments by a series was sparked , 351 S 13 CORNELL CORNELL LAW REVIEW Poor Statistical Reporting, Inadequate Data Presentation and eroux, Poor Statistical Reporting, Inadequate e1002106, at 8 (2015), https://journals.plos.org/plosbiology/arti- . 653, 653 (2015) (attempting and largely failing to replicate Carney, . 653, 653 (2015) (attempting and largely / (Apr. 19, 2016, 9:21 AM), https://slate.com/technology/2016/04 Assessing the Robustness of Power Posing: No Effect Eva Ranehill et al., Assessing the Robustness of Power Posing: CI . S IOLOGY , Megan L. Head, Luke Holman, Rob Lanfear, Andrew T. Kahn & See, e.g., Megan L. Head, Luke Holman, Rob See, e.g., Nelson, Simmons & Simonsohn, supra See infra note 69. See, e.g., Colin F. Camerer et al., See LATE This crisis of knowledge, now commonly known as the known now commonly of knowledge, This crisis S B O 15 16 13 14 12 SYCHOL the reliability of research approaches that, until recently, were research approaches that, the reliability of scrutiny. and without attracting serious applied routinely / cle/file?id=10.1371/journal.pbio.1002106&type=printable [https://perma.cc B9VS-BP2K] (concluding that “p-hacking is rife”); Leif D. Nelson, Joseph Simmons & Uri Simonsohn, Psychology’s Renaissance - biomedicine-facing-a-worse-replication-crisis-than-the-one-plaguing-psychol ] (“[M]uch of cancer research in the lab ogy.html [https://perma.cc/7WNG-BPFZ data analysis . . . and poor experimen- . . . simply can’t be trusted” due to “[s]loppy that tal design . . . .”); see also infra note 126 and source cited therein (explaining earlier but have now come under certain common practices had been criticized renewed and wider-spread scrutiny). cation attempts in economics); Joanna Diong, Annie A. Butler, Simon C. Gandevia cation attempts in economics); Joanna & Martin E. H´ Spin Persist Despite Editorial Advice Experiments in Economics Michael D. Jennions, The Extent and Consequences of P-Hacking in Science PL period of soul-searching led to a broad recognition among led to a broad period of soul-searching research depends (1) the reliability of published scientists that: that had not been rigor in ways crucially on methodological rigor and (2) lack of methodological sufficiently appreciated; that is entirely research results in a manner can contaminate a peer examining those results. undetectable to & n.4 (2018) (reporting evidence of widespread use of p-hacking and reluctance by researchers to consider it “a problem worth worrying about”). Cuddy, and Yap’s power-pose study); Joseph P. Simmons & Uri Simonsohn, Cuddy, and Yap’s power-pose study); Power Posing: P-Curving the Evidence “Replication Crisis,” “Replication \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown 532 Seq: 4 25-MAR-21 9:54 other psychology studies whose results researchers had long long had researchers results whose studies psychology other credible. and trustworthy considered in the early- to mid-2010s that signaled that much of pub- that much that signaled early- to mid-2010s in the replicated. be trusted or could not research lished psychology as eco- fields as diverse psychology to spread from It rapidly and medicine, nomics, pharmacology, journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0202121&type= journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0202121&type= ] (reporting wide use of inappropriate printable [https://perma.cc/X85S-5E2K Daniel Engber, statistical techniques in pharmacology); ken, S attempts to replicate for large numbers of studies). attempts to replicate for large numbers ing based on meta-analysis that the power-pose hypotheses were “currently lack- ing based on meta-analysis that the power-pose infra ing in empirical support”); sources cited of data collection and analysis that were in widespread use for decades made it of data collection and analysis that were in widespread use for decades made those “impossible to distinguish between findings that are true and replicable and that are false and not replicable”). on Hormones and Risk Tolerance in a Large Sample of Men and Women on Hormones and Risk Tolerance in P 42705-crn_106-3 Sheet No. 5 Side B No. 5 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 6 Side A 03/25/2021 10:13:22 R 533 533 Unless the 21 The common 20 note 15, at 512 (arguing that

supra - as they are in academic sci 23 LITIGATION SCIENCE SCIENCE LITIGATION Its focus transcends individual techniques or individual techniques transcends Its focus (“Researchers now understand that the old ways of collecting id. (“Researchers now understand that the These reforms are largely procedural in nature, are largely procedural in These reforms 17 19 Nelson, Simmons & Simonsohn, 22 18 See infra subpart II.A. See infra subpart II.B. in the Replication While the problems uncovered See id. See See id. In this Article, I will use the term “litigation science” for the mode of science See, e.g., There is ample reason to believe that the defects of method There is ample reason to believe The Replication Crisis, this Article argues, holds important Crisis, this Article argues, holds The Replication M K 18 19 20 21 22 23 17 Crisis first came up in the context of projects involving statistical analysis, they Crisis first came up in the context of projects involving statistical analysis, are not limited to this setting.scientific project that involves multiple steps Any has the potential to have its reliability undermined by analytical flexibility. one or produced in connection with civil litigation by an expert witness retained by and analyzing data produce results that are not diagnostic of truth and that a and analyzing data produce results that needed. new, more enlightened approach is have [researchers] of Thousands embraced this notion.”). types and degrees of analytical flexibility are either exposed or types and degrees of analytical no way to assess the extent to eliminated, there is typically contaminated reported research which their presence has results. thread in much of these reforms is the need to constrain re- thread in much of these reforms research process.searchers’ flexibility during the It is this “an- alytical flexibility”—the to adapt, supplement, or even flexibility protocols when the study is al- completely make up research ready underway—that to lead to research has a propensity undetectable ways. results that are unreliable in uncovered in the course of the Replication Crisis are at least as uncovered in the course of the pervasive in litigation science and as-yet-unappreciated lessons for the treatment of scientific lessons for the treatment and as-yet-unappreciated courts.evidence by the conversation is still ongoing, While the around a num- developing in the social sciences a consensus is the reliability of reforms aimed at improving ber of institutional research. \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 5 25-MAR-21 9:54 tools; rather, it centers on methodology itself—on on methodology it centers tools; rather, the pro- how the relia- designed affects protocols are which research cess by the role that institutional and on research outcomes, bility of play in methodological and incentives culture, structures, design. and include stricter regimes for planning a research project, regimes for planning and include stricter is underway, modifications while the project restrictions on of data. and radically increased openness 2021] 2021] - under in scientists’ shift a fundamental represents Crisis tion scientific reliable create to it takes of what standing knowledge. analytical flexibility makes it impossible to distinguish true findings from false analytical flexibility makes it impossible to distinguish true findings from ones). C Y 42705-crn_106-3 Sheet No. 6 Side A No. 6 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 6 Side B 03/25/2021 10:13:22 . . M K EV UB C Y . 727 . P YNERGY EV L. R MPIRICAL : S Improving L ’ SYCHOL NT L. R TTAWA . I , 15 J. E CI [Vol. 106:529 106:529 [Vol. , 20 P Open Forensic Sci- S , 50 O ARSHALL , analytical Second, analytical No scholarship has ORENSIC 25 Psychological Science’s Repli- , 49 J. M , 2 F Psychological Science’s Replicability Cri- 255 (2019) (same). treatments of the The most in-depth CORNELL CORNELL LAW REVIEW IOSCIENCES - of science have tradition Third, since both modes . 320 (2018) (assessing the implications of the Replication Crisis for . 320 (2018) (assessing the implications The Crisis in Scientific Publishing and Its Effect on the Kevin D. Hill, The Crisis in Scientific Publishing and , in both modes of science, researchers face power - face researchers science, of both modes , in First TUD Forensic Science Needs Registered Reports See infra subpart III.A. See 24 Law and Psychology Grows Up, Goes Online, and Replicates Law and Psychology Grows Up, Goes Online, S This Article is the first to address the broad implications of first to address the broad This Article is the & L. 225 (2014) [hereinafter Chin, Y ’ 24 25 OL EGAL focused on the lessons the Crisis holds for judges and practi- focused on the lessons the Crisis tioners of litigation science. point Moreover, a fundamental far: that, in light of the Replica- has been left unaddressed thus practice is simply incapable of tion Crisis, current procedural it needs to make an providing a court with the information of scientific evidence. accurate assessment of the reliability more parties and the term “academic science” for science conducted for the gen- more parties and the term “academic eral purpose of expanding knowledge. L to adopt practices of the “open science” 41 (urging academic forensic science & Alicia Rairden, movement); Jason M. Chin, Gianni Ribeiro ence, 6 J.L. & B \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown 534 Seq: 6 25-MAR-21 9:54 ence. - sis] (identifying weaknesses in the Daubert admissibility standard for the evalua tion of “framework evidence”—expert testimony that relies on existing scientific literature) and Jason M. Chin, Bethany Growns & David T. Mellor, Expert Evidence: The Role of Open Science and Transparency flexibility does not distinguish between the two modes of sci- the two modes between does not distinguish flexibility of scientific find- it impairs the reliability ence: when present, the findings were of the purpose for which ings regardless generated. results selec- to present research ally allowed experimenters can be nature of this impairment tively, the undetectable endemic in both. expected to be knowledge Crisis for the production of scientific the Replication context.in a civil-litigation - scholars have made a connec Few law. tion between the Crisis and evidence ful incentives associated with unreliable research results: results: research unreliable with associated incentives ful that expectations demand norms and and institutional cultural while re- results, produce publishable researchers academic to produce pressure setting are under in a litigation searchers jury. to a that can be presented results Jason M. Chin, Rory McFadden & Gary Ed- empirical legal research; see also Jason M. Chin, Rory McFadden & Gary mond, P (2016) (questioning the utility of peer review as a marker of reliability in the wake (2016) (questioning the utility of peer largely on scientific fraud rather than struc- of the Replication Crisis and focusing Irvine, David A. Hoffman & Tess Wilkinson- tural methodological problems); Krin Ryan, Crisis’s import for litigation are Jason M. Chin, cability Crisis and What It Means for Science in the Courtroom 365 (2019) (proposing reforms for the treatment of forensic evidence in Canadian 365 (2019) (proposing reforms for the treatment of forensic evidence in Canadian the courts in light of the Replication Crisis), but even these articles do not address - most critical implications for the production and assessment of scientific knowl edge in the context of civil litigation. Admissibility of Technical and Scientific Evidence Admissibility of Technical and Scientific 42705-crn_106-3 Sheet No. 6 Side B No. 6 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 7 Side A 03/25/2021 10:13:22 535 535 By the time they 31 Accounting for the role analytical Accounting for Containing or rendering visible the visible the or rendering Containing . 123, 125 (2008) (arguing that Daubert rests on a 28 26 LITIGATION SCIENCE SCIENCE LITIGATION ERSP P The Replication Crisis undermines the tradi- The Replication Crisis undermines EALTH but doing so requires a careful rethinking of the a careful rethinking of but doing so requires 30 27 . H NVTL Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 592–94 (1993); Sheila Jasanoff, Representation and Re-Presentation in Litigation Sci- and the more critical view that the quality of scientific and the more critical view that 29 See infra Part II. See See infra subpart III.B. See Daubert, 509 U.S. at 593 (assigning judges a gatekeeping role with See Id. at 128. These insights also have broader theoretical implications These insights also have broader This Article identifies a number of core principles—drawnThis Article identifies It is critical that courts assess and shore up their capacity capacity up their shore and assess that courts critical It is M K 26 27 28 29 30 31 respect to scientific evidence, “confident that federal judges possess the capacity respect to scientific evidence, “confident that federal judges possess the capacity to undertake this review” based on a flexible set of factors). ence, 116 E flawed assumption that criteria external to the legal process can guide judges in flawed assumption that criteria external to the legal process can guide judges their screening of scientific evidence). flexibility plays in the creation of scientific evidence will require in the creation of scientific evidence flexibility plays of studies than is currently the much more rigorous planning study protocols, with or norm; commitment to predetermined and a wholesale rethinking of without involvement of the court; exchanges between liti- the timing and nature of information gants and the court. law and science.for the intersection between In particular, view that science offers judges they disrupt both the traditional the evaluation of scientific evi- a set of objective principles for dence “invisible contamination” of analytical flexibility will therefore flexibility will of analytical contamination” “invisible reforms. institutional procedural and require I argue that existing legal can be implemented within these reforms frameworks, \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 7 25-MAR-21 9:54 tional view that judges can meet their gatekeeping obligations tional view that judges can meet evidence from reaching the and prevent insufficiently reliable a scientist.” finder of fact by “thinking like entire chain of creation of scientific evidence and a reframing of creation of scientific evidence entire chain of court in that chain. the role of the Replication Cri- of academic science to the from the response sis—that of scientific evi- reforms to the treatment can guide actions. dence in civil evidence can only be evaluated with reference to both process evidence can only be evaluated and results. 2021] 2021] evidence. scientific to evaluate Crisis has The Replication can and incentives structures that institutional demonstrated scientific- reliability of undermine the enhance or greatly production. knowledge examine proposed scientific evidence—afterexamine proposed scientific it has been fully Frye v. United States, 293 F. 1013, 1014 (D.C. Cir. 1923). C Y 42705-crn_106-3 Sheet No. 7 Side A No. 7 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 7 Side B 03/25/2021 10:13:22 R M K 36 C Y . L. 32 AND , 66 V [Vol. 106:529 106:529 [Vol. While assigning While assigning Uncertainty and Informed 33 Reliable, reproducible liti- Reliable, reproducible . 257, 267 (2005) (arguing that as Scientific evidence plays a EV 34 35 . L. R ICH CORNELL CORNELL LAW REVIEW See infra subparts II.A and III.B. Jasanoff, supra note 30, at 129. See infra subparts III.B and IV.A. Michael S. Pardo, The Nature and Purpose of Evidence Theory See, e.g., Margaret A. Berger & Aaron D. Twerski, More broadly, this Article aims to spark a discussion about More broadly, this Article aims The critical view, however, also requires modification. modification. also requires view, however, The critical . 547, 554 (2013). 32 33 34 35 36 EV courts a monitoring role would be a step in the right direction, role would be a step in courts a monitoring that have been to prevent or solve the problems it is insufficient Replication Crisis. exposed by the R gation science requires courts not merely to assume a supervi- requires courts not merely to gation science but to create the respect to its production sory role with to enable party ex- and culture necessary settings, conditions, it.perts to produce - waiting by the gate for the evi Instead of should ride out in its full-fledged form, courts dence to arrive appear on the creators as soon as they and meet evidence of the scientific evidence at horizon, engaging with the creation the process of its creation. an early stage and guarding litigation science and how these methodological standards in evidentiary standards.standards interact with prevailing The stakes are high. important Evidentiary standards “determine in the first place [and] which issues such as who gets to trial verdicts will be allowed to stand.” large role in many civil actions, and a large number of impor- large role in many civil actions, tant cases—particularlyinvolving toxic torts and envi- those ronmental damage—cannot brought at all without the be conducted by expert witnesses. support of scientific studies Sheila Jasanoff, a leading voice in science and technology stud- and technology voice in science a leading Sheila Jasanoff, courts scientific evidence, in evaluating argued that ies, has process of more on the end result and focus less on the should an indicator of reliability. its creation as If courts and litigants fail to absorb the lessons of the Replica- If courts and litigants fail to absorb now understood to inject unrelia- tion Crisis, practices that are scientific data will continue to bility into the creation of in significant and largely unde- contaminate litigation science tectable ways. that depend on The just outcome of civil actions scientific evidence demands an the accuracy and reliability of and practices that engen- understanding of the misconceptions \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown 536 Seq: 8 25-MAR-21 9:54 compiled and summarized—it and compiled the or resolve detect late to is too flexibility. by analytical is caused that of unreliability kind standards of admissibility of scientific evidence have become more exacting, it has standards of admissibility of scientific evidence have become more exacting, a become “nigh impossible” in certain toxic torts cases for plaintiffs to maintain civil action). Choice: Unmasking Daubert, 104 M 42705-crn_106-3 Sheet No. 7 Side B No. 7 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 8 Side A 03/25/2021 10:13:22 An , 31 537 537 § 801(a) (An § ODE But experts . C 37 . 1113, 1119 (analyz- VID EV . E AL ITIGATION . L. R L IS IVIL C I VIDENCE IN The studies that experts can conduct The studies that experts can . 702(a) (An expert can testify if “the expert’s scien- . 702(a) (An expert can testify if “the E 38 LITIGATION SCIENCE SCIENCE LITIGATION VID . R. E CIENTIFIC ED S J. 375, 381 (1991) (reporting results of a three-month study of civil See, e.g., Anthony Champagne, Daniel Shuman & Elizabeth Whitaker, See, e.g., F Testifying expert witnesses play a central role in the U.S. Testifying expert witnesses play The remainder of this Article is organized as follows.Article is organized of this The remainder Part I M K 38 37 URIMETRICS - cases in Dallas, Texas, and finding that experts, including experimental psycholo of gists, economists, biochemists, and engineers, were used in more than half cases); Samuel R. Gross, Expert Evidence, 1991 W Empirical Examination of the Use of Expert Witnesses in American Courts J of evidence.very frequently are also creators They routinely or against a party’s claims or create evidence in support of and analyses tailor-made defenses, by conducting experiments for the case at hand. expert can testify on topics “sufficiently beyond common experience that the expert can testify on topics “sufficiently beyond common experience that opinion of an expert would assist the trier of fact.”). litigation system. interpreters of They are often thought of as explain technical, scientific, or evidence: they present and the finder of fact might find diffi- other specialized matter that expert’s guidance. cult to interpret without an \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 9 25-MAR-21 9:54 ing data on 529 civil jury trials in California State Superior Courts from 1985 and ing data on 529 civil jury trials in California State Superior Courts from 1985 An- 1986 and finding that experts testified in eighty-six percent of those trials); drew W. Jurs, Expert Prevalence, Persuasion, and Price: What Trial Participants 2021] 2021] of the a reconceptualization and Crisis the Replication dered science. of litigation and evaluation creation in the role court’s of liti- and evaluation for the creation the frameworks reviews courts applied in federal are currently as they gation science assump - as the implicit states, as well courts of most and the in them.tions embedded - describes the Replication Cri Part II the reforms and through the social sciences and sis that swept in those that are garnering growing acceptance changed norms scientific communities.- argues that the problems un Part III to be at least as science research are likely covered in social social-science re- science and draws on prevalent in litigation the creation and Crisis to propose reforms for sponses to the litigation. scientific evidence in civil presentation of IV Part measures on the implications of the proposed considers the they reframe the science, including how creation of litigation in the expert-discovery phase of a role of judges and advocates ability to fulfill their gatekeeping litigation and improve judges’ evidence from reaching the finder obligation to keep unreliable of fact. tific, technical, or other specialized knowledge will help the trier of fact to under- tific, technical, or other specialized knowledge in issue.”); C stand the evidence or to determine a fact C Y 42705-crn_106-3 Sheet No. 8 Side A No. 8 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 8 Side B 03/25/2021 10:13:22 R R M K C Y . L. supra ICH If it is Dirty Little 43 [Vol. 106:529 106:529 [Vol. In determining In determining § 350 (“No evidence is § 42 ODE . C In re Countrywide Fin. Corp. VID Modeling Relevance, 75 M . E AL - note 36 (asserting that certain environ . 47, 52 (2007) (“In large cases, it is not . 47, 52 (2007) (“In large cases, it is . L.J. 353, 355 (2016) (stating that expert . L.J. 353, 355 (2016) (stating that ITIG ND . 102 (Federal Rules of Evidence “should be con- . 102 (Federal Rules of Evidence “should . 402 (providing that relevant evidence is admissible . 702 (providing requirements for testimony by expert , as with any other type of evidence, First, as with any other type of evidence, VID VID VID CORNELL CORNELL LAW REVIEW . 402. for evidence to be For views on what it means VID Before an expert can provide testimony in a can provide testimony in Before an expert . R. E . R. E . R. E ED 41 ED ED , for testimony to be admissible, it must be Second, for testimony to be admissible, . R. E ED 44 F 40 - for a plain it is not unusual areas of litigation, In some See, e.g., F See, e.g., F See, e.g., F See , Jones v. United States, No. 2:16-CV-00435-JRS-DLP, 2019 U.S. See, e.g., Jones v. United States, No. 2:16-CV-00435-JRS-DLP, See, e.g., Berger & Twerski, supra 39 Expert evidence is not automatically admissible in state or is not automatically admissible Expert evidence . 1021, 1025–26 (1977) (arguing that evidence is relevant if it tends to make a 41 42 43 44 39 40 EV should be heard as part of the proceeding. should be heard unheard of for one side to designate 20 or more experts . . . .”). unheard of for one side to designate 20 admissible except relevant evidence.”). Mortg.-Backed Sec. Litig., 984 F. Supp. 2d 1021, 1022 (C.D. Cal. 2013) (involving Mortg.-Backed Sec. Litig., 984 F. Supp. of securities litigation); Swirsky v. Carey, analysis of mortgage loans in support analysis of lyrics and melodies in 376 F.3d 841, 845 (9th Cir. 2004) (involving Chin v. Port Auth. of N.Y. & N.J., 685 support of copyright infringement action); F.3d 135, 143–44statistical study in support of race- (2d Cir. 2012) (involving a discrimination action). Dist. LEXIS 14382, at *7 (S.D. Ind. Jan. 30, 2019) (involving an analysis of water Dist. LEXIS 14382, at *7 (S.D. Ind. Jan. tort claims); samples in defense of environmental witnesses appeared in 86% of civil trials in an urban Iowa county, in line with witnesses appeared in 86% of civil trials Robert J. Shaughnessy, results in earlier studies in different locales); Secrets of Expert Testimony, 33 L not relevant to the claims or defenses in the case, it cannot be not relevant to the claims or admitted. whether to admit expert testimony, courts apply jurisdiction- expert testimony, courts whether to admit to an expert’s credentials and dependent admissibility criteria proposed testimony. to be admissible. expert testimony must be relevant federal court. determinations, evidentiary To promote just testimony that on the types and scope of expert rules put limits can be heard. Really Think About Experts, 91 I \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 10 538 25-MAR-21 9:54 are too numerous to list.too numerous are water soil or can examine Experts analyze cases, tort environmental with connection in samples music litigation, dissect to securities data relevant financial disputes—the with copyright in connection samples is end- list less. civil case, the party proffering the expert must justify why this proffering the expert must civil case, the party outsider—an to the with no inherent connection individual facts at issue— knowledge of the case and no independent tiff’s claims to rely heavily, or even primarily, on litigation primarily, on or even to rely heavily, tiff’s claims science. relevant, see, for example, Richard O. Lempert, fact more or less probable than it would be without the evidence); Pardo, of the note 35, at 558 (arguing that evidence is relevant if it supports a theory R witness in federal court). and irrelevant evidence is not admissible); C strued . . . to the end of ascertaining the truth and securing a just strued . . . to the end of ascertaining determination.”). mental tort cases for plaintiffs cannot be brought without expert evidence). mental tort cases for plaintiffs cannot 42705-crn_106-3 Sheet No. 8 Side B No. 8 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 9 Side A 03/25/2021 10:13:22 . . L ’ AL AT VID 87 539 539 . 702 and and , 27 L. ALIDITY ., N FFICE OF 45 VID V TATES . R. E MTY . O S EL . C . R. E XEC CI EL CIENTIFIC NITED S . 702 (same); see ., E The scope of this S U VID ECH ORENSIC . & T F NSURING CI . 702 (same); see also C : E S VID CIENCE IN THE The focus of this Article is of this Article The focus OURTS S 47 C EEDS OF THE N DVISORS ON 48 A . 702 (same); N.J. R. E ORENSIC RIMINAL F C VID 44 n.94 (2016), https://obamawhitehouse. § expert may testify “[b]ased 801(b) (providing an

OUNCIL OF . 702 (providing that experts are witnesses who are . 702 (providing that experts are witnesses . R. E . 702(a) (providing that an expert may testify if “the . 702(a) (providing that an expert may ODE C . 702 (providing that expert testimony must be “based . 702 (providing that expert testimony DENTIFYING THE S ETHODS LL ’ . 702 (same); N.J. R. E LITIGATION SCIENCE SCIENCE LITIGATION I VID VID CIENCE IN . C VID M S VID ON . 401(a) (providing that evidence is relevant if “it has any that evidence is relevant if “it has any . 401(a) (providing . 702 & advisory committee note, reprinted in 28 U.S.C. at VID . TRENGTHENING § as an expert if he has 720(a) (“A person is qualified to testify VID . R. E . R. E VID . E . R. E A typical third requirement is that the testi- is that third requirement A typical , S OMM RESIDENT ED ED AL . R. E ED 46 ORENSIC ODE LL . R. E . R. E OMPARISON , F . C . 702 (same); I ED OUNCIL ED -C § 801(a) (providing that expert testimony is limited to opinions 801(a) (providing that expert testimony is § F F VID C VID When evaluating the admissibility of expert testimony the admissibility of expert When evaluating . E ODE cf. see also C See, e.g., C See See, e.g., F See, e.g., F in criminal cases is often The reliability of scientific evidence introduced See, e.g., F . 223, 224 & n.2 (2008) (presenting probability-based conceptions of rele- (presenting probability-based conceptions . 223, 224 & n.2 (2008) AL 49 RESIDENT The most widely applied standard of reliability of scientific applied standard of reliability The most widely EATURE C . C . R. E HIL P F M K 49 46 47 48 45 EL ESEARCH VID mony be helpful to the trier of fact. helpful to the mony be “[r]elated to a subject that is sufficiently beyond common experience that the “[r]elated to a subject that is sufficiently trier of fact”). opinion of an expert would assist the (2009), https://www.ncjrs.gov/pdffiles1/nij/grants/228091.pdf [https:// of fo- perma.cc/Y7V8-KB93] (offering recommendations to improve the reliability rensic evidence); P on sufficient facts or data,” and “the product of reliable principles and methods” on sufficient facts or data,” and “the the facts of the case”); D that have been “reliably applied . . . to expert’s scientific, technical, or other specialized knowledge will help the trier of expert’s scientific, technical, or other determine a fact in issue”); D fact to understand the evidence or to “qualified as an expert by knowledge, skill, experience, training, or education”); “qualified as an expert by knowledge, D OF attacked on the basis that the methodology that produced it has not been vali- attacked on the basis that the methodology dated. tendency to make a fact [of consequence in determining the action] more or less tendency to make a fact [of consequence evidence”). probable than it would be without the - archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensic_sci (same). [https://perma.cc/6FES-UMXT] ence_report_final.pdf less Article is limited to civil proceedings, where such arguments feature much prominently. the analysis or methodology that forms the basis for the forms the basis that or methodology (b) the analysis standards jurisdiction-dependent must meet certain testimony of reliability. on the reliability determination. on the reliability or technical evidence is the one articulated by the U.S. Su- is the one articulated or technical evidence Dow Pharmaceuticals, Inc. Daubert v. Merrill preme Court in Rule of Evidence in a revision of Federal and since codified 702. R case, even if it does not make that theory more likely than a competing theory); not make that theory more likely than case, even if it does the Best Explanation Ronald J. Allen, Juridical Proof and Michael S. Pardo & & P \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 11 25-MAR-21 9:54 vance); THE 398–401; Dow Pharm., Inc., 509 U.S. 579, 592–94 Daubert v. Merrell (1993). special knowledge, skill, experience, training, or education sufficient to qualify special knowledge, skill, experience, his testimony relates.”). him as an expert on the subject to which E (same); 702(a) (same); I on matter . . . that is of a type that reasonably may be relied upon by an expert in on matter . . . that is of a type that reasonably to which his testimony relates”). forming an opinion upon the subject also 2021] 2021] the testimony proffering the expert (a) reliable: sufficiently matter, subject the regarding testify to be qualified must C Y 42705-crn_106-3 Sheet No. 9 Side A No. 9 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 9 Side B 03/25/2021 10:13:22 M K 55 C Y OTIONS The M 275 (7th 54 AUBERT VIDENCE [Vol. 106:529 106:529 [Vol. D E Construing the Construing Daubert motions filed 1 (2015), https:// RIAL 50 Daubert.”). , T on litigation practice). ISPOSITION OF OLFSON XAMINATION D E Daubert D. W IMING AND ARREN MPIRICAL , T Courts exercise their gatekeeper Courts exercise their gatekeeper 56 E 52 N & W OOPER : A AUET C. C CORNELL CORNELL LAW REVIEW OURTS A. M C AMES J S59, S61 (2005) (“Even in jurisdictions that purportedly follow HOMAS 51 Although the Federal Rules of Evidence and Although the Federal Rules ISTRICT EALTH 53 D . 702(b)–(d). Court steered clear of providing a “definitive checklist checklist providing a “definitive clear of Court steered - the court into the role of “gate Daubert decision thrust . H VID UB See id. at 589, 590–91 n.9. See generally See id. at 9, 10 tbl.6 (reporting the median time from case commencement See, e.g., T See, e.g., Margaret A. Berger, What Has a Decade of Daubert Wrought?, 95 Daubert, 509 U.S. at 600. Since 2000, these requirements are codified in Daubert, 509 U.S. at 593–94. The EDERAL . R. E . J. P F Daubert motion of 647 days in federal courts, with 74% of 52 53 54 55 56 50 51 M ED keeper” tasked with keeping out expert testimony that does not with keeping out expert testimony keeper” tasked meet these requirements. IN to prior to or with a summary judgment motion and 26% after summary judgment and before or during trial). A F standard is observed in federal courts as well as in the Daubert standard is observed in federal states that have adopted it. courts of the more than forty or test” for performing this evaluation, but offered guidance in but offered this evaluation, for performing or test” the theory or tech- considerations: (1) whether the form of four has been) tested”; the expert relies “can be (and nique on which and publica- been subjected to peer review (2) whether it “has associated with or potential rate of error” tion”; (3) “the known theory or tech- and (4) whether the the theory or technique; “relevant scientific “generally accept[ed]” in the nique has been community.” reliability inquiry to be undertaken as “a flexible one,” the as “a flexible be undertaken inquiry to reliability Daubert masonlec.org/site/rte_uploads/files/Daubert%20Report%5B1%5D.pdf [https:// masonlec.org/site/rte_uploads/files/Daubert%20Report%5B1%5D.pdf [https:// perma.cc/YXK6-LSEP] (analyzing the effect of function when ruling on so-called “Daubert motions” and deter- function when ruling on so-called testimony meets the standard of mine whether proffered expert reliability. Even in states that have not adopted the Daubert standard, Even in states that have not admissibility of expert testimony pretrial proceedings on the have become commonplace. \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 12 540 25-MAR-21 9:54 standard, a court is to assess whether “the whether is to assess a court standard, the Daubert under - is scientifi testimony the underlying methodology or reasoning reasoning or methodology “whether that as well as cally valid,” issue.” to the facts in can be applied properly do not prescribe the point in time at which these hear- Daubert do not prescribe the point in typically take the form of an in ings are to take place, they has been completed (i.e., limine motion after expert discovery with a motion for summary shortly before or in conjunction or even during trial, before the judgment, in the lead-up to trial) to the stand to testify. challenged expert is called ed. 2020) (“Thus far, more than 40 states have adopted 42705-crn_106-3 Sheet No. 9 Side B No. 9 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 10 Side A 03/25/2021 10:13:22 R R 58 57 541 541 Sample Daubert While see also . P. 166(i), 60 IV . P. § 2034.260 . P. § IV . R. C C EX The Changing Role of 1, 3 (2006) (conclud- ODE 385, 388 (2001) (“[C]ase . C CTION AL . A IV , Frye, Again: The Past, Present, URIMETRICS , 5 C Daubert jurisprudence.”). While stan- , 41 J ., https://courts.delaware.gov/chancery/ and default requirements are and default TS 59 . C EL , D note 35 (“[E]videntiary rules and standards . . . deter- note 35 (“[E]videntiary rules and standards . P. 26(a)(2) (requiring disclosure of the expert’s iden- . P. 26(a)(2) (requiring disclosure of the . P. 16(b)(3)(B) (providing that a scheduling order may LITIGATION SCIENCE SCIENCE LITIGATION IV IV . R. C . R. C ED ED Weisgram v. Marley Co., 528 U.S. 440, 455–56Weisgram v. Marley Co., 528 U.S. 440, (2000) (holding that challenges and result in a summary judgment and encourag[e] Daubert challenges and result in a summary has “made it more difficult for supra note 56, at S64 (arguing that Daubert See, e.g., F See, e.g., F See See id.; Pardo, supra The procedural mechanics of expert discovery differ from mechanics of expert discovery The procedural The admission or exclusion of expert evidence can alter the alter can evidence of expert or exclusion admission The M K 59 60 57 58 Berger, Ctr. for State Courts, plaintiffs to litigate successfully”); Nat’l Judges in the Admissibility of Expert Evidence when an expert is excluded shortly before trial, plaintiffs do not have a right to when an expert is excluded shortly before expert). salvage their case by substituting a new to trial in the first place, which verdicts mine important issues such as who gets convictions will be overturned.”); will be allowed to stand, and which the details thus vary from court to court and from case to case, the details thus vary from court standard “appears to . . . reduc[e] the number of cases that ing that the Daubert standard “appears to . . . reduc[e] the ‘survive’ is not successful”). the defense to settle if their challenge 192.3(e), 195 (providing detailed description of disclosure requirements relating to 192.3(e), 195 (providing detailed description of disclosure requirements relating expert discovery). ‘general acceptance’ test, judges are citing and analyzing the [older] Frye ‘general acceptance’ test, judges are citing A ruling of inadmissibility can alter the outcome of a trial or can alter the outcome A ruling of inadmissibility judgment. the case will survive summary determine whether affects the parties’ expert evidence in play also Further, having that its ex- the greater a party’s confidence settlement power: a position it to testify at trial, the stronger pert will be permitted negotiations. can take in settlement jurisdiction, jurisdiction to Frye and its progeny . . . .”); David E. Bernstein, and Future of the General Acceptance Test \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 13 25-MAR-21 9:54 often supplemented by agreements between the parties or often supplemented by agreements issued by the court. through case-management orders law under Frye is slowly converging with to jurisdiction, I am not aware of any dards of relevance differ from jurisdiction United States where expert testimony is court of general jurisdiction within the admitted without regards to reliability. ] docs/Sample_Expert_Discovery_Stipulation1.pdf [https://perma.cc/R9Y4-VPV7 (sample expert-discovery stipulation modifying statutory disclosure requirements). modify the timing and nature of disclosures and the scope of discovery); Expert Discovery Stipulation tity and expert’s written report at a specified time); C (requiring exchange of rudimentary information about an expert and the expert’s (requiring exchange of rudimentary information about an expert and the expert’s expected testimony, but not an exchange of expert reports); T 2021] 2021] case dramatically. of a course lean a plaintiff’s claims When not hyper- analysis, it is of an expert’s on the outcome heavily that opin- of or inadmissibility that the admissibility bole to say a case. make or break ion can of the an element When the case evidence, entirely on expert claim rests plaintiff’s or when the testifying is barred from when the expert crumbles testimony. limits the scope of the expert’s court substantially C Y 42705-crn_106-3 Sheet No. 10 Side A No. 10 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 10 Side B 03/25/2021 10:13:22 R M K C Y See See, e.g., [Vol. 106:529 106:529 [Vol. After the expert has After the First, an expert is re- is an expert First, 62 61 . P. 166(i), 192.3(e), 194.2(f), 195.1 IV . R. C EX . P. 26(a)(2)(B)(i)–(ii) that an expert report (providing If any opinions are based on a study or If any opinions IV CORNELL CORNELL LAW REVIEW 64 . R. C ED and other standards of reliability rely on an im- and other standards of reliability Expert reports generally contain all of the opinions generally contain all of the Expert reports 63 See, e.g., F The following characterization of the expert-discovery process is based on The following characterization of the or court-imposed deadlines, Expert disclosures can happen by rule-based exchange of expert reports. Some jurisdictions do not require the note 60. Daubert 64 61 62 63 the author’s experience in eight years of litigation practice. the author’s experience in eight years parties in discovery stipulations, or in by deadlines agreed upon between the response to interrogatories. Beck v. Hirchag, No. G041955, 2011 Cal. App. Unpub. LEXIS 2649, at *15 (Cal. Beck v. Hirchag, No. G041955, 2011 that California procedure does not require Ct. App. Apr. 11, 2011) (acknowledging and “practice guides recommend attorneys an expert to prepare an expert report ‘instruct their experts not to prepare a formal report’”). It is not uncommon in these jurisdictions for parties to agree to exchange expert reports regardless. supra completed her analysis, an expert report is served on opposing is served an expert report her analysis, completed parties. tained by a party, and starts work on an analysis or project. an analysis or starts work on a party, and tained by At retention of the party’s discovery period, during the some point witness at expert as a to call the and its intention the expert opposing parties. disclosed to trial are analysis conducted by the expert, the report typically includes by the expert, the report analysis conducted of the basis for her that the expert considers part those findings about the expert’s with enough information opinions, along steps that were a reader to understand the project to allow at the reported results.taken to arrive Next, opposing parties to depose the expert.have an opportunity Finally, if a party of (some of) the expert’s chooses to challenge the admissibility a motion to exclude or limit the opinions into evidence, it files expert’s testimony. and, typi- Following briefing on the motion the court makes a determination cally, an evidentiary hearing, as well as the reliability, rele- on the expert’s qualifications, expert’s proposed testimony, and vance, and helpfulness of the limited, or excluded. the testimony is either admitted, \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 14 542 25-MAR-21 9:54 the road to presentation of expert-generated scientific evidence evidence scientific expert-generated of presentation road to the as follows. proceeds typically at trial the expert intends to present at trial, along with all the bases to present at trial, along the expert intends for those opinions. (requiring disclosure of an expert’s opinions, the expert’s bases for those opinions, (requiring disclosure of an expert’s opinions, the expert’s bases for those opinions, and the facts that “relate to or form the basis of” those opinions and limiting rule). expert discovery beyond the substance explicitly permitted or required by must contain “a complete statement of all opinions the [expert] witness will ex- must contain “a complete statement of all opinions the [expert] witness will - press and the basis and reasons for them”); N.Y. C.P.L.R. 3101(a)(4)(d)(1) (McKin facts ney 2014) (requiring the proffering party to disclose “the substance of the of the and opinions on which each expert is expected to testify . . . and a summary grounds for each expert’s opinion”); T plicit assumption that, when the time comes to assess the reli- plicit assumption that, when information necessary to make ability of scientific evidence, the to the court—i.e.,that assessment will be available an assump- 42705-crn_106-3 Sheet No. 10 Side B No. 10 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 11 Side A 03/25/2021 10:13:22 R 543 543 CIENCES S OCIAL S note 15, at 512. supra II RISIS IN THE C . P. 26(a)(2)(B)(i)–(ii) that an expert report (providing LITIGATION SCIENCE SCIENCE LITIGATION IV Experts typically do not disclose their hy- not disclose their typically do Experts - report omits experimental re Ordinarily, the 65 . R. C NOWLEDGE 66 ED K HE T In less than a decade, the field had (a) learned that a In less than a decade, the field 67 See infra subparts II.A and III.A. See, e.g., F Nelson, Simmons & Simonsohn, In 2018, psychologists Nelson, Simmons, and Simonsohn In 2018, psychologists Nelson, As will be discussed in Parts II and III of this Article, the As will be discussed in Parts M K 65 66 67 noted that “[i]f a team of research psychologists were to emerge noted that “[i]f a team of research they would not recognize their today from a 7-year hibernation, field.” \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 15 25-MAR-21 9:54 potheses and methodology until they submit a report in which submit a report until they and methodology potheses generally study, of their completed the results they present of the expert-discovery period.toward the end And while the taken to arrive at the steps that were typical report describes at what point it usually does not describe the reported results, when exactly settled on a set of hypotheses, in time the expert was designed, and that led to the findings the research protocol or supplemented the protocol was modified to what extent along the way. sults that do not support the expert’s opinions, as well as re- support the expert’s opinions, sults that do not the results that are did not end up leading to search steps that being reported. expert report does not per- information included in the typical the expert’s findings meet the mit the court to assess whether required standard of reliability. of unreli- There is a dimension is assessed based exclu- ability that is missed when reliability testimony, and by the time the sively on expert reports and is up for adjudication, it is too reliability of the expert’s work late to recover this dimension. majority of its research results could not and should not be majority of its research results that this unreliability could be relied upon; (b) figured out used research practices; (c) blamed on a number of commonly and devised practices and standards to improve the quality reproducibility of research results; and (d) had these updated practices and standards adopted by a large and growing group must include “all opinions the witness will express and the basis and reasons for must include “all opinions the witness will express and the basis and reasons them” and “the facts or data considered by the witness in forming them”). 2021] 2021] - from re be obtained can information necessary that the tion hearing or deposition through or the expert by submitted ports testimony. and its Crisis of the Replication As a discussion assump - make clear, this science will for litigation implications incorrect. tion is C Y 42705-crn_106-3 Sheet No. 11 Side A No. 11 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 11 Side B 03/25/2021 10:13:22 R R R M K 68 C Y but Social TATE OF , 71 S (Nov. 19, HE : T [Vol. 106:529 106:529 [Vol. AUMEISTER TLANTIC 69 , A F. B note 15, at 512 (“Many SYCHOLOGY OY P supra & R OCIAL S INKEL In recognition of psychology’s In recognition J. F DVANCED 70 LI A (Nov. 19, 2018), https://www.nytimes.com/ (Nov. 19, 2018), https://www.nytimes.com/ in Essay: The Experiments Are Fascinating. But Psychology’s Replication Crisis Has a Silver Psychology’s Replication Crisis Has , IMES CORNELL CORNELL LAW REVIEW 72 (“Many have been referring to this period as psychology’s id. (“Many have been referring to this (Feb. 19, 2016), https://www.theatlantic.com/science/archive/ (Feb. 19, 2016), https://www.theatlantic.com/science/archive/ 1, 1 (Eli J. Finkel & Roy F. Baumeister eds., 2d ed. 2019). TLANTIC See infra notes 93–97;note 125 (noting that certain practices see also infra See, e.g., Nelson, Simmons & Simonsohn, See infra notes 93–97. See id. See, e.g., CIENCE This Part describes the events that heralded the Replica- This Part describes the events While the Replication Crisis originated in psychology re- in psychology Crisis originated Replication While the S 70 71 72 68 69 in the past few years, each of these fields has engaged in efforts years, each of these fields has in the past few and the number rigor of its research practices, to shore up the updated standards adopting or recommending of organizations is still growing. Lining, A ‘replication crisis.’”); Paul Bloom, in that are now the subject of renewed interest had been acknowledged previously economics, psychology, and other fields). THE frontrunner position in uncovering and addressing unreliable in uncovering and addressing frontrunner position referring to this psychologists have started research practices, as “Psychology’s Renaissance,” period of reconstruction tion Crisis, the research practices that stood at its center, and tion Crisis, the research practices have drawn from it.the lessons that the social sciences Sub- of the Crisis, how its mag- part II.A presents a brief chronology focus, and how it spread like a nitude gradually came into wildfire from field to field. the “Renais- Subpart II.B describes early Crisis years.sance” period that followed the It reviews the uncovered during the Crisis causes of unreliability that were search, other fields soon followed. other fields soon search, phar- biology, Economics, (or in some discovered each in turn and other fields macology, were that their research methodologies cases rediscovered) triggered alarm type of unreliability that had prone to the same of psychology. bells in the field - 2018), https://www.theatlantic.com/science/archive/2018/11/psychologys ] (“[I]t has become replication-crisis-real/576223/ [https://perma.cc/J75C-HPJ6 painfully clear that psychology is facing a ‘reproducibility crisis.’”). The crisis of confidence that shook the field in the early 2010s 2010s the early in the field shook that confidence crisis of The practices is of research this overhaul precipitated and that Crisis.” as the “Replication referred to commonly \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 16 544 25-MAR-21 9:54 of researchers, research centers, journals, and organizations. journals, centers, research of researchers, 2018/11/19/science/science-research-fraud-reproducibility.html [https:// 2018/11/19/science/science-research-fraud-reproducibility.html mired in a ‘replication’ crisis.”); Ed Yong, perma.cc/R9M8-AA7D] (“Science is out of Excuses Psychology’s Replication Crisis Is Running 2016/02/psychology-studies-replicate/468537 [https://perma.cc/X6LW-CKBV] 2016/02/psychology-studies-replicate/468537 is an opportunity for the field of psychology (arguing that the “Replication Crisis” to “lead the way”); Andrew Gelman, Nobody Can Repeat Them, N.Y. T Psychology: Crisis and Renaissance have been referring to this period as psychology’s ‘replication crisis.’ This makes no sense. follows a long drought a water crisis. . . . We do not call the rain that This is psychology’s renaissance.”); E 42705-crn_106-3 Sheet No. 11 Side B No. 11 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 12 Side A 03/25/2021 10:13:22 R R R R R 80 77 545 545 (Dec. EM and in note 5, at e29081, at 81 J. B NE supra ARYL S O As one might O 75 , 7 PL 551, 556–59 (2012). was a well-respected was a well-respected Researchers who re- note 15, at 513 (The Journal of 73 A Bayes Factor Meta-Analysis of 78 ETHODS . 682, 683 (2011). supra . M EV . R note 15, at 513–14 (describing the Doyen, ULL SYCHOL is “[s]ocial psychology’s most prestigious is “[s]ocial psychology’s most prestigious S ONE e33423, at 3 (2012) (reporting three B O supra , 17 P Back from the Future: Parapsychology and the Bem Back from the Future: Parapsychology 31, 31 (2011) (describing Bem’s study as particularly 31, 31 (2011) (describing Bem’s study LITIGATION SCIENCE SCIENCE LITIGATION or attempted to replicate Bem’s work or attempted to replicate Bem’s note 74, at 33. 79 note 1, at 407. SYCHONOMIC NQUIRER I supra ephane Doyen, Olivier Klein, Cora-Lise Pichon & Axel Cleeremans, KEPTICAL Wagenmakers, Wetzels, Borsboom & van der Maas, Wagenmakers, Wetzels, Borsboom & St´ Alcock, Nelson, Simmons & Simonsohn, The Ironic Effect of Significant Results on the Credi- Ulrich Schimmack, The Ironic Effect of Significant Results Jeffrey N. Rouder & Richard D. Morey, Jeffrey N. Rouder & Richard D. Morey, improper modifications to experimental procedures, improper modifications The journal in which he published his 2011 study is a which he published his 2011 The journal in 76 Replication Crisis Replication See See, e.g., Stuart J. Ritchie, Richard Wiseman & Christopher C. French, See See Bem, supra See James E. Alcock, See See See See 74 Daryl J. Bem, the social psychologist who demonstrated who demonstrated psychologist Bem, the social Daryl J. A. M K 79 80 81 73 74 75 76 77 78 Personality and Social Psychology journal.”). newsworthy because of the “academic stature of its author”); D newsworthy because of the “academic Klein, Pichon, and Cleeremans study as one of the events that precipitated the Klein, Pichon, and Cleeremans study as one of the events that precipitated Replication Crisis). 5–6 attempt to replicate a celebrated age-priming study); (2012) (reporting a failed Nelson, Simmons & Simonsohn, Failing the Future: Three Unsuccessful Attempts to Replicate Bem’s ‘Retroactive Failing the Future: Three Unsuccessful Attempts to Replicate Bem’s ‘Retroactive Facilitation of Recall’ Effect, 7 PL Affair, 35 S bility of Multiple-Study Articles expect to happen with a study purporting to show that the with a study purporting expect to happen met with skepti- the past, Bem’s study was future can change cism. data underlying the reported Researchers delved into the a number of flaws, immediately identified results and almost statistical signifi- in Bem’s calculation of such as defects cance, researcher with a decades-long career of research and publica- a decades-long career of research researcher with tion. the same year, a team of psychologists published a paper dem- the same year, a team of psychologists of psychology researchers were onstrating that large numbers 25, 2014), https://dbem.org [https://perma.cc/M7GE-CN68]. design. and incorrect experimental found no support for his findings. same time, an- Around the study failed to replicate, other high-profile psychology \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 17 25-MAR-21 9:54 peer-reviewed, high-impact journal in his field. peer-reviewed, failed replication attempts). 2021] 2021] and proposed been that have the reforms some of and period in response. implemented [f]uture,” the ] “[f]eel[ that people can analyzed the data Bem’s ESP Claim, 18 P 428. Behavioral Priming: It’s All in the Mind, but Whose Mind? C Y 42705-crn_106-3 Sheet No. 12 Side A No. 12 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 12 Side B 03/25/2021 10:13:22 M K C Y , 23 and 82 85 In one In one 83 Measuring the Estimating the [Vol. 106:529 106:529 [Vol. 1037-b, 1037-b (2016) Comment on “Estimating for example, Daniel T. Gil- , (Aug. 28, 2015), https://sci- CIENCE (Mar. 6, 2016), https://daniel- CIENCE ”, 351 S , https://pubpeer.com/publications/ TATISTICIAN Supplementary Material for EER P Only 39 of the 100 studies replicated Only 39 of the Comment on “Estimating the Reproducibility of UB 86 , 20% S P (Oct. 5, 2012), http://www.decisionsciencenews.com/ (Oct. 5, 2012), http://www.decisionsciencenews.com/ ” , CORNELL CORNELL LAW REVIEW EWS aac4716-1, aac4716-1 (2015). .N - possibility that the low replica To rule out the CI 87 S a “Many Labs 2” project was designed in a manner a “Many Labs 2” project was CIENCE . 524, 524 (2012). 88 Open Sci. Collaboration, Leslie K. John, George Loewenstein & Drazen Prelec, Leslie K. John, George Loewenstein CI ECISION Teams attempting to replicate the original studies to replicate the original Teams attempting . S For criticism of early replication projects, see Estimating the Reproducibility of Psychological Open Sci. Collaboration, Estimating the Reproducibility of Psychological See the replication attempts). See id. (reporting that 69 original teams endorsed See id. at 17. Note that it was not expected that all 100 studies would yield See Surrounding Social Priming Re- See Kahneman on the Storm of Doubts 84 See id. at 4. Prominent researchers issued calls for large-scale replica- for large-scale issued calls researchers Prominent 88 84 85 86 87 82 83 SYCHOL Science, 349 S consistent results on replication. original Given the variability reported in the that approximately 89 studies would repli- studies, the a priori expectation was cate. search, D / ence.sciencemag.org/content/sci/suppl/2015/08/26/349.6251.aac4716.DC1 ] (reporting that 89 teams were able Aarts-SM.pdf [https://perma.cc/89GD-CGGP to obtain original materials). lakens.blogspot.com/2016/03/the-statistical-conclusions-in-gilbert.html lakens.blogspot.com/2016/03/the-statistical-conclusions-in-gilbert.html [https://perma.cc/B94V-B62V]. large-scale project, researchers selected 100 papers that had 100 papers selected project, researchers large-scale of psychology and and relied upon in the field been widely cited by 100 different replication efforts coordinated systematic labs. (criticizing sample size, experimental setup, and analysis employed). For re- sponses to these critiques, see Psychological Science in most cases obtained the original authors’ endorsement of obtained the original authors’ in most cases attempt. the replication successfully. tion efforts, to assess how widespread the use of unreliable the use of how widespread to assess tion efforts, its use. how consequential was and methodology \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 18 546 25-MAR-21 9:54 in the habit of using what they termed “questionable research research “questionable they termed what of using habit in the findings. of their the reliability undermine that practices” tion rate had been caused by inartfully conducted replication caused by inartfully conducted tion rate had been attempts, aimed to reproduce the original testing conditions as closely as the original testing conditions aimed to reproduce possible. (and in almost all cases To this end, they requested authors, materials from the original received) the original - 2012/10/05/kahneman-on-the-storm-of-doubts-surrounding-social-priming-re ] (quoting email by Daniel Kahneman, search/ [https://perma.cc/M3GJ-8YHB he warns of “a train wreck looming” and dated September 26, 2012, in which proposes a systematic replication project). - 120FE2AC75B4C873787611246291A8 [https://perma.cc/BK4G-4B92] (last vis Lakens, The Statistical Conclusions in Gilbert et al. ited May 27, 2020); Dani¨el (2016) Are Completely Invalid Reproducibility of Psychological Science 4, S Reproducibility of Psychological Science bert, Gary King, Stephen Pettigrew & Timothy D. Wilson, the Reproducibility of Psychological Science Prevalence of Questionable Research Practices with Incentives for Truth Telling Prevalence of Questionable Research Practices P 42705-crn_106-3 Sheet No. 12 Side B No. 12 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 13 Side A 03/25/2021 10:13:22 R R R R . 96 SY- , 1 P bi- 547 547 115, False- in 93 SYCHOL , see also P note 15 (exam- 1 (Fin. & Econ. EUROSCIENCE RACTICES 2801, 2801 (2017) : N supra & P neuro-imaging, 95 EVIEWS R ETHODS . 1359, 1359 (2011) (demon- The emphasis on pub- The emphasis As part of this project, this project, part of As CI M HARMACOLOGY 98 89 . S ATURE Is Economics Research Replicable? note 84 (describing two molecular . J. P DVANCES RIT , 18 N - Updating the Guidelines for Data Trans SYCHOL supra Scanning the Horizon: Towards Transparent Scanning the Horizon: Towards Transparent , 1 A 3, 3 (Scott O. Lilienfeld & Irwin D. Waldman Many Labs 2: Investigating Variation in Repli- Many Labs 2: Investigating , 22 P , 174 B pharmacology, note 14 (successfully replicating 11 of 18 eco- note 14 (successfully replicating 11 of o et al., A Manifesto for Reproducible Science 94 531, 532 (2012) (reporting that the Amgen team 531, 532 (2012) (reporting that the “Ironically enough,” one commentator one commentator enough,” “Ironically CRUTINY LITIGATION SCIENCE SCIENCE LITIGATION S 91 92 ATURE A sizable methodological meta-field sprang meta-field sprang A sizable methodological Only 15 of the 28 selected studies were rated studies were of the 28 selected Only 15 NDER 97 . 1, 1 (2017) (“85% of biomedical research efforts are . 1, 1 (2017) (“85% of biomedical 90 note 69. U EHAV supra CIENCE . B Camerer et al., supra Richard A. Klein et al., Richard A. Klein et Christopher H. George et al., Russell A. Poldrack et al., S UM H See, e.g., Head, Holman, Lanfear, Kahn & Jennions, See, e.g., Joseph P. Simmons, Leif D. Nelson & Uri Simonsohn, Raise Standards for Preclinical See, e.g., C. Glenn Begley & Lee M. Ellis, See See See See id. See id. at 477. Yong, See Similar failures to replicate were found in economics, Similar failures see also Marcus R. Munaf` . 443, 446 (2018). M K 97 98 94 95 96 89 90 91 92 93 ATURE CI ology and biomedicine, succeeded in confirming results for only 6 of 53 hematology and oncology stud- succeeded in confirming results for only ies); N Sixty Published Papers from Thirteen Journals Say “Usually Not” Sixty Published Papers from Thirteen No. 2015-083, 2015), https://pa- Discussion Series, Working Paper [https://perma.cc/UN2F- pers.ssrn.com/sol3/papers.cfm?abstract_id=2669564 59 economics studies). 9PWW] (successfully replicating 29 of nomics studies); Andrew C. Chang & Phillip Li, nomics studies); Andrew C. Chang & Phillip Cancer Research, 483 N ining published study results across disciplines including cognitive sciences, in- ining published study results across disciplines including cognitive sciences, formation and computing sciences, and medical and health sciences to conclude that reproducibility concerns are “rife”). (reporting mixed results of a series of replication attempts in pharmacology). (reporting mixed results of a series of wasted . . . .”); Open Sci. Collaboration, wasted . . . .”); Open Sci. Collaboration, a replication rate of 11% and 25%, biology replication studies that had respectively). 36 labs in 36 countries collaborated to rerun a set of 28 well- set of rerun a to collaborated 36 countries in 36 labs studies. known 115 (2017) (reporting that neuroimaging has replication problems similar to those 115 (2017) (reporting that neuroimaging found in other scientific fields). noted, “it seems that one of the most reliable findings in psy- reliable findings one of the most seems that noted, “it can be suc- studies of psychological is that only half chology repeated.” cessfully and other fields. \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 19 25-MAR-21 9:54 CHOLOGICAL up that aimed to make sense of these dismal replication rates, to make sense of these dismal up that aimed evidently risked on the research practices that focusing both and institu- results and the cultural producing irreproducible that led to their use. tional incentives to have replicated. to have S strating how common research tools can generate unreliable findings); 2021] 2021] criticisms. early address to intended Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant and Reproducible Neuroimaging Research cability Across Samples and Settings parency in the British Journal of Pharmacology—Data Sharing and the Use of Scatter Plots Instead of Bar Charts Open Sci. Collaboration, Maximizing the Reproducibility of Your Research C Y 42705-crn_106-3 Sheet No. 13 Side A No. 13 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 13 Side B 03/25/2021 10:13:22 R R R R R R R M K p- C Y . 615, CI . 0696, o et al., When I’m ED . S supra note note 15, at The Rules of SM 104 O SYCHOL supra [Vol. 106:529 106:529 [Vol. . 543, 547 (2012) P CI ON . S , 2 PL . This put pressure This put ERSP 100 SYCHOL note 98, at 1359–60 (using note 82 (“[Q]uestionable prac- P , 7 P ON note 15, at 515 (arguing that . supra note 98, at 617. supra Passed on from generation to Passed on from ERSP supra supra 103 105 , 7 P Scientific Utopia: II. Restructuring Incentives CORNELL CORNELL LAW REVIEW As early as 2011, Joseph P. Simmons and co- Joseph P. Simmons and As early as 2011, p-hacking is “widespread throughout science”); John P.A. Ioan- 99 101 Simmons, Nelson & Simonsohn, Subsequent work confirmed that these common prac- confirmed that these common Subsequent work See generally Head, Holman, Lanfear, Kahn & Jennions, See, e.g., Nosek, Spies & Motyl, a p < 0.05 has become a de See id. (“[T]he nominal false-positive rate of See id. See See, e.g., John, Loewenstein & Prelec, M. Wicherts, See id.; Marjan Bakker, Annette van Dijk & Jelte Why Most Published Research Findings Are False note 94 (identifying apophenia, confirmation bias, and hindsight bias as 102 These practices included: Statistical significance has long been the holy grail for re- the holy grail has long been significance Statistical 99 105 100 101 102 103 104 the Game Called Psychological Science facto criterion for publishing.”). norm.”). tices may constitute the prevailing research 0697–98 flexibility in design, number of tested relationships, (2005) (arguing that and sample size all contribute to unreliable research outcomes); Munaf` supra (approaches involving multiple small, underpowered studies are “the optimal (approaches involving multiple small, significant results where there is no true strategy” for manufacturing statistically effect); Nelson, Simmons & Simonsohn, hacking is “the only honest and practical way to consistently get underpowered studies to be statistically significant” (emphasis omitted)). searchers aiming to publish their results. aiming to publish searchers many Traditionally, not meet results that did not publish journals would scientific significance. levels of statistical threshold on researchers to generate results that would meet those re- would meet results that to generate on researchers quirements. 616 (2012) (arguing for institutional change to improve reliability of social science 616 (2012) (arguing for institutional change research). ed., 2017) (suggesting best practices to improve reliability of research); Brian A. ed., 2017) (suggesting best practices Nosek, Jeffrey R. Spies & Matt Motyl, and Practices to Promote Truth over Publishability \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 20 548 25-MAR-21 9:54 lication of statistically significant findings was identified as a as identified was findings significant statistically of lication culprit. major generation in social science labs and the labs of adjacent disci- science labs and the labs generation in social the flexibility they endowed researchers with plines, they had results from limited data, gener- needed to squeeze maximum where there should not ate statistically significant conclusions wittingly or unwittingly, achieve have been any, and thereby, not be relied upon. and publish results that could authors demonstrated how “unacceptably easy” it is to find how “unacceptably authors demonstrated whether evidence for any hypothesis, statistically significant analytical meth- through the use of everyday true or false, ods. tices, employed in pursuit of statistical significance, lay at the in pursuit of statistical significance, tices, employed Crisis. heart of the Replication standard research methodology to prove that listening to the Beatles’ standard research methodology to prove age). Sixty-Four shaved a year and a half off participants’ 1 (concluding that nidis, contributors to flawed data analysis); Simmons, Nelson & Simonsohn, 98 (demonstrating how “unacceptably easy it is to accumulate (and report) statis- 98 (demonstrating how “unacceptably easy it is to accumulate (and report) 42705-crn_106-3 Sheet No. 13 Side B No. 13 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 14 Side A 03/25/2021 10:13:22 R R p- 107 , 31 549 549 Statistical Practice seven 108 See, e.g., Bakker, van Dijk & Practices two through 106 note 94, at 2. LITIGATION SCIENCE SCIENCE LITIGATION P-values are frequently misunderstood, even by scientists 337, 340–45 (2016). o et al., supra note 104; Gelman, supra note 69. Munaf` supra failing to commit to a hypothesis before the start of the start of the before a hypothesis to to commit failing - reverse-engi free to the researcher leaving instead study, the data; and analyzing after collecting neer a hypothesis protocol before overall research commit to an failing to researcher free leaving the of the study, instead the start during or of steps sometime the exact sequence to choose collection; after data the to a data-gathering protocol before failing to commit free to instead leaving the researcher start of the study, deciding into account incoming data in examine and take data, or data collection, collect additional whether to stop the way data are collected; make changes to up of to a protocol for the cleaning failing to commit data—includingof outlier data criteria for the removal missing or corrupted data—leavingand for handling the identify- try different approaches until researcher free to in a desirable outcome; ing one that results data—a used to analyze the tweaking the methodology potentially infinite number of times—until it results in a desirable outcome; correlative analyses performing regressions and other of variables and varia- with a potentially infinite number outcome; and tions, until hitting on a desirable and reporting only the testing a wide variety of variables ones that show a desirable outcome. PIDEMIOLOGY p-value expresses as a number between 0 and 1 the likelihood of finding -hacking is the intentional or unintentional use of “data-contingent anal- P-hacking is the intentional or unintentional A See P-Values, Confidence Intervals, and Power: A Guide to Misinterpretations Practice one is referred to as “hypothesizing after the re- Practice one is referred to as 1. 2. 3. 4. 5. 6. 7. . J. E M K UR 107 108 106 ysis decisions,” typically aimed at manufacturing statistically significant results. ysis decisions,” typically aimed at manufacturing Id. at 3. in the observed outcome (or an outcome farther removed from the null hypothesis) the event that the null hypothesis is true. In other words, it expresses the likeli- pure hood that there is no effect and that the measured outcome came about by chance. The lower the p-value, the stronger support the measurement lends to the tested hypothesis. who use them. twenty-five common misconceptions of For an overview of values and other measures of confidence, see Sander Greenland et al., Tests, E .” sults are known” or “HARKing Wicherts, because they aim to create a p-value—thebecause they aim to create most commonly significance—inused indicator of statistical a range that will be published. allow the research results to hypothesis”).tically significant evidence for a false is not The list provided here intended to be exhaustive. in- Other practices criticized for impairing reliability plain fraud. clude the use of small samples and \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 21 25-MAR-21 9:54 seven are forms of a practice now known as “p-hacking,” seven are forms of a practice 2021] 2021] C Y 42705-crn_106-3 Sheet No. 14 Side A No. 14 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 14 Side B 03/25/2021 10:13:22 R R R R R R R R M K C Y 111 by See, 112 note 15, at 109 Id. at 1361. supra note 98 When used supra [Vol. 106:529 106:529 [Vol. In this Article I In this 113 false positive . . . .”). 110 note 15, at 518 (“P-hacking is a note 98 (emphasis omitted). supra o et al., supra note 94 (arguing that 114 supra To understand why this is the case, To understand why this is the 115 CORNELL CORNELL LAW REVIEW o et al., supra note 94, at 3. note 82 (terming methodologies capitalizing on analytical flexibility note 82 (terming methodologies capitalizing Munaf` Nelson, Simmons & Simonsohn, supra This flexibility has also been termed “researcher degrees of freedom.” This flexibility has also been termed “researcher For example, the mere flexibility to choose to add ten data points to an See id. at 1359 (“Perhaps the most costly error is a See id. at 1361 (finding that mere flexibility to add ten data points com- See See See generally Head, Holman, Lanfear, Kahn & Jennions, It is worth noting that this type of flexibility has been It is worth noting that this What these practices have in common is that they impart is that they have in common practices What these 111 112 113 114 115 109 110 initially collected data set raises the false-positive rate by 50 percent. a re- bined with a choice of variables and control variables virtually guarantees searcher statistically significant support for a hypothesis, even if the hypothesis being tested is false). opportunistically, Analytical Flexibility virtually guarantees a Analytical Flexibility virtually opportunistically, (and therefore publishable) researcher statistically significant evidence for a false hypothesis. use the term “Analytical Flexibility” to refer to a researcher’s to refer to a Flexibility” term “Analytical use the up a re- completely make or even freedom to adapt range of underway. while the research is already search protocol shown to be harmful even in the hands of an honest, well- shown to be harmful even in intentioned researcher. an undue level of flexibility on a researcher. level of flexibility an undue Even small measures of Analytical Flexibility, such as the abil- of Analytical Flexibility, Even small measures of different vari- more data points or the use ity to add a few variables, can have large consequences ables and control \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 22 550 25-MAR-21 9:54 has been termed “publication bias” or, when it causes entire entire causes when it or, bias” “publication been termed has away in stored and unpublished to remain projects research .” problem the “file drawer file drawers, metaphorical raising significantly the false-positives rate: the likelihood of the false-positives rate: raising significantly significant, but is that shows up as statistically finding a result to be repli- in the sense that it is unlikely not a “true” result, were to be repeated. cated if the experiment imagine such a researcher setting to work analyzing a set of imagine such a researcher setting that she, in good faith, be- data, choosing an initial approach approach.lieves to be the most suitable researcher’s first If the publishable result, even a well- choice does not result in a e.g., Simmons, Nelson & Simonsohn, pervasive problem precisely because researchers usually do not realize that they pervasive problem precisely because researchers usually do not realize that are doing it or appreciate that what they are doing is consequential.” (citation omitted)). -hacking is widespread and harmful); John, Lowenstein & 2, 8 (arguing that p-hacking is widespread and harmful); Prelec, (“[F]lexibility in data collection, analysis, and reporting dramatically increases (“[F]lexibility in data collection, analysis, actual false-positive rates.”). research efforts are “wasted” when they do not yield reliable results); Ioannidis, research efforts are “wasted” when they supra note 105, at 0696–98 in designs, definitions, (“The greater the flexibility scientific field, the less likely the research outcomes, and analytical modes in a Nelson & Simonsohn, findings are to be true.”); Simmons, “questionable research practices”); Munaf` 42705-crn_106-3 Sheet No. 14 Side B No. 14 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 15 Side A 03/25/2021 10:13:22 R R R R R R See 119 All 551 551 120 117 note 15 (analyzing supra note 15, at 512 (finding that note 98, at 1362. In Scenario B, the re- 122 supra supra p-hacking is rife”). supra note 105, at 0699 (“Most research p-values are not. (Apr. 6, 2011), https://xkcd.com/882 [https:// And all of these practices contribute contribute of these practices And all In Scenario A, the researcher has a In Scenario A, the researcher If any of these methods renders a result result a renders methods of these If any XKCD 118 LITIGATION SCIENCE SCIENCE LITIGATION 121 116 o et al., supra note 94, at 3; Simmons, Nelson & Simonsohn, -hacking is “something that benevolent researchers engage in (P-hacking is “something that benevolent note 98, at 1362; Ioannidis, Munaf` Nelson, Simmons & Simonsohn, Head, Holman, Lanfear, Kahn & Jennions, Head, Holman, Lanfear, Kahn & Jennions, Simmons, Nelson & Simonsohn, p-value below 0.05 indicates a probability of less than 5 percent that the See This example is adapted from Randall Munroe’s xkcd web comic. A See id. See See See note 98, at 1362–63. supra The practices listed above have another important feature above have another important The practices listed M K 120 121 122 116 117 118 119 findings are false for most research designs and for most fields.”) (capitalization findings are false for most research designs and for most fields.”) (capitalization omitted). to unreliability (i.e., irreproducibility) of research results. (i.e., irreproducibility) of research to unreliability hunch that green jelly beans cause acne.hunch that green jelly beans He performs a study finds a statistically significant using only green jelly beans and between consumption of green positive correlation (p<0.05) acne. jelly beans and incidence of perma.cc/PY7N-3QTU]. result came about by pure chance. By convention in many fields of science, results with associated p-values smaller than 0.05 are considered statistically significant, while results with larger while trying to understand their otherwise imperfect results.”). while trying to understand their otherwise analysis that make it too ] on methods of data collection and “rely[ researchers results”); Simmons, Nelson & Simon- easy to publish false-positive, nonreplicable sohn, published study results to conclude that “ published study results to conclude that of these practices have traditionally been exceedingly common been exceedingly traditionally practices have of these sciences. in the social in common: unless a researcher reporting research results is a researcher reporting in common: unless these recording and disclosure requirements, held to stringent to be invisible.practices tend Analytical Flexibility that When neither explicitly eliminated nor was present in the study is no way to assess the extent to documented, there is typically contaminated or even invalidated which its presence may have the research results. two scenarios Consider as an example the link between acne and jelly involving a researcher studying bean consumption. While a research journey can be unpredictable, and adjust- journey can be unpredictable, While a research that con- necessary, it is now understood ments are sometimes possible is Flexibility to the extent straining Analytical research. creation of reliable, reproducible essential to the \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 23 25-MAR-21 9:54 searcher performs 20 separate studies, each examining the searcher performs 20 separate consumption of a specific color relationship between acne and to statistically significant, she might be tempted to be tempted to she might significant, to statistically that is close more data gathering a few time after analysis a second run the the earlier results. points confirm to see if these points, 2021] 2021] of alternative number try a likely to is researcher intentioned methods. statistical supra Randall Munroe, Significant, C Y 42705-crn_106-3 Sheet No. 15 Side A No. 15 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 15 Side B 03/25/2021 10:13:22 R R R M K C Y p-hacking. [Vol. 106:529 106:529 [Vol. The replication 126 note 15, at 514 & n.4 (“Method- supra none of the 20 types of jelly beans are In both scenarios, the researcher scenarios, the In both Conservative Tests Under Satisficing Models of 123 e0149590, at 1 (2016) (collecting sources indicating 124 But many researchers did not take the But many researchers did not NE CORNELL CORNELL LAW REVIEW 125 S O O See infra note 180. Nelson, Simmons & Simonsohn, -value of 0.05 indicates a 1 in 20 probability that a result came about by p-value of 0.05 indicates a 1 in 20 probability Reported measures of confidence can sometimes be corrected to account Reported measures of confidence can See, e.g., Ioannidis, supra note 105, at 0696 (raising concern in 2005 that See A The methodological problems raised by the Replication Cri- The methodological problems 124 125 126 123 ologists in other fields had brought up the problem we now know as found with statistical significance to cause acne) is only 36%.found with statistical significance to In other words, beans, regardless of their color, do not even if the null hypothesis is true (jelly alternative hypotheses is more likely than cause acne), a researcher who tests 20 up as statistically significant. not to find at least one result that shows for the presence of flexibility, but only if the presence and nature of the flexibility is reported in detail. “most current published research findings are false”); Justin McCrary, Garret Christensen & Daniele Fanelli, pure chance, assuming the null hypothesis is true.pure chance, assuming the null hypothesis alterna- This means that if 20 expect (on average) one of the results tive false hypotheses are tested, one would to have an associated p-value below 0.05. In fact, the probability of not obtaining at least one result with p < 0.05 (here: that Publication Bias, 11 PL sis were not novel. \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 24 552 25-MAR-21 9:54 of jelly bean.of jelly con- bean jelly between is no link there Even if - statisti find a to is likely researcher the acne, and sumption colors (say, one of the 20 (p<0.05) for result cally significant purely by chance. green), can report with a 95% confidence level that green jelly beans that green confidence level with a 95% can report cause acne. a true level provides A, this confidence In Scenario only a 5% there is of the finding: of the variability measure by chance, and if the result was obtained purely likelihood that likely obtain the repeated Scenario A, he would the researcher same result. Scenario B were to be repeated, If the study in would be ob- is unlikely that the same result conversely, it level overstates B, the reported confidence tained; in Scenario of the result.the reliability the researcher publishes Unless a peer reviewing research protocol he used, full details of the the reported result unable to tell by looking at the findings is Scenario A (a single finding is the result of whether the reported significant result) or Scena- experiment yielding a statistically from a series of experiments rio B (a selectively reported result significant result just by that collectively yielded a statistically whether a repeat experiment chance) and unable to predict the finding (as in Scenario A) or would be likely to reproduce not (as in Scenario B). that publication bias, to various extents, had been recognized in psychology, that publication bias, to various extents, had been recognized in psychology, of economics, medicine, and other fields more than three decades before the start the Replication Crisis). threat of unreliable research methodology seriously until they threat of unreliable research that the very foundations of were confronted with the reality their discipline might be compromised. 42705-crn_106-3 Sheet No. 15 Side B No. 15 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 16 Side A 03/25/2021 10:13:22 R R R R R R 132 130 553 553 While Hiring, 619, 619 131 P-hacking is 128 ATURE , 543 N note 15, at 515 (“ note 98, at 1361. note 104 (stating that approaches (Mar. 15, 2017, 5:49 AM), https:// supra supra supra In short, researchers have am- note 98. The Science ‘Reproducibility Crisis’—and consistently” obtain statistically significant 133 Reproducibility Blues ONVERSATION o, C HE note 98, at 615 (arguing that institutional incentives note 98, at 615 (arguing that institutional , T LITIGATION SCIENCE SCIENCE LITIGATION Peer-reviewed journals typically publish only journals typically publish only Peer-reviewed At all levels of experience, a researcher’s career a researcher’s of experience, At all levels 129 127 Nelson, Simmons & Simonsohn, Nosek, Spies & Motyl, supra Simmons, Nelson & Simonsohn, See See id. at 617 (“[T]he nominal false-positive rate of a p < 0.05 has become a See See See Bakker, van Dijk & Wicherts, See, e.g., Marcus Munaf` See Nosek, Spies & Motyl, supra note 98, at 626 (“[P]ublishing is a central, The Replication Crisis made clear: those levels of statistical Crisis made clear: those levels The Replication It has been suggested that methodological lapses and lapses that methodological been suggested It has M K 129 130 131 132 133 127 128 promotion, tenure, research funding, and the ability to recruit research funding, and the promotion, tenure, a steady stream of collaborators all depend on students and publications. depends on being able to publish novel findings. depends on being - theconversation.com/the-science-reproducibility-crisis-and-what-can-be-done - about-it-74198 [https://perma.cc/DN4B-9ZFT] (arguing that the Replication Cri sis “is actually a publication bias crisis”). (2017) (describing how perverse incentives undermine the scientific method); (2017) (describing how perverse incentives Nosek, Spies & Motyl, supra “inflate the rate of false effects in published science”). “inflate the rate of false effects in published results.). de facto criterion for publishing.”). the only honest and practical way to immediate, and concrete objective for [a researcher’s] career success.”); Ottoline immediate, and concrete objective for Leyser, Danny Kingsley & Jim Grange, What Can Be Done About It a carefully planned study is more likely to yield results of a high study is more likely to yield a carefully planned a scattershot approach degree of reliability and replicability, run on the same dataset in involving a large set of analyses statistically significant result is hopes that one will lead to a meet publication criteria. more likely to yield results that significance are easier to achieve by researchers who (willfully easier to achieve by researchers significance are research. take flexible approaches to or blunderingly) However, perhaps because they did not demonstrate that this was a problem However, perhaps because they did did not propose concrete and practical worth worrying about or because they did not have perceivable consequences solutions to prevent it . . . their concerns reported in their fields.” (citation omitted)). on how research was conducted and Experiments on small samples, moreover, are a dream come Experiments on small samples, for time: smaller samples are both true for a researcher pressed to yield a statistically signifi- more likely than larger samples “true” result and typically involve cant result when there is no less work for the researcher. \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 25 25-MAR-21 9:54 results that meet acceptable levels of statistical significance. acceptable levels of statistical results that meet 2021] 2021] most of the some for doubt: room left little however, projects, robust not as simply were methods research applied commonly to be. had assumed them researchers as many unchecked to proliferate long allowed shortcuts were harmful in danger- were aligned incentives researchers because among ous ways. involving multiple small, underpowered studies described as “the optimal strat- involving multiple small, underpowered studies described as “the optimal C Y 42705-crn_106-3 Sheet No. 16 Side A No. 16 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 16 Side B 03/25/2021 10:13:22 R R R R R R R M K C Y —a 135 see also Jour- 139 [Vol. 106:529 106:529 [Vol. note 15; see also supra Indeed, by now it is e3000246, at 6 (2019); 138 Indeed, science has been science has been Indeed, p-hacking). IOLOGY 136 Open Science Challenges, Benefits and Open Science Challenges, Benefits and S B note 15, at 517 (arguing that failed replica- O 134 note 94, at 4 (“Improving the quality and trans- supra , 17 PL note 96, at 115–16 (explaining that low power note 94, at 2. CORNELL CORNELL LAW REVIEW note 94, at 7 (quoting Richard Feynman as saying, “The first o et al., supra 137 o et al., supra Munaf` Chris Allen & David M. A. Mehler, See See Head, Holman, Lanfear, Kahn & Jennions, note 105 (“The smaller the studies con- See generally Ioannidis, supra note 105 (“The smaller the studies Munaf` to see patterns in random See id. at 1 (defining apophenia as “the tendency o et al., supra In addition to publication pressure, even honest scientists even honest scientists pressure, to publication In addition impediments Crisis, there were few Prior to the Replication 137 138 139 134 135 136 tion attempts and the prevalence of successful, underpowered studies in the tion attempts and the prevalence of successful, underpowered studies scientific press are evidence of widespread Nelson, Simmons & Simonsohn, principle is that you must not fool yourself—and you are the easiest person to fool”). Munaf` parency in the reporting of research is necessary to address” reproducibility parency in the reporting of research is necessary to address” reproducibility problems.). ducted in a scientific field, the less likely the research findings are to be true. . . . ducted in a scientific field, the less likely field, the less likely the research findings The smaller the effect sizes in a scientific are to be true. . . . tested The greater the number of and the lesser the selection of less likely the research findings are to be relationships in a scientific field, the true. . . .analyti- The greater the flexibility in designs, definitions, outcomes, and likely the research findings are to be cal modes in a scientific field, the less true. . . . scien- The greater the financial and other interests and prejudices in a findings are to be true. . . .tific field, the less likely the research a The hotter research findings are to be true.”). scientific field . . . , the less likely the data”). nals were not requiring disclosures of this information and peer nals were not requiring disclosures face a risk of being carried away by enthusiasm. away by of being carried face a risk Enthusiastic been said knowledge have a search for motivated by scientists patterns in noise” . . . to see a “natural tendency to have to the—conscious or unconscious—use and crea- of shortcuts of data.tive exploitation culture that was lacking In a research which methodolog- understanding of the way in awareness and results, au- might contaminate research ical decision making not typically submitted for publication would thors of papers that would help a reader assess include the kind of information Flexibility may have contami- the extent to which Analytical results. nated the reported research egy” for manufacturing statistically significant results where there is no true egy” for manufacturing statistically effect); Poldrack et al., supra reduces the likelihood of finding a true result and raises the likelihood of finding a reduces the likelihood of finding a true false-positive result). \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 26 554 25-MAR-21 9:54 ple incentive to gravitate toward low-effort, high-reward, but but high-reward, low-effort, toward to gravitate incentive ple methodologies. low-reliability -hacking was rife in the social sciences. clear that p-hacking was rife in the social tendency known as “apophenia.” tendency described as “an ongoing race between our inventing ways to ongoing race between our described as “an to avoid fooling and our inventing ways fool ourselves, ourselves.” Tips in Early Career and Beyond 42705-crn_106-3 Sheet No. 16 Side B No. 16 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 17 Side A 03/25/2021 10:13:22 R R R R R R p- see NGI- 12, note 555 555 . E P-hack- Even Even . 160384, 143 NVTL , 115 PNAS CI Issues with 140 S SYCHONOOM , 34 E PEN P E o et al., supra O Students were Students Y ’ OC 141 S note 15, at 515 (“ OYAL Nature agreed that there is a note 15, at 515 (“Researchers see also Marc A. Edwards & supra and a growing concern , 3 R 146 there was a growing aware- supra 145 note 142. (Nov. 16, 2018), https://psyarxiv.com/cfzyx note 94 (stating that “90% of respondents to a supra were not failing.”); Paul E. Smaldino & Richard REPRINTS LITIGATION SCIENCE SCIENCE LITIGATION 7 Easy Steps to Open Science: An Annotated uwell et al., 7 Easy Steps to Open Science: An Annotated P The year 2011 was being described as the was being described as the The year 2011 IV X Indeed, some have quipped that there is a have quipped that there is Indeed, some R 144 A o et al., supra SY 142 The Natural Selection of Bad Science Munaf` . 51, 55 (2017) (describing the tendency of people to see themselves as . 51, 55 (2017) (describing the tendency Nelson, Simmons & Simonsohn, Andrew W. Brown, Kathryn A. Kaiser & David B. Allison, Andrew W. Brown, Kathryn A. Kaiser improper research in spite of improper research because of rather than CI Replication Renaissance Replication Renaissance S See, e.g., Sophia Cr ¨ See, e.g., Eric-Jan Wagenmakers, A Year of Horrors, 27 D See See, e.g., Nelson, Simmons & Simonsohn, See Smaldino & McElreath, See By the mid- to late-2010s, many researchers in the field of late-2010s, many researchers By the mid- to - Leyser, Kingsley & Grange, supra note 128 (“Murmurings of low reproducibil B. M K 144 145 146 141 142 143 140 - Analyti of the dangers of aware generally who were researchers were engag- that they necessarily aware were not cal Flexibility and could discredited that had been practices ing in research of their outcomes. the reliability undermine methodology. absorbing bad research practices as they learned from their they learned practices as bad research absorbing suc- many cases became and in supervisors more experienced cessful 12 (2012) (“[T]he year 2011 can go in the books as a true annus horribilis.”); 12 (2012) (“[T]he year 2011 can go in the books as a true annus horribilis.”); also ity began in 2011—the ‘year of horrors’ for psychology . . . .”). NEERING ing allowed researchers to think, . . . ’most of my studies work; [critics of ing allowed researchers to think, . . people.’”);hacking] must be talking about other Maintaining Scientific Siddhartha Roy, Academic Research in the 21st Century: and Hypercompetition Integrity in a Climate of Perverse Incentives [2017] survey in Nature agreed that there is a ‘reproducibility crisis’”). Data and Analyses: Errors, Underlying Themes, and Potential Solutions Data and Analyses: Errors, Underlying are reporting guidelines, “authors do not 2563, 2567 (2018) (even when there guidelines, nor do peer reviewers demand always report information specified in that the information be reported”). 94, at 1 (“90% of respondents to a recent survey in ‘reproducibility crisis.’”). - [https://perma.cc/5QV4-3S35] (“Most scientists agree that there is a reproduc ibility crisis, at least to some extent.” (citation omitted)); Munaf` at 2 (2016) (arguing that science is a cultural activity that is evolutionary in at 2 (2016) (arguing that science is poor research methods). nature and that powerful incentives propagate more honest than their peers as the “Muhammad Ali effect”). more honest than their peers as the “Muhammad psychology had come to the realization that the field was exper- come to the realization that psychology had iencing a crisis. \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 27 25-MAR-21 9:54 “year of horrors” for psychology; “natural selection of bad science,” as unrigorous practices were of bad science,” as unrigorous “natural selection researcher. generation to generation of passed on from ness of failed replication attempts 2021] 2021] information. for the to ask unlikely were reviewers Reading List 3, P did not learn from experience to increase their sample sizes precisely because did not learn from experience to increase their underpowered studies McElreath, C Y 42705-crn_106-3 Sheet No. 17 Side A No. 17 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 17 Side B 03/25/2021 10:13:22 R R R , M K C Y RAC- RAC- Why The The . 487, note 93 & P & P as they as they 147 and that SYCHOL 148 ETHODS ETHODS 154 . P [Vol. 106:529 106:529 [Vol. M EV M note 15, at 515 journal, whose mis- . R - and centers, so o et al., supra NN 150 DVANCES supra DVANCES , 69 A , 1 A , 1 A . 3, 3 (2018) (inaugurating new Writing Empirical Articles: Trans- note 14, at 529. CI . S supra note 144 (explaining the Society for the note 144 (explaining the Society for SYCHOL P renaissance.”). noting that psychology was becoming a noting that psychology RACTICES note 69 (arguing that the Replication Crisis is an oppor- CORNELL CORNELL LAW REVIEW Optimistic researchers spoke of “psychol- Optimistic researchers 153 ., https://osf.io/4znzp/wiki/home/ [https://perma.cc/ & P 152 . 403, 403 (2018) (asserting that psychology is now . 403, 403 (2018) (asserting that ¨ OUND 149 CI supra . 166, 166 (2018) (noting in 2016, the Society for Personality and . 166, 166 (2018) (noting in 2016, the Society By 2018, practices aimed at increasing the integ- aimed at increasing the By 2018, practices . F CI . S CI ETHODS . S S 151 M Bloom, Introducing Advances in Methods and Practices in Psychological Science Introducing Advances in Methods and PEN SYCHOL Nelson, Simmons & Simonsohn, See id. at 512 (“[R]eferring to this period as psychology’s ‘replication crisis’ See See, e.g., Cruwell et al., supra , Nelson, Simmons & Simonsohn, See, e.g., Nelson, Simmons & Simonsohn, Id. at 517. Id. See, e.g., Morton Ann Gernsbacher, SYCHOL note 14, at 511 (stating that events from 2010 to 2012 “sparked a period of note 14, at 511 (stating that events from P The Crisis sparked a period of methodological reflection reflection period of methodological sparked a The Crisis P DVANCES 152 153 154 151 147 148 149 150 research practices that were facing criticism were described as as described were criticism facing that were practices research [the] discipline,” integrity of threat to the “the biggest TICES TICES discussions about research practices have gone mainstream research practices have discussions about ogy’s renaissance,” leader in developing reliable research methodology, leader in developing Social Psychology set up a Presidential Task Force on Publication and Research); Social Psychology set up a Presidential Task Force on Publication and Research); About, O Improvement of Psychological Science was founded “to further promote improved Improvement of Psychological Science research field”); Daniel J. Simons, methods and practices in the psychological Editorial, . . . makes no sense. the rain that follows a long drought a water We do not call crisis. . . . This is psychology’s tunity for psychology to “lead the way”). parency, Reproducibility, Clarity, and Memorability parency, Reproducibility, Clarity, and 1 A Advances in Methods and Practices in Psychological Sciences Advances in Methods and Practices in sion it is to foster discussions about research); Anthony N. Washburn et al., sion it is to foster discussions about research); Do Some Psychology Researchers Resist Adopting Proposed Reforms to Research Do Some Psychology Researchers Resist Practices? A Description of Researchers’ Rationales that is still ongoing today.that is still ongoing - a flurry of papers, con It prompted and policy proposals ference presentations, TD3Q-WQSS] (explaining that the Center for Open Science was founded in 2013, TD3Q-WQSS] (explaining that the Center for Open Science was founded in “with a mission to increase openness, integrity, and reproducibility of scientific research”). \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 28 556 25-MAR-21 9:54 that published research findings could not be trusted. not be could findings research published that “enabled researchers to achieve the otherwise mathematically otherwise mathematically to achieve the researchers “enabled studies to underpowered most of their feat of getting impossible be significant.” cieties, and journals dedicated to improvement of methods and dedicated to improvement cieties, and journals practices. “confront[ing] . . . questionable research practices”); Munaf`“confront[ing] . . . questionable research rity of the discipline of psychology were “orders of magnitude of psychology were “orders rity of the discipline more common.” 491–93 (2018) (reviewing recent methodological literature). (“[J]ournals are filled with the 5%” of studies that show false-positive results, (“[J]ournals are filled with the 5%” of the [other] 95% . . . .”). “while the file drawers . . . are filled with (“The field of metascience . . . is flourishing.”); Nelson, Simmons & Simonsohn, (“The field of metascience . . . is flourishing.”); supra call Psychology’s Renaissance”); Patrick E. methodological reflection that we . . . Construction: Shrout & Joseph L. Rodgers, Psychology, Science, and Knowledge Crisis Broadening Perspectives from the Replication 42705-crn_106-3 Sheet No. 17 Side B No. 17 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 18 Side A 03/25/2021 10:13:22 R R R R R R R R R R 557 557 - neuros 157 note 162, at 2604 (“The While perfection is While perfection note 14, at 7 (listing initia- note 14 (noting that pharma- 160 163 Similar advances were advances Similar supra , 115 PNAS 2600, 2605 (2018) supra supra 155 - Guidelines for Transparency and Open note 98, at 626 (arguing that the main eroux, eroux, note 144 (noting that there is still a lot of note 144 (noting that there is still a note 150, at 502 (stating that replication note 150, at 502 (stating that replication behavioral ecology, behavioral see also supra note 125 and sources therein note 96 (stating that U.K. Academy of Medical note 96 (stating that U.K. Academy of 156 347, 347 (2017) (stating that symposium convened 347, 347 (2017) (stating that symposium and other fields. and other 164 LITIGATION SCIENCE SCIENCE LITIGATION 159 COLOGY note 150. but a growing consensus has been forming consensus has been forming but a growing . E note 15, at 529 (“[T]he Middle Ages are behind us, and the note 15, at 529 (“[T]he Middle Ages uwell et al., supra 161 EHAV supra Others merely represent increased attention to or represent increased attention Others merely supra medicine, Shrout & Rodgers, supra Nosek, Ebersole, DeHaven & Mellor, Nosek, Spies & Motyl, supra Poldrack et al., supra Leigh W. Simmons, Editorial, Diong, Butler, Gandevia & H´ Diong, Butler, Gandevia & H´ 158 162 See, e.g., Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven & See See See See See See See, e.g., Cr ¨ Simons, See The remainder of this subpart reviews some of the mea- of this subpart reviews some The remainder It is widely acknowledged that many scientific fields still scientific fields that many acknowledged It is widely M K 162 163 164 157 158 159 160 161 155 156 methodological or procedural devices that were not in use or procedural devices that methodological made in pharmacology, made around a number of proposals to modify accepted research of proposals to modify around a number practices. are true innovations— Some of these modifications before. attempts by original researchers are now common in neuroscience). attempts by original researchers are now and bad reporting and noting that tives from different fields to address awareness science”). “[t]here is considerable momentum throughout David T. Mellor, The Preregistration Revolution (reporting a “cultural shift” toward preregistration). the rules of statistical inference have no empathy for how hard it is to acquire data.”). cology journals published a series of editorials aimed at improving standards of cology journals published a series of data analysis and reporting). Sciences convened a meeting about reproducibility). ness (TOP), 28 B a need for a guide); Leyser, Kingsley confusion and misinformation and identifying opining that psychol- & Grange, supra note 128 (quoting psychologist Jim Grange out of the woods); Nelson, Simmons & ogy is leading the way but is not yet Simonsohn, sures that have been proposed in response to the Replication sures that have been proposed Crisis. for reform, but They are by no means the only proposals have garnered the most support they are among the ones that have a lot of work to do to improve the reliability of the research reliability of the to improve the of work to do have a lot they produce, likely unattainable, there is much that can—andlikely unattainable, should—be research practices the deleterious effects of the done to reduce Crisis. that led to the \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 29 25-MAR-21 9:54 insistence on existing best practices. insistence on to discuss transparency initiatives in behavioral ecology). to discuss transparency initiatives in cience, 2021] 2021] - hap are practices and publishing to research upgrades and speed. unprecedented with pening barriers to change may not be technical or financial, but rather social or cultural barriers to change may not be technical or financial, but rather social or cultural within the research community); (indicating earlier, less widely spread knowledge of the importance of certain (indicating earlier, less widely spread knowledge of the importance of certain aspects of methodological rigor). Enlightenment is just around the corner.”). Enlightenment is just around the corner.”). C Y 42705-crn_106-3 Sheet No. 18 Side A No. 18 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 18 Side B 03/25/2021 10:13:22 R R R R R R M K C Y note page [Vol. 106:529 106:529 [Vol. o et al., supra A result that was cher- 167 A major lesson offered by the A major lesson 168 note 137, at 3 (suggesting that restricting note 150, at 491 (arguing that unexpected supra HARKing (hypothesizing after results HARKing (hypothesizing after note 151 (noting that Perspectives on Psychological 166 When a researcher sets out on a research When a researcher CORNELL CORNELL LAW REVIEW supra 165 note 94, at 2 (arguing that HARKing is a threat to reproduci- is less likely to be a “true” result, and more likely to is less likely to be a “true” result, Shrout & Rodgers, supra Planning Planning See, e.g., Allen & Mehler, See See id. at 2600. for an ex post facto construc- See id. (explaining that hindsight bias allows note 150 (characterizing HARKing as a “questionable research practice”); o et al., supra 1. project Flexibility from a research Eliminating Analytical Hypothesis. Formulating a 167 168 165 166 are known) is now generally recognized as a research practice are known) is now generally it has a propensity to diminish that is inappropriate because the reliability of research results. path without first formulating an unambiguous hypothesis path without first formulating a hypothesis based on what (prediction) and instead constructs she risks being captured she observes in the data (postdiction), such as hindsight bias or by psychological phenomena apophenia—the for and find patterns in data tendency to look where there are none. interactions are usually attributable to randomness and that even when support interactions are usually attributable to randomness and that even when support for the interaction can be found in literature after the fact, the researcher should remain skeptical when the result was not specifically anticipated). 94 (explaining that apophenia can lead to false conclusions). 94 (explaining that apophenia can lead tion of a narrative that imbues meaning to randomness); Munaf`tion of a narrative that imbues meaning \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 30 558 25-MAR-21 9:54 prediction is the importance of separating Replication Crisis from postdiction. and that may be the most instructive in rethinking how the how the in rethinking instructive the most be that may and be reformed. should science litigation of practice be They can and pres- commitment, planning, into three categories: divided entation. in turn. I discuss each stage.begins at the planning a start a research project in To reliability, it is generate results of adequate manner that will (b) deter- an unambiguous hypothesis; critical to (a) formulate a detailed research of the project; and (c) design mine the scope protocol. rypicked from a set of data generated with no particular hy- rypicked from a set of data pothesis in mind—e.g., jelly bean result supra the green — 124–25 a result that generated with be an irreproducible fluke than only a specific question in mind. ble science); Simons, researchers’ opportunities for continuous learning by digging into data without a researchers’ opportunities for continuous learning by digging into data without formulated hypothesis “may be the price of unbiased science”); Gernsbacher, supra Munaf` Sciences changed reporting practices in an effort to limit the practice). 42705-crn_106-3 Sheet No. 18 Side B No. 18 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 19 Side A 03/25/2021 10:13:22 R R R R R R R R R R R R R A or 559 559 supra S ONE 171 O See infra Among - a hypoth 173 , 11 PL 169 note 137, at 4; Gern- note 96, at 117 (arguing note 96, at 116–17; see note 96, at 117. supra note 98, at 1361–62 (discuss- 172 note 104, at 552. note 104, at 552; Poldrack et al., supra note 104, at 543 (“[T]he typical studies note 151, at 169–70 (finding that only supra The Replication Crisis revealed Crisis The Replication Allen & Mehler, supra but there is broad support for the but there is broad Poldrack et al., supra See 176 See note 98, at 1363 (recommending researchers either See Poldrack et al., supra LITIGATION SCIENCE SCIENCE LITIGATION note 98, at 4 (asserting that low-powered research is note 98, at 4 (asserting that low-powered supra Not all researchers are convinced that these prac- are convinced that these Not all researchers note 149, at 405. 170 175 Simmons, Nelson & Simonsohn, Bakker, van Dijk & Wicherts, supra Bakker, van Dijk & Wicherts, and setting sample sizes based on an a priori power sizes based on an a priori and setting sample supra See See, e.g., Washburn et al., supra The statistical concept of “power” refers to the likelihood that a study will The statistical concept of “power” refers See See the pretense” that small See, e.g., id. (arguing that researchers should “end A researcher can commit to a hypothesis through preregistration. A researcher can commit to a hypothesis requires clear differentiation be- Testing only pre-formulated hypotheses note 96, at 117. the minimum sample size A power analysis determines 174 Determining the Scope. the Scope. Determining should be a size in a thoughtful manner Selecting a sample Any researcher has the power to prevent HARKing: all the all HARKing: prevent to the power has researcher Any Bakker, van Dijk & Wicherts, supra M K 175 176 171 172 173 174 169 170 “still the norm” as of 2013). also that inappropriately small sample sizes cannot be justified solely on the basis that that inappropriately small sample sizes cannot be justified solely on the basis lack of resources prevented the researchers from using larger samples); Simmons, Nelson & Simonsohn, sbacher, detect an effect if an effect exists. it is The more powerful a study, the more likely likely to conclude that there is no effect to detect the existing effect and the less when in reality there is one. et al., sample sizes can ever be adequate); Poldrack tween exploratory work (aimed at generating hypotheses) and confirmatory work tween exploratory work (aimed at generating (aimed at testing hypotheses). proposed approaches are the adoption of minimum sample are the adoption of proposed approaches sizes standard part of the planning of a research study. of the planning of a research standard part that many researchers traditionally have not planned their traditionally have that many researchers sizes that are care, tending to use sample studies with enough powered,” therefore render a study “poorly too small and \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 31 25-MAR-21 9:54 even leaving the size of a study open. even leaving the esis before starting a research project and then use any gath- and then use research project starting a esis before that hypothesis—and to test solely ered data no additional hypotheses. e0149794, at 10 (2016) (arguing that traditional sample sizes are too small); Open e0149794, at 10 (2016) (arguing that traditional Sci. Collaboration, supra tices should be routine, tices should be analysis. 2021] 2021] to commit and do is formulate needs to researcher are insufficiently powerful . . . .”); Alexander Etz & Joachim Vandekerckhove, are insufficiently powerful . . . .”); Alexander Project: Psychology Bayesian Perspective on the Reproducibility supra level needed to observe the desired minimum effect size with a desired minimum of statistical significance. forty percent of surveyed researchers believe that an a priori power analysis should be a required part of a standard research practice). adhere to minimum sample sizes or justify why they are unable to do so). section III.B.2. subpart II.A & note 114. ing the harm in ceasing data collection based on an interim analysis); ing the harm in ceasing data collection C Y 42705-crn_106-3 Sheet No. 19 Side A No. 19 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 19 Side B 03/25/2021 10:13:22 R R R R R R R M K C Y Nosek, They Degrees See 182 [Vol. 106:529 106:529 [Vol. note 15, at 516–17 The flexibility The flexibility When Analytical When Analytical note 98, at 1360 (noting supra 180 181 see also Bakker, van Dijk & . 1, 3 (2016) (offering a check- note 98 (stating that the likeli- note 104 (demonstrating that 177 supra note 98, at 6 (stating that Psycho- SYCHOL supra supra P note 162, at 2601 (correcting diagnosticity of note 162, at 2601 (correcting diagnosticity RONTIERS , 7 F supra CORNELL CORNELL LAW REVIEW The problem does not lie in the individual lie in the individual does not The problem - becomes invisible in the ulti When this choice 178 note 104, at 545 (stating that a majority of researchers admit to 179 Simmons, Nelson & Simonsohn, supra -values and other measures of confidence to account for its presence. p-values and other measures of confidence See See, e.g., Simmons, Nelson & Simonsohn, See Bakker, van Dijk & Wicherts, See supra subpart II.A. known, it is often possible to adjust If the extent and nature of flexibility is See, e.g., Open Sci. Collaboration, supra It is impossible to provide an exhaustive list of types of It is impossible to provide an Designing a Detailed Research Protocol. a Detailed Designing 181 182 178 179 180 177 choices that researchers make, but rather in the existence of make, but rather choices that researchers choice itself. mate publication—i.e., only those when a researcher reports to the reported, steps that led directly specific methodological results—itpossibly cherrypicked to tell whether is not possible from the presence are an artifact resulting the reported results the result of a or whether they were of Analytical Flexibility protocol (and faithfully executed research preplanned and likely to be a “true” result). therefore more hood of finding a false-positive finding at the 5% level is necessarily greater than hood of finding a false-positive finding at the 5% level is necessarily greater 5%). reported the level of confi- Typically, a reported p-value would increase significantly (i.e., a result of this adjustment. dence it expresses would decrease) as Ebersole, DeHaven & Mellor, -values to account for the number of tests that was performed is possible, but p-values to account for the number of rarely done). Flexibility remains unreported, the statistical significance of unreported, the statistical Flexibility remains overstated. reported results is likely to be Analytical Flexibility, because the ways in which a researcher Analytical Flexibility, because are innumerable. can vary or improvise her approach researchers have to modify the way they gather or analyze data gather or analyze the way they have to modify researchers of re- the irreproducibility greatly to can contribute midstream results. search \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 32 560 25-MAR-21 9:54 notion that sample sizes should be set before the start of the of the the start before be set should sizes sample that notion way. purposeful deliberate, in a study, using “Questionable Research Practices” and many underestimate the using “Questionable Research Practices” and many underestimate consequences). Wicherts, list covering thirty-four types of Analytical Flexibility); that common “degrees of freedom” include choice of sample size and dependent that common “degrees of freedom” include choice of sample size and dependent variables and covariates, and selective reporting); Jelte M. Wicherts et al., of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking requires authors to disclose how they selected their sample sizes logical Science requires authors to disclose how they exclusion of observations). and how large the samples were after commonly used “questionable research practices” can lead to false-positive find- commonly used “questionable research & Simonsohn, ings up to 40 percent); Nelson, Simmons Flexibility that are generally thought to (explaining that even levels of Analytical rates from 5 to 61 percent and that, “[i]n be acceptable can raise false-positive false-positive rate to be very close to truth, it is not that hard to get a study’s 100%”). 42705-crn_106-3 Sheet No. 19 Side B No. 19 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 20 Side A 03/25/2021 10:13:22 R R R R R R R R 186 561 561 Design- 188 The more 189 191 Simmons, Nelson & removal of outli - of removal See 183 note 98, at 5 (noting that note 98, at 1362–63; see also and the analysis itself. the analysis and A plan should include a A plan should supra 185 note 98, at 6 (“By committing to a pre- 190 note 182, at 8 (explaining that even for a note 182, at 8 (explaining that even for note 94, at 2 (observing that enthusiastic scien- note 94, at 2 (observing that enthusiastic note 98 (“[I]t is common (and accepted practice) for LITIGATION SCIENCE SCIENCE LITIGATION note 137, at 3 (suggesting that a decrease in the efficiency note 98, at 1361 (stating that approximately seventy percent of note 98, at 1361 (stating that approximately o et al., supra However, by allowing flexibility in a research pro- flexibility in a by allowing However, supra 187 Simmons, Nelson & Simonsohn, Munaf` at 1360 (noting many options for the treatment of outliers are See id. at 1360 (noting many options for the pre-analysis data cleanup, data pre-analysis See, e.g., Wicherts et al., supra See See E.g., Open Sci. Collaboration, supra See id. at 11 (explaining that the plan should include all conditions, vari- Outliers are data points that are out of line with the rest of the data.Outliers are data points that are out of There supra See generally Open Sci. Collaboration, in a potentially infinite number of Researchers can perform data analysis a possible correlation between two For example, a researcher may explore When a study includes a pilot phase, a researcher can manipulate the end When a study includes a pilot phase, Researchers may choose to explore different options for different options to explore may choose Researchers without to stamp out flexibility entirely It is impossible 184 M K 188 189 190 191 184 185 186 187 183 ing a detailed research plan can prevent a researcher from research plan can prevent ing a detailed cognitive biases. falling prey to surveyed researchers reported having decided whether to continue data analysis surveyed researchers reported having based on an interim data analysis). to spot patterns in noise); Simmons, tists motivated by discovery have a tendency Nelson & Simonsohn, supra what researchers to explore various analytic alternatives . . . to then report only ‘worked.’”). of continuous learning “may be the price of unbiased science”). ables, covariates, as well as when data collection will stop and start). Simonsohn, outliers and many reasonable options for are no set criteria for the removal of doing so. introduces problems). justifiable, but the availability of choice ways. used statistics software offers multiple standard statistical analysis, commonly is typically not described in articles). different options, but the choice of method for legitimate reasons. variables or control for a variety of variables Allen & Mehler, supra of the pilot phase and the start of the actual study. of the pilot phase and the start of the a research protocol can be turned into a “script” setting out a a research protocol can be turned the narrower the opportunity for predetermined set of steps, legitimate reasons, and digging into the data to explain unex- the data to explain digging into reasons, and legitimate of p- or an example nefarious results is not necessarily pected hacking. their job, but the way of scientists’ ability to do standing in the research proto- researchers can eliminate from more flexibility are. reliable the resulting outcomes cols, the more \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 33 25-MAR-21 9:54 detailed description of how the researcher(s) will collect data, of how the researcher(s) detailed description conditions they will apply, and how they will clean data, what in what manner. which analyses they will run and ers, tocol, a researcher risks being “hijacked by the data” and ob- by the data” being “hijacked researcher risks tocol, a not be able to that other researchers would taining results sample. redrew and reanalyzed the reproduce if they 2021] 2021] collection, data regarding decisions include specified analysis plan, one can avoid common cognitive biases.” (citation specified analysis plan, one can avoid common cognitive biases.” (citation omitted)). there are more than 200 ways to clean data). there are more than 200 ways to clean C Y 42705-crn_106-3 Sheet No. 20 Side A No. 20 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 20 Side B 03/25/2021 10:13:22 R R R R R R R M K A C Y and 196 eNeuro note 151 199 . 137, 137 supra Association Be- supra note 128 SYCHOL [Vol. 106:529 106:529 [Vol. . P es Dechartres, Phi- . 94, 94–95 (2020) CI OC But see Aba Szollosi et S and described as and described - The use of preregis , 45 S 194 195 OGNITIVE has changed reporting require- Improving the Way Science Is C note 15, at 519. 197 e0373, at 2 (2017) (stating that RENDS supra EURO N , 24 T E 871, 872 (2014) (discussing guidelines in the 871, 872 (2014) (discussing guidelines note 137, at 3. note 96, at 120 (recommending a preregistration note 96, at 120 (recommending a preregistration note 150, at 493. , 4 Scientific Rigor and the Art of Motorcycle Mainte- o et al., Scientific Rigor and the Art of Motorcycle CORNELL CORNELL LAW REVIEW In doing so, it functions as a commitment In doing so, it functions as ., July 2016, at 1, 1 (observing that preregistration has be- ., July 2016, at 1, 1 (observing that preregistration IOTECHNOLOGY note 162 (pointing at 8,000 preregistrations filed through the note 162 (pointing at 8,000 preregistrations 198 ED B Preregistration has been called “the most impor- has been called “the most Preregistration ATURE 193 Allen & Mehler, supra Marcus Munaf` Commitment Commitment at 6. 192 See, e.g., Christophe Bernard, Editorial, See Id. See Poldrack et al., supra Shrout & Rodgers, supra Nelson, Simmons & Simonsohn, is not yet the norm, but See id. at 520 (observing that preregistration See Preregistration forces researchers to think through possi- forces researchers to think Preregistration 2. to to commit to force researchers a simple solution There is Is Preregistration Worthwhile? 197 198 199 193 194 195 196 192 Open Science Framework as evidence of a cultural shift). Open Science Framework as evidence will soon implement preregistration); Leyser, Kingsley & Grange, of (“[M]any psychology journals now recommend or require the preregistration Lakens, Editorial, Registered Reports: A studies . . . .”); Brian A. Nosek & Dani¨el Method to Increase the Credibility of Published Results requirement to avoid HARKing in neuroimaging studies); Simons, requirement to avoid HARKing in neuroimaging three to five years); Agn`should become the norm in the next & Isabelle Boutron, lippe Ravaud, Ignacio Atal, Carolina Riveros tween Trial Registration and Treatment Effect Estimates: A Meta-Epidemiological tween Trial Registration and Treatment Study, 14 BMC M (2014) (stating that in 2014, Social Psychology published its first issue with only preregistered replication studies). sign of a cultural shift, some journals now make preregistra- shift, some journals now sign of a cultural for publication. tion a precondition (arguing that while preregistration may increase transparency, it is unlikely to (arguing that while preregistration may improve scientific reasoning and theory development). come the norm rather than the exception in medicine, and that the World Health come the norm rather than the exception platform); Nosek, Ebersole, DeHa- Organization launched its own preregistration ven & Mellor, supra al., ble contingencies and expected outcomes before starting a re- and expected outcomes before ble contingencies search project. a predetermined hypothesis, scope, and research plan: prereg- and research plan: scope, hypothesis, a predetermined istration. “the only way for authors to convincingly demonstrate that for authors to convincingly “the only way were not p-hacked.” their key analyses tration has grown rapidly over the past few years, and some rapidly over the past few tration has grown few years. become the norm in the next predict that it will \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 34 562 25-MAR-21 9:54 -hacking toward a favorable a favorable toward p-hacking or inadvertent intentional result. form of checklists for preregistration of studies in medicine, animal studies, ob- form of checklists for preregistration servational epidemiology, and other fields). device, reducing the risk of both “unconscious error” device, reducing the risk of tant outcome[ ] of the replication crisis” outcome[ tant Done, Evaluated, and Published ments to avoid HARKing). nance, 32 N (noting that Perspectives on Psychological Science 42705-crn_106-3 Sheet No. 20 Side B No. 20 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 21 Side A 03/25/2021 10:13:22 R R R R R R R R R R R R R R 207 or note 563 563 205 supra note 150, at 504 note 197, at 138 (dis- supra ., https://cos.io/prereg CI note 162. note 98, at 7 (positing that 208 S supra note 162, at 2602 (discussing PEN note 15, at 519 (explaining that supra supra Seven Selfish Reasons for Preregis- . O But cf. Washburn et al., note 182, at 1, 3 tbl.1 (suggesting a supra TR supra It increases the credibility of the credibility It increases supra If the particulars of an analysis of an If the particulars 201 - note 192 (describing preregistration checkl 203 note 144, at 17. note 144, at 10 (describing private, self-archiving Deviations from a preregistered re- Deviations from note 150, at 404. note 201 (stating that a journal can offer early in- 209 note 197 (noting that eNeuro “nearly guarantees publi- LITIGATION SCIENCE SCIENCE LITIGATION The strongest form of preregistration in- preregistration form of The strongest Preregistration can be done publicly Preregistration supra supra / (Oct. 31, 2016), https://www.psychologicalscience.org o et al., supra and rescuing researchers from “being taken “being from researchers rescuing and 202 its primary value lies not in the publication of lies not in the publication its primary value 204 supra 200 206 What Is Preregistration?, C BSERVER ¨ uwell et al., supra Gernsbacher, Nosek, Ebersole, DeHaven & Mellor, Cr¨ Cruwell et al., supra Bernard, See See supra subpart II.A. See See (suggesting that, ideally, a research plan includes all steps the See id. (suggesting that, ideally, a research See, e.g., Munaf` See, e.g., Eric-Jan Wagenmakers & Gilles Dutilh, Eric-Jan Wagenmakers & Gilles Dutilh, See See Preregistration does not prevent authors from reporting does not prevent authors Preregistration M K 206 207 208 209 203 204 205 200 201 202 151, at 169 (noting that some researchers avoid preregistration out of concern for 151, at 169 (noting that some researchers avoid preregistration out of concern being “scooped”). search plan, necessitated by unexpected findings or unantici- search plan, necessitated by cludes a full specification of a researcher’s hypothesis, study hypothesis, of a researcher’s a full specification cludes and analysis plan. design, ] (describing preregistration through submission [https://perma.cc/ZR9V-FUG7] (describing preregistration through to a public registry). tration, APS O researchers will take); Open Sci. Collaboration, of study design and analysis preregistration should include full specification plan). observational epidemiology, and other ists used in medicine, animal studies, fields); Nosek, Ebersole, DeHaven & Mellor, options); Nelson, Simmons & Simonsohn, some preregistration platforms allow for researchers to lock preregistered studies from the public or share them anonymously). unexpected findings. In addition to enforcing the use of predetermined research pro- the use of predetermined In addition to enforcing of research an opportunity to separate scrutiny tocols, it offers scrutiny of the results. methodology from \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 35 25-MAR-21 9:54 observer/seven-selfish-reasons-for-preregistration [https://perma.cc/L69J- observer/seven-selfish-reasons-for-preregistration 3PWZ]. research results. research are dependent on the outcome of earlier steps, the plan can steps, the outcome of earlier on the are dependent specify- procedures or list of operating a decision tree include under each antici- the researcher(s) will follow ing which steps pated condition. confidentially; hostage by [their] own data.” [their] by hostage the research protocol but in the binding of the researcher. but in the binding of the research protocol 2021] 2021] bias, cognitive cussing the phenomenon of “CARKing”: Critiquing After Results are Known); cussing the phenomenon of “CARKing”: Critiquing After Results are Known); Wagenmakers & Dutilh, principle acceptance of articles based on a preregistered research protocol). checklist covering thirty-four types of Analytical Flexibility). checklist covering thirty-four types of the preregistration of “decision trees”); Shrout & Rodgers, the preregistration of “decision trees”); the Open Science Foundation’s decision (lauding as “systematic and thoughtful” reports expressing uncertainty about to allow researchers to file preregistration what measure to use); Wicherts et al., cation” of preregistered studies); Nosek & Lakens, C Y 42705-crn_106-3 Sheet No. 21 Side A No. 21 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 21 Side B 03/25/2021 10:13:22 R R R R R R R R R R M K C Y 1, 1 To allow OLLABRA Psychological Some jour- - Preregistra 214 [Vol. 106:529 106:529 [Vol. , 1 C note 162, at 2602 216 note 201. 210 supra supra It merely constrains constrains It merely note 98, at 11 ( note 162, at 2604. note 162, at 2602. note 15, at 518–19 (noting that Are We Wasting a Good Crisis? 211 Preregistration “does not Preregistration supra supra supra They should disclose all vari- 212 215 note 137, at 2–3. also allows Preregistration e Crosas et al., Data Policies of Highly-Ranked note 137, at 3–4 (arguing that researchers may 213 Wagenmakers & Dutilh, 217 See CORNELL CORNELL LAW REVIEW supra note 197, at 1 (stating that since 2017, eNeuro requires e.g., Open Sci. Collaboration, supra Allen & Mehler, supra Allen & Mehler, supra Nelson, Simmons & Simonsohn, Bernard, Presentation requires full disclosure of sample-size selection, exclusion of observa- See See, See See Nosek, Ebersole, DeHaven & Mellor, See Nosek, Ebersole, DeHaven & Mellor, See supra subpart II.A. note 157 (stating that Behavioral Ecology will only publish papers docu- When research results are published with limited or selec- results are published with limited When research 3. 215 216 217 210 211 212 213 214 some peer reviewers now demand disclosure of all measures, conditions, exclu- some peer reviewers now demand disclosure sions, and other analytical decisions). The Availability of Psychological Research Data After the Storm menting all design decisions and analytical decisions); Wolf Vanpaemel, Maarten menting all design decisions and analytical decisions); Wolf Vanpaemel, Maarten Vermorgen, Leen Deriemaecker & Gert Storms, Science tions, and all experimental conditions, including failed manipulations); Simmons, supra make changes to a preregistered plan but should justify and discuss each change make changes to a preregistered plan & Mellor, at publication); Nosek, Ebersole, DeHaven of their predictions and can help build a researchers to take credit for the quality researcher’s reputation. (suggesting that for some types of study, preregistration could take place in (suggesting that for some types of study, steps to be registered, and then executed, in stages, with each stage informing the the next stage). readers to assess the presence of Analytical Flexibility, re- readers to assess the presence information necessary to allow searchers should report all others to replicate their study. nals now require extensive disclosures as a precondition for nals now require extensive disclosures checklists to ensure all neces- publication, sometimes using sary detail is reported. tion also does not take away researchers’ “narrative license” in “narrative away researchers’ does not take tion also the data. and presenting explaining tively reported information about the process that produced information about the process tively reported which their relia- cannot evaluate the extent to them, readers impaired by Analytical Flexibility. bility has been (2015) (noting that Science and Nature require openness of data). \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 36 564 25-MAR-21 9:54 pated violations of foundational assumptions, would not not would assumptions, of foundational violations pated to be expected be but would study the invalidate necessarily when taken into account they can be clearly so that reported findings. of the study’s the reliability assessing their flexibility to adjust their hypothesis or methodology in the or methodology their hypothesis to adjust their flexibility research project. course of the eliminate the possibility of poor statistical practices, but it does of poor statistical practices, eliminate the possibility make them detectable.” ables collected, as well as details about all analytical decisions, ables collected, as well as details that failed. including descriptions of analyses inclusion of computer code); Merc` 42705-crn_106-3 Sheet No. 21 Side B No. 21 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 22 Side A 03/25/2021 10:13:22 R R R 565 565 CIENCE S (Mar. 30, 2018), https://osf.io/ APERS III P ITIGATION IV L In this context, incomplete or In this context, X R note 137, at 3–4 (arguing that changes to prere- A 218 OC note 192 (noting that Nature has introduced reporting LITIGATION SCIENCE SCIENCE LITIGATION ESSONS FOR note 98, at 17 (When a study has been preregistered, “the supra L o, supra Allen & Mehler, supra Reliability Problems in Litigation Science Reliability Problems in Litigation See There is ample reason to believe that the problems that reason to believe that the There is ample in the course of the Repli- The defects of method uncovered - that facili culture a help create can preregistration Public A. M K 218 checklists). preprints/socarxiv/9h7ay [https://perma.cc/C3SB-A8EM] (noting that as of - January 2018, 155 of 291 social sciences journals surveyed have a data-disclo sure policy); Munaf` gistered plan may be made but should be justified and discussed); Open Sci. gistered plan may be made but should be justified and discussed); Open Collaboration, stood at the center of the Replication Crisis are at least as stood at the center of the Replication it is in academic science.present in litigation science as Sub- and institutional norms in liti- part III.A explains why cultural as likely as academic science to gation science make it at least flaws uncovered in the be vulnerable to the methodological as much in need of a critical look. Crisis, and therefore at least learned from the Replication Subpart III.B draws on lessons way litigation science is created Crisis to propose reforms to the and assessed for admissibility. are likely to be at least as perva- cation Crisis and its aftermath in connection with civil litigation sive in experiments conducted science.as they have been in academic expert witnesses When or experiments, just like their are retained to conduct analyses they face powerful incentives colleagues in academic science, of result: findings that can with- to produce a particular type Social Science Journals 8, S \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 37 25-MAR-21 9:54 selective reporting of methodological steps taken would be of methodological steps selective reporting easily determine interest; a reader could against the author’s lack of a detailed was preregistered and the that the study would raise suspicion. description of methodology 2021] 2021] disclosures. these tates steps analytical When all envisioned and re- the steps taken confirming preregistered, have been all analyses—notthe outcome of porting analy- just successful ses—isstep. logical next a the stage, At the publication as envi- took the steps that she either confirms researcher the prere- deviations from justifies any or explains and sioned plan. gistered research investigator can later demonstrate that his or her published analysis matched his investigator can later demonstrate that his or her published analysis matched or her original plan—or, if any changes were necessarily, detail what was changed and why.”). C Y 42705-crn_106-3 Sheet No. 22 Side A No. 22 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 22 Side B 03/25/2021 10:13:22 R R . . M K If EV UB C Y 220 221 . J. P . L. R M ROOK [Vol. 106:529 106:529 [Vol. , 73 B Until that time, challenge “has plenty to 224 And Analytical Flexi- And Analytical See, e.g., Beck v. Hirchag, No. There is no reason to There is no 223 Daubert 222 -hacking, and other forms of ex- p-hacking, and A Cognitive Scientist Looks at Daubert, 95 A CORNELL CORNELL LAW REVIEW Research scientists aim for statistically significant re- significant statistically aim for scientists Research S114, S117 (2005) (An expert facing a S114, S117 (2005) (An expert facing See supra subpart II.A. Indeed, many expert witnesses are academic scientists who occasionally See supra note 63 and accompanying text. jurisdictions do not Some See, e.g., Susan Haack, Of Truth, in Science and in Law See supra subpart II.A & note 100. See George P. Lakoff, 219 Moreover, the undetectable nature of this impairment of Moreover, the undetectable EALTH 222 223 224 219 220 221 branch out to litigation science. require a report at any stage in the proceeding. G041955, 2011 Cal. App. Unpub. LEXIS 2649, at *15 (Cal. Ct. App. Apr. 11, 2011) G041955, 2011 Cal. App. Unpub. LEXIS 2649, at *15 (Cal. Ct. App. Apr. 11, (noting that expert reports are not required in California). In those jurisdictions, an expert’s disclosures may be limited to information the expert (or retaining at a party) chooses to disclose, plus any disclosures in response to questions deposition or evidentiary hearing. the Replication Crisis has taught the world one thing, it is this: world one thing, has taught the Crisis the Replication is likely to findings significant to generate statistically pressure more reliable re- of behavior that leads to provoke the type search results—HARKing, Flexibility. ploitation of Analytical they typically design and perform their experiments hidden they typically design and perform protecting much of their from view, a cloak of litigant privileges and scrutiny, presenting selected work product from discovery have been completed and re- outcomes only after experiments bility does not distinguish between academic science and litiga- between academic bility does not distinguish of scientific present, it impairs the reliability tion science; when which they were of the purpose for findings regardless generated. be endemic in both modes of reliability can be expected to science. in litiga- Just as in academic science, experimenters to present research results tion have traditionally been allowed selectively. do not dis- As described in Part I, experts typically they submit a report in which close their methodology until completed study, typically to- they present the results of their period. ward the end of the expert-discovery expert witnesses aim for statistically significant results be- significant for statistically witnesses aim expert the Daubert hurdle. them to pass will enable cause they \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 38 566 25-MAR-21 9:54 (or equivalent) motion and be presented to a be presented and motion equivalent) (or a Daubert stand jury. believe that individuals performing research in support of liti- performing research believe that individuals of pressure than less influenced by this type gation would be on the academic side. their colleagues 985, 988 (2008) (“[S]cientists complain about . . . the professional insult of being 985, 988 (2008) (“[S]cientists complain ‘Dauberted’ or ‘dauberted out’ . . . .”). sults because it will help them get those results published; them get those it will help sults because H lose: his or her reputation for professional expertise . . . .lose: his or her reputation for professional is When a scientist ‘Dauberted out’ of a trial, the repercussions for the scientist are serious.”). 42705-crn_106-3 Sheet No. 22 Side B No. 22 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 23 Side A 03/25/2021 10:13:22 R 226 227 567 567 Roundup Prods. Liab. See In re Just as in some of the academic the academic some of as in Just 225 p-hacking” appears. . P. 26(a)(2)(B)(i)–(ii) that an expert report is (providing LITIGATION SCIENCE SCIENCE LITIGATION IV . R. C ED motions (or equivalent motions under Frye or motions (or equivalent motions and (2) even if they were aware, by the time the and (2) even if they were aware, The power of the adversarial process to bring to light The power of the adversarial . P. 194.2(f)(3) (providing that a party may request a disclosure of “the 229 IV 228 See supra Part I. See supra subpart II.A. See, e.g., F See supra subpart II.A and note 180. Major legal research databases Lexis and Westlaw contain only one court The corrosion in reliability that results from Analytical in reliability that results from The corrosion Daubert . R. C M K EX 226 227 228 229 225 Litig., 390 F. Supp. 3d 1102, 1137 (N.D. Cal. 2018). The case poignantly illus- trates the court’s lack of awareness of problems relating to Analytical Flexibility. general substance of the expert’s mental impressions and opinions and a brief general substance of the expert’s mental impressions and opinions and a summary of the basis for them”). opinion in which the term “ Flexibility is difficult to detect in civil actions that proceed ac- to detect in civil actions Flexibility is difficult procedure described in Part I: cording to the current standard the reliability of proffered ex- by the time the court considers on the expert’s experimental re- pert testimony that is based whether a presented result sults, it is impossible to determine of data manipulation or the was the result of unsavory forms that was reliably ap- result of reliable research methodology plied. once an expert starts gathering and analyzing data, nothing starts gathering and analyzing once an expert on the fly. from modifying the protocol stops him or her as Just Crisis, once sciences before the Replication in the academic no pressure on the been completed, there is experiments have one has access to unfavorable results, and no expert to disclose results. file drawer containing undisclosed the metaphorical \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 39 25-MAR-21 9:54 sciences before the Replication Crisis, they are able to choose choose able to are they Crisis, Replication the before sciences method - and analytical setup fixing their experimental between they go. it up as front and making ology up watches No one they exploit to which check the extent shoulder to over their results. obtain favorable Flexibility to Analytical is cur- There a study to preregister to force an expert mechanism rently no it.prior to conducting in the academic sciences, Just as flaws in proposed evidence is limited when an important cause flaws in proposed evidence is of unreliability is undetectable. other standards), as currently implemented, have not proven to other standards), as currently testimony for this particular be adequate vehicles for screening dimension of unreliability. in the Daubert This is not a flaw result of an interplay between standard itself, but rather the litigants lack awareness of the multiple factors: (1) courts and unreliability into an expert’s way Analytical Flexibility injects process; required to include “all opinions the witness will express and the basis and rea- required to include “all opinions the witness will express and the basis and sons for them” and “the facts or data considered by the witness in forming them”); T 2021] 2021] been analyzed. have sults C Y 42705-crn_106-3 Sheet No. 23 Side A No. 23 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 23 Side B 03/25/2021 10:13:22 R M K C Y stage, note 30, at supra [Vol. 106:529 106:529 [Vol. Opposing par- Opposing p-hacking, perform- 230 Litigation-Generated Sci- . 117 (2008) (arguing that ERSP In adopting the Daubert P 232 EALTH . H NVTL L.J. 39, 51 (2013) (describing “deep tensions There is also no way to account for the There is also no way to account , 116 E ALHOUSIE Id. CORNELL CORNELL LAW REVIEW 231 Proposals See supra subpart II.A. See id. See generally Leslie I. Boden & David Ozonoff, Even if a court were able to detect or discover that Analyti- were able to detect or discover Even if a court Courts have an important role to play in addressing this Courts have an important role B. 230 231 232 ties can question an expert about the expert’s methodology and expert’s methodology about the question an expert ties can so long as hearing, but or evidentiary at a deposition results, and analytical about the research protocol crucial information an expert report, employed can be left out of steps the expert they need will generally lack the information these adversaries -hacking or HARK- that the expert engaged in p to demonstrate ing.- Flexibility can severely under In other words, Analytical is largely invisible of scientific evidence, but mine the reliability gatekeeping proceedings. under current process, at the played a role in an expert’s cal Flexibility way to predict—letDaubert stage there is typically no alone quantify—the flexibility has contaminated ways in which this the resulting data. ing tests using a number of different measures and reporting only those results ing tests using a number of different that were favorable. the argument on the ground that the The court rejected was defendant had not shown why the measure for which results were reported an inappropriate choice. The defendant argued that a plaintiff’s expert had engaged in The defendant argued that a plaintiff’s \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 40 568 25-MAR-21 9:54 expert presents her results—typically her presents expert completed she has after a report—there filed work and her to assess no way generally is the results creation of Flexibility in the of Analytical the role did in fact Flexibility whether Analytical way to assess (i.e., no the expert’s process). into inject unreliability contamination in interpreting the data or to determine whether contamination in interpreting the presence of Analytical the data might be reliable despite creation.Flexibility in the process of its Daubert At the to admitting the proposed testi- the court’s options are limited mony—leavingto try to convince the finder of the parties free fact of the data’s unreliability—or excluding it. problem.the differences A rich scholarly literature examines science” and “academic sci- and similarities between “litigation to which litigation science ought ence” and debates the extent that apply in academic to import the practices and standards science into its courtroom setting. “the problems with litigation-generated science . . . are very general and apply to “the problems with litigation-generated science . . . are very general and apply much or most science”); Susan Haack, A Match Made on Earth: Getting Real About Science and the Law, 36 D between science and the culture of (at least U.S.) law”); Jasanoff, ence: Why Should We Care? 42705-crn_106-3 Sheet No. 23 Side B No. 23 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 24 Side A 03/25/2021 10:13:22 R R R R R , 569 569 stan- Litiga- 234 Law’s Knowl- 238 235 While the Daubert 237 Only if the reference field has en- Only if the reference 236 note 30, at 124. note 30, at 123–24. LITIGATION SCIENCE SCIENCE LITIGATION note 30, at 123–24 (“In Daubert’s epistemological frame- S49, S56-S57 (2005) [hereinafter Jasanoff, S49, S56-S57 (2005) [hereinafter Jasanoff, - on judges to “think like scien framework calls Law’s Knowledge, supra note 232, at S51 (“[T]he law needs supra EALTH supra note 30, at 123–24 serve as inscrip- (“Judges, in this view, should In this view, correct scientific inquiry, is governed is governed inquiry, view, correct scientific In this . H UB 233 Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 592–93Daubert v. Merrell Dow Pharm., Inc., (1993); Jasanoff, Jasanoff, Daubert supra See See Jasanoff, See Daubert, 509 U.S. at 593–94 (holding that in making a reliability deter- See Jasanoff, supra See . J. P Sheila Jasanoff has argued that litigation science and aca- Sheila Jasanoff has argued that The M M K 234 235 236 237 238 233 tists” and, in that role, to assess the validity of experimental role, to assess the validity tists” and, in that practices in estab- with reference to and analytic methodology science. lished fields of facts as necessary adjuncts to doing justice; science seeks facts more as an end in facts as necessary adjuncts to doing justice; science seeks facts more as an itself.”). are work, scientists establish the criteria for what counts as science, and judges charged with importing these into admissibility decisions . . . .”). and mination, a court may consider factors such as peer review, publication, general acceptance). tion science and academic science are simply two are simply academic science and tion science litigation of science, with practice of a singular instantiations to a specific set to generate knowledge relating science aiming science aiming to in a case, and academic of facts at issue not limited in for its own sake, typically generate knowledge any particular set of facts. application to demic science are too dissimilar to allow for a direct importa- demic science are too dissimilar a courtroom setting, and that tion of scientific standards into is possible rests on an any notion that such direct importation 124–25; in Legal Settings Sheila Jasanoff, Law’s Knowledge: Science for Justice 95 A \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 41 25-MAR-21 9:54 dard leaves room for departures from laboratory standards as dard leaves room for departures constraints—time,necessitated by litigation-specific cost, of disputes even after the emer- norms against the relitigation gence of new information—experts producing evidence for are expected to hew as closely presentation in a civil proceeding academic science. as possible to the practice of dowed the methodology with certain indicia of reliability—such with certain indicia dowed the methodology acceptance in the field—willas passing peer review or general reliable enough to be employed the methodology be considered in connection with civil litigation. ] (listing “contextual factors” that “[may] lead to different evaluations of the edge] (listing “contextual factors” that “[may] same science in different contexts”). by the “scientific method,” and any knowledge-generating en- knowledge-generating and any “scientific method,” by the of this method. the rules should aim to follow deavor 2021] 2021] that an assumption on leaned Court Supreme the standard, as as much science mimic academic should science litigation possible. Jasanoff, tion . . . devices, automatically writing scientists’ standards of reliability and tion . . . devices, automatically writing evidence . . . .”). validity into their assessments of the C Y 42705-crn_106-3 Sheet No. 24 Side A No. 24 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 24 Side B 03/25/2021 10:13:22 R R R R R M K C Y that - ar Jasanoff [Vol. 106:529 106:529 [Vol. 239 While Jasanoff ap- While Jasanoff Daubert framework “rest[s] 241 note 232, at S53 (arguing that the note 232, Law’s Knowledge, at S50 supra supra court’s notion that the ideal prac- court’s notion - academic sci science and Litigation . 247, 250–51 that judges cannot (2018) (arguing note 30 (“The mistake, in law[,] . . . is to believe that note 30 (“The mistake, in law[,] . . . is 240 EV 242 CORNELL CORNELL LAW REVIEW note 30, at 129 (“As referees of science-in-the-making, note 30; see also James R. Dillon, Expertise on Trial, 19 is a step in the right direction, but a focus on is a step in the right direction, supra . L. R 243 supra ECH supra at least as currently applied, does not allow courts to applied, does not allow at least as currently . & T Jasanoff, CI Id. at 123–24; see also Jasanoff, Jasanoff, Jasanoff, (arguing that courts’ application of the See id. (arguing that courts’ application of the See . S As described in subpart II.A, there are research practices subpart II.A, there are research As described in OLUM 241 242 243 239 240 tice space for litigation science should “imitate the ideal litigation science should tice space for persuasively that she has argued scientific laboratory,” Daubert, ideal. approach that pears to accept the Daubert pears to accept process is not enough. contami- To make visible the “invisible in litigation science, the nation” caused by Analytical Flexibility in new ways, and up- creation process should be structured C on idealized, misleading, or misinformed assumptions about the scientific on idealized, misleading, or misinformed , method”); Jasanoff, Law’s Knowledge judges would focus on the process through which litigation science is generated rather than on its validity or invalidity.”). there are pregiven, determinate standards, extrinsic to what scientists themselves there are pregiven, determinate standards, competing claims.”). would invoke when confronted by specific, possess substantive expertise across all possible fields of scientific evidence and possess substantive expertise across all possible fields of scientific evidence therefore cannot be expected to interpret scientific expert testimony effectively). gues that when judges purport to apply scientific norms in in norms scientific to apply purport when judges that gues apply they necessarily evidence, of scientific their evaluation rather than norms are, those scientific idea of what their own field apply in the relevant actually norms that would scientific science. of academic that render research results fundamentally unreliable, in ways that render research results fundamentally examining the reported find- that are undetectable to anyone unless we rethink the ings, and that will remain undetected evidence is created and evalu- procedures by which scientific play in those procedures.ated, and the role that courts Jasa- scientific data, courts noff’s argument that in evaluating their creation as an indicator of should focus on the process of their reliability \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 42 570 25-MAR-21 9:54 overly idealized view of the scientific method. scientific of the view idealized overly ence operate within bounds and contexts that are different that are bounds and contexts within ence operate of Daubert another that interpretations enough from one ask judges to “think like scientists” have led to the creation of a like scientists” have led ask judges to “think “‘junk science works.” science’ of how majority “assumed that there is a well-articulated model of ‘good science’ Daubert majority “assumed that there is a well-articulated applied to offers of scientific evidence (the whose standards can be objectively myth of ‘scientific method’)”). (“[I]n an ironic turn, the ‘science’ that the Court officially embraced remained (“[I]n an ironic turn, the ‘science’ that the Court officially embraced remained con- profoundly a creation of the law’s own biases, needs, and misconceptions cerning scientific inquiry . . . .”). 42705-crn_106-3 Sheet No. 24 Side B No. 24 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 25 Side A 03/25/2021 10:13:22 R 571 571 This im- 244 Just as academic sci- 246 LITIGATION SCIENCE SCIENCE LITIGATION note 30, at 123. supra the lessons from the Replication Crisis should be the lessons from the Replication 245 See supra subpart II.A. Jasanoff, See supra subpart II.B. The institutional framework in which litigation science is The institutional framework The social academic sciences are reforming the way in The social academic sciences It is now understood that research results can be relied results can be that research understood It is now M K 244 245 246 proved understanding of what is needed to allow a reviewer to of what is needed to proved understanding what information validity should inform assess a study’s science for to courts evaluating litigation should be available evidence.admission into this information available to Making the generation parties, in turn, requires courts and opposing earlier in the re- of the required information and preservation search process. a traditional view, in which Even under the ]” and adjudicate the to “think like [a] scientist[ judge’s role is through the application of reliability of scientific evidence corresponding field of academic norms and criteria from the science, \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 43 25-MAR-21 9:54 ences are redesigning the research process with publication in ences are redesigning the research to creation, so too should mind, working back from publication be rethought to ensure that the litigation science processes desired research results, but process generates not only the to assess their reliability.also the information necessary In Flexibility and the impor- light of the invisibility of Analytical evidence for reliability, there tance of screening scientific way in which research is con- should be a reevaluation of the ducted in connection with litigation. this created is in some ways uniquely suited to bring about cultural shift—in some respects more suited than academic science. While academic scientists remain largely free to con- - duct their work any way they want and journals and organiza tions can at most encourage adherence to updated norms and instructive. science ex- When the reference field of academic and undergoes a large-scale cul- periences a knowledge crisis the process of litigation science tural shift, participants to ought to take note. by redesigning the chain of which science is being conducted end. creation starting from the back 2021] 2021] - is gath that of information the type guide should norms dated process. to the creation with respect and disclosed ered set of by a well-defined is governed if their creation upon only if their hypothesis, by a predetermined and guided procedures that types of information by certain is accompanied reporting evaluation and if their their reliability, reader to assess allow a indicia of Analytical Flexibility. takes into account C Y 42705-crn_106-3 Sheet No. 25 Side A No. 25 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 25 Side B 03/25/2021 10:13:22 R R M K C Y [Vol. 106:529 106:529 [Vol. Conducting some Frye, and other) 249 Litigation science also science Litigation 247 Daubert, note 232, at 120 (“A competent attorney, supra CORNELL CORNELL LAW REVIEW certain conduct. certain require Early Proceedings on Proposed Litigation Science Early Proceedings on Proposed See infra subpart III.B. See, e.g., Boden & Ozonoff, See supra Part I and note 54. 248 The consequences of contaminated methodology forming of contaminated methodology The consequences of the process by which This subpart begins a rethinking 1. courts typically assess the In current litigation practice, 247 248 249 or all of the reliability assessment much earlier in the process or all of the reliability assessment change that courts could make in would be the most impactful Cri- recognition of the problems uncovered in the Replication sis. By adopting an early and/or staged process for admissibil- the basis of expert testimony can be serious.the basis of expert - If the contamina detected, a court methodology cannot be tion in a presented the jurisdiction’s that the methodology meets may determine on the methodol- and that testimony based reliability standard, ogy—testimony unreliable ex- may present fundamentally that were reliable—isperimental results as if they therefore admissible. presented, and evaluated for ad- scientific evidence is created, missibility. that could be It describes a number of measures ( taken within the existing of the reliability of proposed ex- frameworks for the evaluation aimed at ensuring the rig- pert evidence: (1) early proceedings science; (2) updated and orous planning of proposed litigation requirements; and (3) consider- more detailed expert disclosure as part of a court’s evaluation of ation of Analytical Flexibility evidence. the reliability of proposed expert shortly before or during trial, admissibility of expert evidence fully formed opinions and the through a review of the expert’s opinions. expert’s stated bases for those \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 44 572 25-MAR-21 9:54 - can af courts science, to litigation comes when it standards, firmatively holds a greater potential for peer enforcement: while in aca- while for peer enforcement: greater potential holds a to peer norms is allocated of some enforcement demic science reviewers—typically likely and therefore anonymous, unpaid, review—in scrupulous to perform in their motivation varying is allo- of norms of the enforcement science, some litigation litigants—acated to opposing motivated to find group uniquely bring them to adversaries’ experts’ work and flaws in their light. aided by competent experts, should be in a better position to expose the flaws in aided by competent experts, should be in a better position to expose the flaws takes [an opposing side’s expert’s] research than is the peer reviewer, who often probe less time than the expert in a legal case and has more limited resources to than does the cross-examining attorney.” (citation omitted)). 42705-crn_106-3 Sheet No. 25 Side B No. 25 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 26 Side A 03/25/2021 10:13:22 R R R R 253 To 573 573 250 The inter- 254 note 162. supra By requiring an expert to de- By requiring 252 note 30, at 129. LITIGATION SCIENCE SCIENCE LITIGATION - Analytical Flexibility would be this remaining supra 251 Nosek, Ebersole, DeHaven & Mellor, Jasanoff, See supra notes 195–204 cited therein. and sources See supra subpart II.B and note 204. See supra subpart II.B. See See Early proceedings on the reliability of proposed expert Early proceedings on the reliability a process whereby the par- Sheila Jasanoff has argued for In substance, early admissibility proceedings could mirror admissibility proceedings In substance, early M K 250 251 252 253 254 scribe her intended study in detail, including every variable she study in detail, including scribe her intended step, and the every envisioned analytical intends to collect, and in interpreting to use in excluding data criteria she is going it to deter- court gains visibility that enables the results, the flukes or “real” results. mine whether results are statistical the extent that any aspects of the protocol were left open for were left of the protocol that any aspects the extent decision later, ventions suggested here follow that approach, but go further. ventions suggested here follow discovery, in this model, is not The court’s role during expert discovery disputes be- limited to that of a referee adjudicating dis- tween litigants, but lies primarily in ensuring that experts close and commit to a hypothesis and research plan up front. By routinely facilitating early proceedings relating to litigation methodology could take a variety of forms.methodology could take a variety could be con- They entirely between the parties ducted by the court or happen without the court’s involvement. be voluntary or They could mandatory. element or not, They could involve an adjudicative or not.with a presumption of admissibility main purpose The device: just as in aca- of early proceedings is as a commitment forces a researcher to commit to demic science preregistration research plan, so would early a predetermined hypothesis and evidence force an expert witness proceedings relating to expert to make the same commitment. negotiations about protocols to be ties engage in discussions or ]” not of the validity as a “referee[ followed, with the judge acting agreed-upon methodology but of or correctness of the resulting to arrive at it. the process the parties employed \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 45 25-MAR-21 9:54 come visible. adopted in many practices that are being the preregistration science. fields of academic 2021] 2021] largely the currently contain could courts ity determinations, Flexibility. of Analytical dimension invisible - an ex Requiring (b) a scope hypothesis; unambiguous commit to (a) an pert to would radi- protocol a detailed research project; and (c) for the HARKing, prevent Analytical Flexibility, the expert’s cally limit for p-hacking. the opportunities limit and significantly C Y 42705-crn_106-3 Sheet No. 26 Side A No. 26 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 26 Side B 03/25/2021 10:13:22 R R R R M K C Y note 25, at 390 [Vol. 106:529 106:529 [Vol. supra note 162, at 2602 (an ideal supra In practice, this plan would In practice, this 256 Chin, Growns & Mellor, standard, these hearings would take the standard, these hearings would . P. 26(a)(2)(B)(i)–(ii); 3101(4)(d)(1) (Mc- N.Y. C.P.L.R. IV . P. 194.2(f)(3). CORNELL CORNELL LAW REVIEW IV . R. C Indeed, the report could take the form of a could take the form of Indeed, the report . R. C ED 257 EX 255 Nosek, Ebersole, DeHaven & Mellor, Cf. notes 196–197 cited therein (indicating cultural shift to- and sources See See, e.g., F note 98, at 11 (preregistration should include full specification of study In a process directed by the court, a party engaging an a party engaging by the court, directed In a process Submitting a report stripped of a results and conclusions stripped of a results and Submitting a report element of early proceedings Adjudication is an optional 255 256 257 likely resemble the description of the methodology in the expert the description of the methodology likely resemble submitting later in expert would ordinarily be report that the the process. ward preregistration in social sciences). expert witness would submit a proposed plan to the court at an plan to the a proposed would submit expert witness out the expert has carried before the in discovery, early stage work. in- report, the plan would Similar to a preregistration hypothesis and description of the expert’s clude a detailed of data collec- expert plans to take in the course every step the and analysis. tion, cleanup, \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 46 574 25-MAR-21 9:54 expectation an expectation shift: cultural about a bring can courts science, and commitments necessary the make experts that disclosures. typical expert report with the “results” and “conclusions” sec- report with the “results” and typical expert tions omitted. (before experiments have com- section early in the process experiments and analysis have menced) rather than later (after than just the timing.been completed) changes more In sub- up front, the expert com- mitting a methodological description and binds her own hands mits to the described methodology or experiment with different against the ability to improvise methodological steps. a presub- From the court’s perspective, expert followed a predetermined mitted plan ensures that the hypothesis, whereas a post- set of steps to test a preselected guessing at how much of the submitted report leaves the court was determined up front and expert’s methodology and theory the way. how much was decided along relating to litigation science. of the ex- Following submission could hold a Daubert-type hear- pert’s proposed plan, the court ing to evaluate the plan. the state courts In federal courts and applying the Daubert on the admissibility of scien- same shape as ordinary hearings tific testimony. Just as in a regular Daubert hearing, the court (here: proposed) methodology for would evaluate the expert’s Kinney 2014); T (discussing voluntary preregistration by expert witnesses). design and analysis plan); see also research plan should include all steps to be taken); Open Sci. Collaboration, research plan should include all steps to be taken); Open Sci. Collaboration, supra 42705-crn_106-3 Sheet No. 26 Side B No. 26 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 27 Side A 03/25/2021 10:13:22 R 575 575 note 232, at 43 supra At this stage, the only At this stage, Joiner hearing, after the second 261 Daubert’s key distinction between the A ruling that incorporates this A ruling that incorporates this 262 263 Daubert, holding that part of a trial court’s This opportunity typically will have This opportunity Id. at 146; see also Haack, 259 258 LITIGATION SCIENCE SCIENCE LITIGATION A second, generally much more limited, hearing much more limited, A second, generally Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 593–94Daubert v. Merrell Dow Pharm., Inc., (1993). 260 A party opposing the admission of the expert’s opinions would, of course, See, e.g., Fed. Hous. Fin. Agency v. JPMorgan Chase & Co., 11 Civ. 6189 See See supra subpart II.A. See id. a The later-stage hearing could be termed A court issuing an early ruling on a proposed methodology A court issuing an early ruling By conducting gatekeeping proceedings early in the pro- early in the proceedings gatekeeping By conducting M K 262 263 258 259 260 261 disappeared by the time an ordinary admissibility hearing the time an ordinary admissibility disappeared by and fully ana- data have been gathered takes place: after lyzed. still be free to make objections on the basis of relevance under the jurisdiction’s still be free to make objections on the basis of relevance under the jurisdiction’s applicable rules of evidence. (DLC), 2012 U.S. Dist. LEXIS 173768, at *37–38 (S.D.N.Y. Dec. 3, 2012) (granting with a well-defined set of steps and goals would do well also to with a well-defined set of steps types of arguments still availa- offer a precise description of the opposing an expert’s proposed ble at a later stage to parties plan. any argument For example, the court might rule that at the early hearing is waived, that the party could have made not made and could not have but that any argument that was been made—because information that came to light it relies on later—will remain available. cess, a court can evaluate the methodology separately from the evaluate the methodology cess, a court can the methodology the process of executing results and before and inject unrelia- to alter the methodology has had a chance process. bility into the \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 47 25-MAR-21 9:54 questions before the court would be whether the expert applied the court would be whether questions before exactly as described, and the pre-approved methodology the pre-approved methodology whether any departures from methodology to a sufficiently se- reduced the reliability of the expert’s conclusions inadmissible vere degree as to render the under the Daubert standard. would take place at the end of expert discovery, after all data at the end of expert discovery, would take place and analyzed. have been gathered 2021] 2021] or known the methodology’s as such of reliability, indicators can be methodology the whether and rate, error potential review, has to peer has been subjected has been tested, tested, the relevant accepted” in is “generally and/or been published, community. scientific case in the Daubert trilogy. General Elec. Co. v. Joiner, 522 U.S. 136 (1997). In between methodology and conclu- Joiner, the Court walked back a full separation sions that had been read into - gatekeeping role is to be on guard for conclusions that have too tenuous a connec the tion to the methodology, where there is “too great an analytical gap between data and the opinion offered.” (arguing that Joiner “quietly jettisoned methodology an expert uses and the conclusion he reaches”). C Y 42705-crn_106-3 Sheet No. 27 Side A No. 27 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 27 Side B 03/25/2021 10:13:22 M K S. C Y see HARLES C [Vol. 106:529 106:529 [Vol. RANSACTIONS , 50 T In a small number of other financial-crisis In a small number of other financial-crisis CORNELL CORNELL LAW REVIEW 265 In addition, requiring the opposing party to make the opposing party In addition, requiring 319, 322 (2014) (“[B]locking the way of inquiry is much worse than 319, 322 (2014) (“[B]locking the way Y ’ 264 OC , Nat’l Credit Union Admin. Bd. v. RBS Sec., No. 11-2340-JWL, See, e.g., Nat’l Credit Union Admin. Bd. v. RBS Susan Haack, Do Not Block the Way of Inquiry S Early hearings relating to the admissibility of expert opin- Early hearings relating to the MASTR Adjustable Rate Mortgs. Tr. 2006-OA2 v. UBS Real Estate Sec., Inc., MASTR Adjustable Rate Mortgs. Tr. 2006-OA2 v. UBS Real Estate Sec., Inc., EIRCE 265 264 No. 12-CV-7322 at *1 (S.D.N.Y. Mar. 11, 2015) (plaintiffs submitted a letter appli- No. 12-CV-7322 at *1 (S.D.N.Y. Mar. 11, 2015) (plaintiffs submitted a letter cation for early proceedings regarding their expert’s proposed sampling study); Mass. Fed. Home Loan Bank of Bos. v. Ally Fin., Inc., No. 11-10952-GAO, at *3 (D. Meth- May. 1, 2014) (plaintiff filed a “Motion for Approval of Statistical Sampling 165, odology”); Mass. Mut. Life Ins. v. Residential Funding Co., 989 F. Supp. 2d 2014 U.S. Dist. LEXIS 59893, at *7 (D. Kan. Apr. 30, 2014) (plaintiff submitted 2014 U.S. Dist. LEXIS 59893, at *7 (D. sam- early in limine motion on reliability of expert testimony based on proposed pling scheme); MBIA Ins. Corp. v. Countrywide Home Loans, Inc., No. 602825/08, The 2010 N.Y. Misc. LEXIS 6182, at *9 (N.Y. Sup. Ct. Dec. 22, 2010) (same); People’s Notice of Motion in Limine Re Sampling Methodology, People v. Morgan Stanley & Co., No. CGC-16-551238 (Cal. Super. Ct. Feb. 15, 2018) (same); also preapproval to the plaintiff’s expert’s proposed methodology, but reserving de- preapproval to the plaintiff’s expert’s that had not yet been available to it at the fendants’ rights to file later challenges preapproval stage). \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 48 576 25-MAR-21 9:54 type of specification would not visit injustice upon litigants on on litigants upon injustice not visit would of specification type side.either in to be diligent opposing party it forces the While discovery, of expert at an early stage its interest defending of its argu- preserve some dry and keep its powder rather than of fully it stops clear round of briefing, a subsequent ments for and its methodology into the expert’s the inquiry” “blocking develop - of subsequent stage in light at a later application ments. mere inefficiency . . . to set up a philosophy which barricades the road of further mere inefficiency . . . to set up a philosophy offense in reasoning . . . .”). advance toward the truth is the one unpardonable P any argument available to it at this early stage (or waive it available to it at this early any argument with full disclo- full disclosure of methodology forever) balances in opposition.sure of arguments It increases the transparency of the early eviden- the process and the utility and efficiency of tiary battle. option should describe its A court choosing this it very precisely, to arguments foreclosed by decision and the meaningless by a from being rendered prevent the ruling by dressing an attempt to reargue its position skilled litigant’s old argument in new facts later. thus far they have been re- ions are not a novel idea, though quested—typically by plaintiffs—on different grounds. In a the 2008 Financial Crisis, plain- handful of cases arising out of process sought ad- tiffs facing an expensive expert-discovery of their expert’s proposed vance approval from the court would be unreasonable and un- methodology, arguing that it to incur the expense of an duly burdensome for these plaintiffs expert analysis with- unusually extensive and time-consuming resulting findings would end up out some assurance that the being admissible. 42705-crn_106-3 Sheet No. 27 Side B No. 27 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 28 Side A 03/25/2021 10:13:22 R 577 577 see also But cf. MASTR 268 , 984 F. Supp. 2d at *2 (acknowledging See also infra note 270. Daubert pre-clearance” as “primarily [a , at *1–2 (declining to grant early approval , 2014 U.S. Dist. LEXIS 59893 at *1 (granting preap- , 2014 U.S. Dist. LEXIS 59893 at *1 (granting Fed. Home Loan Bank of Bos. LITIGATION SCIENCE SCIENCE LITIGATION Most courts faced with early evidentiary motions motions evidentiary with early faced courts Most , 2010 N.Y. Misc. LEXIS 6182, at *15 (granting MBIA Ins. Corp., 2010 N.Y. Misc. LEXIS 6182, at *15 have been willing to engage in early proceedings on on in early proceedings willing to engage have been 266 267 In re Countrywide Fin. Corp. Mortg.-Backed Sec. Litig. See, e.g., , 2012 U.S. Dist. LEXIS 173768, at *33 See, e.g., Fed. Hous. Fin. Agency, 2012 U.S. Dist. LEXIS 173768, at *33 investments in securities backed These cases all involved disputes about Countrywide Fin. Corp. Mortg.-Backed Sec. Litig., 984 F. Supp. 2d In re Countrywide Fin. Corp. Mortg.-Backed As an alternative to court-run proceedings, a mechanism a mechanism proceedings, to court-run As an alternative M K 268 266 267 that the court had the power to grant “ the matter] of case management,” but declining to do so in this instance, because court “likely . . . would still need to revisit the methodology, and its application” after the expert had issued a final report). plaintiff’s early motion in limine, noting that a court has the ability to “regulate plaintiff’s early motion in limine, noting expedient resolution of the case before it the conduct of the trial” to promote an was “within [that] inherent power”); and that deciding the motion before it (ordering the parties to submit early expert-study proposals after defendants (ordering the parties to submit early sampling methodol- brought a Daubert challenge to plaintiff’s expert’s proposed ogy); 1021, 1025 (C.D. Cal. 2013) (same). retained by the plaintiff(s) to examine by large pools of mortgage loans and experts these pools—aa representative sample of loans from process that was commonly even when applied only to a understood to be expensive and time-consuming limited sample. in the Although I am not aware of early-approval requests made the potential utility of an early admission context of different types of disputes, cases.determination is not unique to mortgage-backed-securities aside Even costly and/or time-consuming expert from reliability concerns, any case involving request.analysis could be a venue for a similar mecha- Similarly, no procedural am not aware of any nism bars defendants from making similar motions, but I this step. case in which defendants have taken Nat’l Credit Union Admin. Bd. methodology, but reserving defendants’ proval to the plaintiff’s expert’s proposed , 2012 U.S. Dist. LEXIS rights to file later challenges); Fed. Hous. Fin. Agency 173768, at *36–37, 61–62 proposed sampling plan (pre-approving plaintiff’s , 989 F. Supp. 2d at 165 (deny- before it had been executed); Mass. Mut. Life Ins. ing, at the proposal stage, motion to exclude expert testimony, but without prejudice); at 1021 (ordering parties to submit proposed sampling plans). Adjustable Rate Mortgs. Tr. 2006-OA2 be a because it “would either be a non-binding, advisory opinion . . . or it would a binding ruling made on an inadequate record at the wrong juncture in the case, shot from the judge’s hip”); for early exchanges of information could be orchestrated by the be orchestrated information could exchanges of for early parties.- confer without the court’s in Parties could meet and to discuss at the court’s encouragement, volvement, possibly they plan to to be performed by any experts proposed work retain. detail below, both proponents As discussed in more to gain from early of an expert’s testimony tend and opponents expert work. proceedings relating proposed formal or informal to pin down a pro- benefits from an opportunity The opponent stage, blocking the methodology at an early ponent’s expert’s 172 (D. Mass. 2013) (magistrate judge recommending that the court set a briefing (magistrate judge recommending that 172 (D. Mass. 2013) following a petition by the plaintiff). schedule for early-determination proceedings, \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 49 25-MAR-21 9:54 of this ilk the admissibility of proposed expert methodology. expert of proposed the admissibility 2021] 2021] of the initiative at took place proceedings early similar cases, court. the C Y 42705-crn_106-3 Sheet No. 28 Side A No. 28 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 28 Side B 03/25/2021 10:13:22 R R M K C Y 269 Fed. An 271 . 99, 108 First, a EV Id. 270 . L. R [Vol. 106:529 106:529 [Vol. Law’s Knowledge, LB , 64 A Jasanoff, Weisgram v. Marley late-stage See note 56, at S64 (“[T]here is little

Navigating Expert Reliability: Are Daubert motions to exclude defendant’s early stage is not precluded. . P. 16(b)(3)(B) (providing that a scheduling order may . P. 16(b)(3)(B) (providing that a scheduling IV CORNELL CORNELL LAW REVIEW . R. C ED Weisgram v. Marley Co., 528 U.S. 440, 455–56 (2000) (stating that See See, e.g., F testimony are brought much In general, challenges of proposed expert There are powerful structural reasons for a party who has structural reasons for a There are powerful Participation in court-run proceedings or party-orches- court-run proceedings or Participation in 271 269 270 has been invoked most often to exclude plain- supra note 232, at S50 (“Daubert has been invoked most often to exclude tiffs’ testimony . . . .”); D. Michael Risinger, their after expert has been excluded shortly before trial, plaintiffs cannot salvage case by substituting a new expert). While under exclusion of an essential expert is typically fatal to the plaintiff’s ability to prevail (or on a claim that relies on the expert’s testimony, substitution of a new expert methodology) following exclusion at an more frequently by defendants than by plaintiffs. more frequently by defendants than by Criminal Standards of Certainty Being Left on the Dock? unfavorable early ruling, on the other hand, might inform a unfavorable early ruling, on for litigating its case is fatally plaintiff that its planned strategy decisions about its contin- flawed, enabling it to make informed favorable decision on the motion reduces the risk that the ex- favorable decision on the motion analytical work, only pert performs expensive, time-consuming it at the end of the proceed- to be barred from testifying about ing. reduces the For a plaintiff, a favorable ruling immediately its suit close to trial (or risk it faces of having to abandon weakened form), as a conse- continuing its suit in substantially barred from testifying. quence of a crucial expert being But as discussed below, parties may well be willing to submit to below, parties may well be But as discussed the process in- voluntarily, especially when such a process step. cludes an adjudicative ruling on the ad- to seek an early (partial) retained an expert on extensive or expert testimony that relies missibility of future for plaintiffs. expensive analysis, in particular \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 50 578 25-MAR-21 9:54 expert’s ability to exploit Analytical Flexibility to his or her or her to his Flexibility Analytical exploit to ability expert’s later.benefit to hammering may be amenable The proponent for the op- in exchange protocol agreed-upon research out an - expert on methodologi not to oppose the commitment ponent’s later.cal grounds their memorialize parties could The the court. to be filed with optionally in a stipulation, agreement by the court. could be voluntary or mandated trated exchanges that allow them case-management powers Courts have broad methodology. early process regarding expert to mandate an (2000) (observing that almost ninety percent of expert challenges in civil cases are (2000) (observing that almost ninety percent of expert challenges in civil cases brought by defendants); see also Berger, supra point in plaintiffs going to the expense of experts until they know if their case will proceed.”). , at *2 (stating that the court has the power to grant Home Loan Bank of Bos., at *2 (stating that the court has the of case management”). “Daubert pre-clearance” as “primarily [a matter] modify the timing and nature of disclosures and the scope of discovery); modify the timing and nature of disclosures 42705-crn_106-3 Sheet No. 28 Side B No. 28 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 29 Side A 03/25/2021 10:13:22 Id. 579 579 Id. at *7. Be- Id. at *10. The court reviewed the First, an early exchange might , a favorable ruling on an ruling , a favorable Second Id. at *7, *10. The plaintiff now could either , early proceedings relating Fourth, early proceedings 272 273 , by opposing an expert’s proposed , by opposing an expert’s all loans at issue in the case or abandon its claims. LITIGATION SCIENCE SCIENCE LITIGATION Second 274 - expert-discov with a pre-approved Third, a plaintiff On the other hand, it is possible that, on the whole, litigation funders will See Weisgram, 528 U.S. at 455–56 (holding that plaintiffs cannot salvage Homeward Residential Inc. v. Sand Canyon An illustrative example is Homeward Residential Inc. v. Sand Opposing parties stand to gain from early exchanges relat- Opposing parties stand to gain M K 273 274 272 contractual provisions at issue and concluded that a finding of liability would contractual provisions at issue and concluded that a finding of liability require a breach determination for each individual loan at issue. the cause the proposed sampling analysis would yield only aggregate information, court rejected the sampling proposal. proceed with an analysis of at *10. to an expert’s proposed methodology may afford a party a low- methodology may afford to an expert’s proposed that is on to gain approval of a methodology risk opportunity therefore perhaps end of the spectrum (and the less rigorous opt for a more and/or time-consuming), and less expensive option ends up only if the less rigorous rigorous methodology being rejected first. toward the se- This incremental approach not available when admissibility lection of a methodology is end of expert discovery, shortly decisions are made at the before trial. be willing to fund a smaller number of cases, choosing to wait and see which cases be willing to fund a smaller number of cases, choosing to wait and see which survive the discovery-plan hurdle. their case by submitting an updated expert report or substituting a new expert). , No. 12 Civ. 5067 (JFK), 2017 WL 5256760 (S.D.N.Y. Nov. 13, 2017).Corp., No. 12 Civ. 5067 (JFK), 2017 WL 5256760 The to prove its contract claims by analyz- plaintiff petitioned the court for permission of loans. ing a sample of loans from a large pool ing to expert discovery as well. ing to expert discovery as well. help the party refine its strategy. to productive It may also lead about the way discovery will negotiations between the parties be structured. the party has an opportunity to methodology at an early stage, by the expert.influence the work to be performed In particular, the expert’s ability to deploy Ana- it has an opportunity to limit advantage.lytical Flexibility to his or her more a party can The the expert is to obtain results limit this ability, the less likely gate, but that in fact cannot that can pass through the Daubert \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 51 25-MAR-21 9:54 early expert motion immediately strengthens a party’s position position party’s a strengthens immediately motion expert early negotiations.in settlement case cor- value of a The settlement at trial, will prevail that the plaintiff with the likelihood relates expert of critical on the admissibility turn depends which in evidence. options. litigation finance access to may have easier ery plan take on the litiga- may be more willing to Litigation financiers that risk—a one significant component of tion risk when criti- at trial—has expert being barred from testifying cally important already been eliminated. 2021] 2021] action. of the litigation ued C Y 42705-crn_106-3 Sheet No. 29 Side A No. 29 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 29 Side B 03/25/2021 10:13:22 M K C Y [Vol. 106:529 106:529 [Vol. , as with the party proffering the ex- the proffering party with the , as Third CORNELL CORNELL LAW REVIEW 275 276 See supra subpart II.A. A limitation along these lines could be imposed in conjunction with the A court could impose conditions on a party seeking an a party seeking conditions on could impose A court to plan multiple experiments To protect an expert’s ability 275 276 expert’s early disclosure of a research protocol, with metes and bounds carefully expert’s early disclosure of a research protocol, with metes and bounds carefully delineated based on the expert’s specific plans. An expert’s selective reporting differ- could be relatively harmless in a scenario where the expert performed three ent experiments, each with a different goal and different experimental approach, and chose to present just one result at trial. In this scenario, barring the opposing party from inquiring about the two unreported experiments does not necessarily raise publication bias or p-hacking concerns; the three experiments are likely confi- sufficiently independent that selective reporting would not falsely inflate the dence level for the single reported study. In contrast, a scenario whereby an and expert tried 100 different analyses, all aimed at proving the same hypothesis, \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 52 580 25-MAR-21 9:54 be relied upon. be relied - im would negotiations in settlement position the party’s pert, ruling. obtained a favorable if it prove considerably by the a commitment it could require ruling: early admissibility methodol - evidence to the expert confine its proposed party to expand the scope described: that is, (1) not to ogy and analysis not to retain a dif- proposed testimony, and (2) of the expert’s topics. testify on the same or similar ferent expert to a Such the other side with party’s ability to surprise restriction on the the litigation may evidence at a later stage of unexpected expert as well, and the appealing to the other side make the tradeoff to submit the able to reach mutual agreement parties might be along these lines. to the court on conditions expert’s proposal that prefers not to a voluntary process, a party In the case of instead to retain a compromise, preferring bind itself to such of its expert testimony in the its ability to expand the scope motion and take its chances at future, is free to forego an early end of discovery, per the more an admissibility hearing at the traditional procedural sequence. risk being This party would on grounds faced with a Daubert (or equivalent) motion later, were contaminated by Ana- that the results its expert obtained unreliable.lytical Flexibility and therefore that is cost- A party may well be willing to take this conscious, on the other hand, on how it can prosecute its bargain, even if it results in limits case. of fact only some results but but choose to present to the finder an opposing party’s ability to not others, courts could limit dropped from the expert’s reference precleared experiments presentation. be carefully con- Such a limitation would have to room it provides for file-drawer sidered, however, in light of the problems. 42705-crn_106-3 Sheet No. 29 Side B No. 29 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 30 Side A 03/25/2021 10:13:22 R R R 581 581 note 162, at 2604 (explain- 278 supra LITIGATION SCIENCE SCIENCE LITIGATION a court could consider expert testimony expert testimony could consider a court While this option would deny the parties the would deny the parties the While this option 277 279 Nosek, Ebersole, DeHaven & Mellor, See supra note 208 and sources cited therein. Cf. Cf. supra note 206 and sources cited therein (describing private, locked, A more limited intervention would have a party “prere- intervention would have a A more limited As understanding of Analytical Flexibility expands, courts As understanding of Analytical early proceedings are not a It is important to recognize that An early ruling by the court could be binding or nonbinding nonbinding or be binding could the court by ruling An early M K 277 278 279 based on a presubmitted protocol to be presumptively admissi - be presumptively protocol to a presubmitted based on described out the plan as had carried the expert ble, provided study irrel- rendered the had not developments and intervening evant. reduce the parties’ freedom and An early ruling does not to the finder to present their expert’s opinions narrative license of their choosing. of fact in a manner and “self-archiving” options for preregistration). gister” its expert’s plans with the court before execution, under plans with the court before gister” its expert’s (or even neces- disclosure to opposing parties seal and without or other feed- and without any adjudication sarily the court), back process. chose to present only the most favorable analysis, raises serious reliability chose to present only the most favorable analysis, raises serious reliability concerns. \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 53 25-MAR-21 9:54 attendant benefits of early exchanges of information, it would of early exchanges of information, attendant benefits of early proceedings: to be a serve the most important function commitment device. presented their By the time the expert the party could rely on the completed work for admissibility, that the expert’s study tested a sealed filing to demonstrate had been conducted according predetermined hypothesis and to a predetermined protocol. scientific evidence to submit to could require proponents of as a prerequisite for having the early reliability proceedings evidence later.expert’s opinion admissible into Participation in an expert’s credibility and might early proceedings could boost being attacked on grounds of protect the expert’s opinions from p-hacking and HARKing later. panacea. simple to be Some research projects are simply too amenable to this process.proposed study is In cases where the expensive, an expert could make neither time consuming nor process by performing an end-run around the preregistration needed to figure out the parame- the study as many times as ters that yield favorable results, then submit a plan consisting 2021] 2021] or future admissibility of in a presumption result could and not. a prere- publish will presumptively as some journals Just study, gistered ing that preregistration does not deprive researchers of their “narrative license” in ing that preregistration does not deprive researchers of their “narrative license” explaining and presenting their data). C Y 42705-crn_106-3 Sheet No. 30 Side A No. 30 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 30 Side B 03/25/2021 10:13:22 R R R R M K C Y 286 [Vol. 106:529 106:529 [Vol. supra note 98, at 17 Courts should de- Courts should 281 283 Yet the fact that early that the fact Yet To determine whether an To determine 280 282 . 131, 131 (2014) (“[A]ttorneys can do data . 131, 131 (2014) (“[A]ttorneys can do EV note 137, at 3–4 (noting that a researcher can See, e.g., Jonah B. Gelbach, Expert Mining and . L. R HI . P. 26(a)(2)(B)(i)–(ii); 1301(4)(d)(1) (Mc- N.Y. C.P.L.R. IV . P. 194.2(f)(3). CORNELL CORNELL LAW REVIEW IV . R. C . R. C ED EX If increased disclosure requirements are combined If increased disclosure requirements Courts could use checklists to guide these disclo- Courts could use checklists Updated Disclosure Requirements Disclosure Requirements Updated 285 See supra section II.B.2. See, e.g., F Cf. supra notes 216–217 cited therein (arguing that research- and sources Cf. supra note 217 and sources cited therein (describing use of checklists Cf. Allen & Mehler, supra When a study is truly inexpensive and quick to perform, a party might even When a study is truly inexpensive and parties to sign a statement Courts could require experts and retaining 284 2. abil- the court’s that would improve intervention A second 282 283 284 285 286 280 281 mand detailed disclosures of all analytical steps performed by mand detailed disclosures of from a predetermined research the expert and all deviations plan. make changes to a preregistered plan, but should justify them and discuss them make changes to a preregistered plan, but should justify them and discuss in the resulting publication); Open Sci. Collaboration, ers should disclose all data collected, all analytical steps made, and all departures ers should disclose all data collected, all analytical steps made, and all departures from their preregistered plan). by scientific journals). (“[D]etail what was changed and why.”). ask multiple experts to perform an analysis and retain the expert who performed ask multiple experts to perform an analysis the most favorable analysis. Required Disclosure, 81 U. C performed prior to the filing of the expert’s disclosing what expert work had been research plan. for This, too, would present opportunities for evasion, however, experts, whose identity, retention, and example through the use of consulting work product are typically not discoverable. mining’s dirty work by hiring multiple experts, asking each to provide an expert mining’s dirty work by hiring multiple on the stand only the one who provides the report on the same issue, and then put most favorable report.”). ity to accurately determine the reliability of litigation science determine the reliability of ity to accurately just as journals at the disclosure stage: could be implemented more detailed science, the court could require do in academic expert reports.disclosures in Just as in academic science, by Ana- a study has been contaminated determining whether of the process by requires detailed knowledge lytical Flexibility were produced. which the results \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 54 582 25-MAR-21 9:54 of those parameters to the court. to the parameters of those proceedings are not useful in all cases should not prevent them them prevent not should in all cases useful are not proceedings in many cases. a feature from becoming expert’s opinions are reliable enough to be presented to the are reliable enough to be expert’s opinions more detail than is court needs considerably finder of fact, the in an expert report. typically disclosed with early proceedings in which the expert commits to a hy- with early proceedings in which the disclosures would be fairly pothesis and research plan, either confirm that he per- straightforward: the expert would at the outset or provide formed the plan exactly as described departures from the plan. detail and justification for any sures. Kinney 2014); T 42705-crn_106-3 Sheet No. 30 Side B No. 30 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 31 Side A 03/25/2021 10:13:22 R see 583 583 The 289 As measures to 290 See Chin, Psychological Science’s The Daubert court offered a number of LITIGATION SCIENCE SCIENCE LITIGATION 288 - scientific evidence for admissibil Courts evaluating Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 593–94 (1993); Consideration of Analytical Flexibility Consideration of 287 See supra subpart II.A. See Jason Chin has argued that Analytical Flexibility issues can be assessed as Frye v. United States, 293 F. 1013, 1014 (D.C. Cir. 1923). The prevailing legal frameworks used to assess the reliabil- The prevailing legal frameworks Courts could even choose to go so far as to require the require as to so far to go choose even could Courts 3. courts need reliability determinations, Finally, in making M K 287 288 289 290 also supra Part I. part of Daubert’s “generally accepted” prong. Replicability Crisis, supra note 25. standard that is still applied in the courts of a small num- Frye standard that is still applied inquiry: whether the meth- ber of states focuses on a narrower gained “general acceptance in the ods used by the expert have ].” belong[ particular field to which [they] \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 55 25-MAR-21 9:54 factors a court might consider but stressed that they were not factors a court might consider or test.”intended as a “definitive checklist Flexibility Analytical be added to the set. is a consideration that could readily ity should focus not just on the methodological choices, but ity should focus not just on by which those choices were also on the timing and manner extent to which the process that made, and in particular the for Analytical Flexibility.produced the evidence left room Par- by bringing contamination by ties have a role to play as well attention. Analytical Flexibility to the court’s easily accommodate this dimen- ity of scientific evidence can sion of inquiry. to The Daubert standard instructs the court that produced scientific evi- assess whether the methodology by looking at a “flexible” set of dence is “scientifically valid” indicia of reliability. 2021] 2021] null results—results of reporting did that conducted tests of results or that yielded results statistically significant not yield party. to the retaining unfavorable that were would While this it is hard to process, more reliable a more transparent, result in this signifi- welcome litigants would that courts and imagine adversarial system. of the cant disruption and take into ac- of the Replication Crisis to heed the lessons in research re- Analytical Flexibility is playing count the role sults. relying on descriptions of At the gatekeeping stage, indicators of on reported p-values or other methodology and of research to assess the reliability confidence is insufficient results. contain Analytical Flexibility are becoming increasingly com- contain Analytical Flexibility mon in academic science, an inquiry regarding their presence or absence should organically become a part of a Frye inquiry. C Y 42705-crn_106-3 Sheet No. 31 Side A No. 31 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 31 Side B 03/25/2021 10:13:22 R M K C Y [Vol. 106:529 106:529 [Vol. a facilitator creates con- A party that did partici- A party that did 294 291 and unlike a referee who and unlike a referee who 293 IV MPLICATIONS I note 30, at 129. CORNELL CORNELL LAW REVIEW 292 supra Jasanoff, The Role of the Judge See supra subpart II.A. Id. See Daubert, 509 U.S. at 579–80. See The lessons from the Replication Crisis, if heeded by the The lessons from the Replication evidence requires an up- Ensuring the reliability of expert If a court invites parties to engage in voluntary early pro- early in voluntary engage to parties court invites If a A. 291 292 293 294 litigation community in the way suggested above, have far- litigation community in the role of a judge, the balance be- reaching implications for the by which we judge admis- tween the parties, and the standards sibility. I discuss each in turn. role in the expert discovery dated conceptualization of a judge’s process. of creation of By becoming involved in the process the judge’s role evolves from litigation science at an early stage, a facilitator.that of a gatekeeper to that of a gatekeeper Unlike in its completed form and who waits until evidence is presented adjudicates its admissibility, \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 56 584 25-MAR-21 an proffering party and the work expert regarding ceedings 9:54 an infer- could draw the court to participate, expert declines or -hacking tainted by p may have been the results ence that to be sufficiently reliable not be and may therefore HARKing, admissible. and re- to a hypothesis Without precommitment have a the party would stage plan, at the gatekeeping search that inference. hard time rebutting pate in early proceedings but then had its expert present addi- but then had its expert pate in early proceedings should raise the had not been precommitted tional data that that there may in equal measure on the basis court’s suspicion but did not work that the expert performed be additional choose to present. ditions for its generation. or invites When a court mandates execution—whetherfiling of an expert’s plan before publicly or with adjudication or not—thesealed, and whether combined most important role it plays is as a facilitator of a commitment device. Whether the court issues a ruling on the expert’s pro- posed plan is of secondary importance—of interest from a view- the point of efficiency and case management, but not from monitors the process of its creation, 42705-crn_106-3 Sheet No. 31 Side B No. 31 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 32 Side A 03/25/2021 10:13:22 R 585 585 297 o et al., supra note Munaf` See note 98, at 1362–63. - a commit (1) creating By 295 supra LITIGATION SCIENCE SCIENCE LITIGATION When the early exchange of expert plans is When the early exchange of 296 298 Daubert, 509 U.S. at 579–80; Frye v. United States, 293 F. 1013, 1014 The Balance Between Plaintiffs and Defendants The Balance Between Plaintiffs To create reliable, reproducible scientific knowledge, it is essential to con- See supra subpart II.B. E.g., See supra subpart III.B. If early proceedings include an adjudicative component, If early proceedings include If early proceedings include an adjudicative component, If early proceedings include Judges have the power to create a cultural shift toward the shift toward to create a cultural have the power Judges B. M K 295 296 297 298 ment device and (2) demanding more extensive disclosures, the the disclosures, extensive more demanding and (2) device ment to the type of information that allows a process court facilitates of a reliable assessment to make that is necessary be generated later. reliability structured as a voluntary process between the parties without structured as a voluntary process gatekeeping proceedings will still the court’s involvement, then have some bite. the arguments Even in that scenario, however, testimony may be significantly available to a party opposing the plan to an opposing party at curtailed: disclosure of an expert’s of that exact plan, should an early stage, followed by execution in p-hacking and HARKing off take any arguments sounding the table. between plaintiffs and de- this might affect the power balance fendants. Much has been written about the impact the Daubert strain Analytical Flexibility to the extent possible. 94, at 2; Simmons, Nelson & Simonsohn, (D.C. Cir. 1923). the eventual gatekeeping moment becomes a less momentous the eventual gatekeeping moment occasion. has been When the reliability of the methodology admissibility proceedings affirmed at an early stage, pretrial the expert carried out the plan will be limited to confirming that any deviations from the preap- as intended or to examining proved plan. use of more reliable methodology.use of more reliable They can encourage early of information. and broader disclosures information exchanges not yield the in- that current practice does By acknowledging that allow need, and by facilitating procedures formation they at appropri- and exchange of that information for the creation into being a pro- a litigation, judges can bring ate junctures in need to fulfill the the information they cess that will generate serving. they will ultimately be gatekeeping function by And that reaches can ensure that scientific evidence doing so, they of reliability. meets required standards the finder of fact \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 57 25-MAR-21 9:54 2021] 2021] reliability. of ensuring perspective C Y 42705-crn_106-3 Sheet No. 32 Side A No. 32 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 32 Side B 03/25/2021 10:13:22 R R R R R R M K , 8 C Y EDONA The The Fur- 299 301 Daubert, but [Vol. 106:529 106:529 [Vol. Admissibility Regimes: The . 699, 700 (2008) (contrast- supra note 219, at 988–92

EV Judge and Attorney Experiences, supra note 219, at 990 (“[O]n the . U. L. R As discussed in subpart III.B, As discussed in subpart III.B, W John Roberts, 2015 Year-End Report on note 270 (almost ninety percent of expert note 270 (almost ninety percent of expert 302 See Carol Krafka, Meghan A. Dunn, Molly see also Carol Krafka, Meghan A. Dunn, Molly , 36 S - note 36 (arguing that as standards of admissi supra has made it harder . . . to get scientific Daubert has made it harder . . . to get scientific supra note 56, at S64 (“Daubert has undoubtedly CORNELL CORNELL LAW REVIEW & L. 309, 322 (2002) (noting that judges reported having & L. 309, 322 (2002) (noting that Some commentators have argued that the pen- have argued that Some commentators Y ’ OL 300 was decided at the early dawn of large-scale the early dawn of large-scale Daubert was decided at . P UB Berger & Twerski, supra see also The Sedona Conference, Cooperation Proclamation, 10 S .P These exchanges may be envisioned by the 2015 amendments to Federal See generally Berger, See See, e.g., Berger, supra note 56, at S61–S63 of (surveying the impact Early information exchanges would offer an opportunity for Early information exchanges . J. 331, 331–32 (2009) (urging a “paradigm shift for the discovery process” Daubert and Frye admissibility regimes); Haack, ONF SYCHOL 302 300 301 299 challenges in civil cases are brought by defendants). challenges in civil cases are brought by thermore, that fol- and technological developments electronic discovery, have sharpened its bite.lowed the decision Early adjudication a swing back more civil actions would represent proceedings in of the pendulum. to expert discovery, as cooperative, cost-effective approaches for negotiations on the scope of well as early opportunities discovery or even settlement. Rules of Civil Procedure 1 and 16. whole, at least in civil cases, testimony admitted.”); Risinger, has become “nigh impossible” for certain bility have become more exacting, it plaintiffs to maintain a civil action); on toxic-tort litigation); David L. Faigman, Daubert on toxic-tort litigation); David L. Faigman, Exceptions to Scientific Evidence, the Scien- “Opinion Rule” and Other Oddities and tific Revolution, and Common Sense and defendants and made it more difficult shifted the balance between plaintiffs Haack, for plaintiffs to litigate successfully.”); Miletich, Treadway Johnson, Joe S. Cecil & Dean Testimony in Federal Civil Trials Practices, and Concerns Regarding Expert P decision is often said to have shifted some litigation risk toward toward risk litigation some shifted to have often said is decision on expert more reliant tend to be since plaintiffs plaintiffs: are in de- defendants their claims than to establish evidence of relia- higher standards having to meet against them, fending more than plaintiffs tends to affect expert testimony bility of defendants. toward a culture that promotes “open and forthright information sharing” and toward a culture that promotes “open and forthright information sharing” trans- “the development of practical tools to facilitate cooperative, collaborative, C \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 58 586 25-MAR-21 9:54 decision has had on parties’ use of expert evidence. of expert use on parties’ had has decision - too far: that Daubert has made it prohibi dulum has swung cases, including to pursue certain types of tively difficult successfully tort cases that cannot be brought environmental studies. and expensive expert without time-consuming ing of reliability standards). (describing the historical development 41% post-Daubert). excluded or limited the testimony of 25% of challenged experts pre- excluded or limited the testimony of 25% the Federal Judiciary 5–7 (2015), https://www.supremecourt.gov/publicinfo/ year-end/2015year-endreport.pdf [https://perma.cc/Y7LG-C4PJ] (Amended work Rule 1 “highlights the point that lawyers . . . have an affirmative duty to of dis- together, and with the court, to achieve prompt and efficient resolutions putes.”); 42705-crn_106-3 Sheet No. 32 Side B No. 32 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 33 Side A 03/25/2021 10:13:22 R R 587 587 When the risk is 305 note 36. LITIGATION SCIENCE SCIENCE LITIGATION 306 Additionally, as discussed above, a plaintiff seeking Additionally, as discussed above, An early determination on the admissibility of an An early determination on 304 Berger & Twerski, supra See supra text accompanying notes 270–272. See supra subpart III.B. See id. 303 Separating assessments of methodology from assessments Separating assessments of methodology Up-front clarity about the admissibility of expensive or about the admissibility of Up-front clarity M K 303 304 305 306 reduced that they will be unable to present expert testimony at reduced that they will be unable stage, plaintiffs may be trial or use it at the summary-judgment more easily and at more able to obtain litigation finance favorable terms. of application may change parties’ approach to objections to of application may change parties’ of a study are not yet known, the expert evidence: when results them may be more capable of methodology that will produce being judged on the merits. parent discovery” and arguing that “[c]ooperation does not conflict with the ad- parent discovery” and arguing that “[c]ooperation does not conflict with the vancement of [lawyers’] clients’ interests [but rather] enhances it”). \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 59 25-MAR-21 9:54 to engage in a costly litigation may have better access to litiga- to engage in a costly litigation of litigation risk has been re- tion finance if a major source proceeding. moved at an early stage in the expert’s proposed methodology lowers that risk and may un- expert’s proposed methodology group of aspiring plaintiffs with lock access to justice to a larger who might not otherwise be able potentially meritorious claims that the proposed method- to bring suit: if the court determines evidence, it does so before the ology cannot support admissible resources on procuring the evi- plaintiff has spent considerable dence. 2021] 2021] proceedings to early component is an adjudicative there when may plaintiffs testimony, of expert admissibility the regarding for the in exchange more transparency to provide be willing on admissibility. determination of an early prospect A their litiga- improve would immediately early ruling favorable help them ruling would early and an unfavorable tion position, strategy—including litigation decide their consider whether to case.abandoning the - stand to gain from early pro Defendants about an opposing by receiving information ceedings as well, early exclusion of stage and the potential expert at an early methodology. proposed expert barrier to litiga- methodologies may lower the time-consuming whether to bring plaintiff who is considering tion for a putative a lawsuit. to bring suit when the A plaintiff might hesitate very costly to ac- required to bring a claim is expert evidence being excluded is signifi- quire and the risk of the evidence cant. C Y 42705-crn_106-3 Sheet No. 33 Side A No. 33 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 33 Side B 03/25/2021 10:13:22 R , M K C Y A more any type of 308 note 58 (“[T]he note 58 [Vol. 106:529 106:529 [Vol. supra ONCLUSION challenge is contributing to the Vanishing On the Supposed Jury-Dependence of Evidence C The knowledge crisis that roiled the The knowledge crisis that roiled 309 CORNELL CORNELL LAW REVIEW . 165, 172 (2006) (“[T]he number of criminal and civil jury EV . L. R - routinely ruled on the admissi If courts instead ENN 307 The Disappearance of Civil Trial in the United States John H. Langbein, The Disappearance of Civil Trial in the L.J. 522, 522 (2012) (“Since the 1930s, the proportion of civil cases L.J. 522, 522 (2012) (“Since the 1930s, See Similar arguments might well be made with respect to other types of evi- See supra Part I. ALE The frameworks that apply to the admissibility of scientific The frameworks that apply to If early proceedings relating to expert evidence with an ad- with an evidence to expert relating proceedings If early 307 308 309 robust record of evidentiary decisions is beneficial to all players evidentiary decisions is beneficial robust record of benefit from in- and defendants both in the field: plaintiffs motions. of the outcome of evidentiary creased predictability linger in the pre-litigation stage Parties to disputes that still to a broader pool of decisions. would also benefit from access of the types of evidence that Having a more developed record admit in support of the contem- courts have been willing to to predict with more accu- plated claims would allow parties that will meet the court’s racy their ability to produce evidence their chances in litigation. requirements, and better assess 122 Y dence. on the admissibility of Indeed, perhaps early proceedings evidence in civil litigation rely on an implicit assumption: that evidence in civil litigation rely of the evidence will have the the court assessing the reliability that assessment at the time it is information needed to make called upon to do so. \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 60 588 25-MAR-21 over prevalent, more to become were component judicatory 9:54 that library of precedents a valuable would create time courts does not exist.currently and “Vanishing Trial” In light of the settle before litigations majority of civil that the vast the reality the cur- stage, the summary-judgment trial or even they reach the very issues toward evidentiary of adjudicating rent custom of a body of prece- limits the development end of proceedings dent—the before there of cases are terminated vast majority of the available on the admissibility of any ever is a ruling evidence. trials has declined substantially in recent years.”). scientific community has cast serious doubt on the validity of scientific community has cast this assumption.on a dimension of It has thrown the spotlight in admissibility proceed- reliability that is typically overlooked bility of expert evidence at the outset of discovery, the body of evidence at the outset of discovery, bility of expert pace. would grow at a more rapid relevant case law failure to ‘survive’ the Daubert Trial . . . .”); Frederick Schauer, Law, 155 U. P concluded at trial has declined from about 20% to below 2% in the federal courts concluded at trial has declined from about 20% to below 2% in the federal and below 1% in state courts.”); Nat’l Ctr. for State Courts, evidence would be a welcome development. But such arguments are beyond the scope of this paper. 42705-crn_106-3 Sheet No. 33 Side B No. 33 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 34 Side A 03/25/2021 10:13:22 R R R R 313 589 589 have note 216 314 supra If courts are to transition If courts are to 311 312 note 98, at 623; Jonathan P. Tennant et 310 note 144, at 11–13 (describing open data LITIGATION SCIENCE SCIENCE LITIGATION (June 13, 2017), https://f1000research.com/articles/ (Nov. 29, 2017), https://f1000research.com/articles/6- note 151, at 169 (reporting researchers’ views on open- uwell et al., supra ESEARCH ESEARCH Nosek, Spies & Motyl, supra See See supra subparts II.A & III.A. See supra subpart II.B. See supra subpart III.B. See, e.g., Cr ¨ The Analytical Flexibility that stands at the root of the at the root that stands Flexibility The Analytical the tools to start a conversation about This Article aims A broader conceptual discussion may be called for as well. A broader conceptual discussion Four Simple Recommendations to Encourage Best Practices in Research Four Simple Recommendations to Encourage A Multi-Disciplinary Perspective on Emergent and Future Innovations in Peer M K 314 310 311 312 313 (assessing empirically whether researchers are willing to provide their research (assessing empirically whether researchers are willing to provide their research data). al., Review, F1000R practices aimed at improving reliability and reproducibility); Rafael C. Jim´enez et and reproducibility); Rafael C. Jim´enez practices aimed at improving reliability al., Software, F1000R problem cannot be eradicated entirely, but as the recent sys- but as the entirely, cannot be eradicated problem has the social sciences inquiry in methodological tematic its impact to curb that is motivated a legal community proven, tools at its disposal. has a wealth of suggesting a re- in a civil-litigation context, that might work scientific evidence courts’ role in the creation of framing of the reforms for the procedural and methodological and a series of in the courts. treatment of science and post-publication or “continuous” peer review, and post-publication or “continuous” \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 61 25-MAR-21 9:54 to the role of a facilitator, in addition to their role as gatekeeper facilitator, in addition to their to the role of a optimal ways of work is necessary to explore or referee, further making this transition—ways transparency that promote both rights to funda- while preserving the parties’ and efficiency, mental fairness. since the Daubert decision, Almost three decades have passed of scientific practice has and during that time, the architecture changed dramatically. at the In addition to the developments mechanisms for the sharing and center of this Article, novel such as open-data platforms reviewing of ideas and data, changed the nature of the interactions between researchers, changed the nature of the interactions those who consume it.the science they produce, and This and cultural changes that are Article focused on procedural legal frameworks for the creation possible within the dominant in the civil context.and evaluation of scientific evidence It is greater progress could be made if worth exploring whether even 1151 [https://perma.cc/ST4H-QHWZ]. 6-876 [https://perma.cc/Q54G-THLJ] (recommending practices “designed around Open Source values” for the development of software used in research); Washburn et al., supra data practices); Vanpaemel, Vermorgen, Deriemaecker & Storms, 2021] 2021] as proceedings these in evaluated be fact cannot and in ings conducted. are currently they C Y 42705-crn_106-3 Sheet No. 34 Side A No. 34 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22 42705-crn_106-3 Sheet No. 34 Side B 03/25/2021 10:13:22 M K C Y [Vol. 106:529 106:529 [Vol. CORNELL CORNELL LAW REVIEW \\jciprod01\productn\C\CRN\106-3\CRN302.txt unknown Seq: 62 590 25-MAR-21 9:54 those frameworks were to be replaced by a new structure—one a new by be replaced were to frameworks those practice of scientific nature iterative and the open reflects that today. the relia- depend on of litigations that just outcome The engagement a continued demands scientific evidence bility of questions. with these 42705-crn_106-3 Sheet No. 34 Side B No. 34 Side Sheet 42705-crn_106-3 03/25/2021 10:13:22