A Bayesian Treatise on The Replication Crisis in Science by True Gibson A thesis submitted to Oregon State University Honors College in partial fulfillment of the requirements for the degree of Honors Baccalaureate of Science in Biochemistry and Biophysics Honors Scholar Presented May 25, 2018 Commencement June 2018 AN ABSTRACT OF THE THESIS OF True Gibson for the degree of Honors Baccalaureate of Science in Biochemistry and Biophysics presented on May 25, 2018. Title: A Bayesian Treatise on The Replication Crisis in Science. Abstract approved: __________________________________________________ Jonathan Kaplan Abstract Over the past decade, it has come to light that many published scientific findings cannot be reproduced. This has led to the replication crisis in science. Many researchers feel that they can no longer trust much of what they read in scientific journals, and the public is becoming ever more dubious of published research findings. I argue herein that the replication crisis is a result of the mistaken belief that the methods of classical statistical inference provide a coherent objective basis for evaluating scientific hypotheses. To solve the replication crisis, I suggest that the scientific community ought to invoke a Bayesian framework of statistical inference which integrates both successful and failed replication attempts to determine the confidence they should have in any given scientific hypothesis. Key Words: Inductive Reasoning, Replication Crisis, Reproducibility, Statistical Inference, Bayesian Statistics Corresponding e-mail address: [email protected] © Copyright by True Gibson May 25, 2018 All Rights Reserved A Bayesian Treatise on The Replication Crisis in Science by True Gibson A thesis submitted to Oregon State University Honors College in partial fulfillment of the requirements for the degree of Honors Baccalaureate of Science in Biochemistry and Biophysics Honors Scholar Presented May 25, 2018 Commencement June 2018 Honors Baccalaureate of Science in Biochemistry and Biophysics project of True Gibson presented on May 25, 2018. APPROVED: ______________________________________________________________ Jonathan Kaplan, Mentor, representing Department of Philosophy ______________________________________________________________ Sharyn Clough, Committee Member, representing Department of Philosophy ______________________________________________________________ Chong Fang, Committee Member, representing Department of Chemistry ______________________________________________________________ Toni Doolen, Dean, Oregon State University Honors College I understand that my project will become part of the permanent collection of Oregon State University, Honors College. My signature below authorizes release of my project to any reader upon request. __________________________________________________________________ True Gibson, Author I would like to thank Dr. Jonathan Kaplan for his valuable input into this project, as well as his patience and willingness to assist me through this tumultuous process. I would also like to thank Dr. Sharyn Clough and Dr. Chong Fang for their support and their contributions to the project. Finally, I must thank my family and friends for giving me the strength and encouragement to see this thesis through to the end. Table of Contents Section I. Introduction........................................................................1 i. The Replication Crisis.....................................................................1 ii. Responses from the Scientific Community.........................................8 iii. A Superior Response.....................................................................14 Section II. Faults in the Classical Approach......................................17 i. Choosing a Test Statistic.................................................................17 ii. Misinterpretations of Significance...................................................19 iii. Randomization as Justification.......................................................22 iv. Deciding Between Hypotheses.......................................................25 Section III. The Bayesian Framework...............................................30 i. Bayes’s Theorem...........................................................................30 ii. Understanding Probabilities............................................................33 iii. Convergence of Beliefs.................................................................36 iv. Bayes Factors..............................................................................39 v. Obstacles and Limitations...............................................................43 Section IV. Conclusion......................................................................47 i. The State of Science........................................................................47 ii. Looking Ahead..............................................................................48 Section V. References........................................................................51 Section I. Introduction i. The Replication Crisis Concerns about the reproducibility of scientific discoveries have deep roots, but in recent years these qualms have been magnified drastically - especially in the fields of psychology and medicine. The resulting methodological crisis has been coined the “replication crisis.” It has been famously estimated that “most claimed research findings are false.” (Ioannidis, 2005a). This troubling realization has caused a good deal of strife in the scientific community: scientists are becoming increasingly unsure of how to react to the publications they read, and the public has developed an attitude of distrust concerning the state of modern science. The foregoing situation has caused many people to wonder how we got to this point, and why we are becoming less certain about our scientific hypotheses despite having greater access to more powerful tools for data collection and analysis than ever before in history. I propose that the main causal factor of the replication crisis has been the scientific community’s adherence to the classical methodology of statistical inference. The methods laid out in the 20th century by eminent statisticians such as Ronald Fisher, Jerzy Neyman, and Egon Pearson have been incorporated ubiquitously into the scientific method, due to the belief that they provide an objective basis for making inferences about scientific hypotheses. This belief is, however, mistaken. Upon closer analysis, it will be shown that the classical methods of statistical inference contain inexorable elements of subjectivity and - 1 - moreover fail to yield a coherent objective basis of inference concerning scientific hypotheses. To solve this deep-seated issue in the methodology of science, I propose a Bayesian framework. This Bayesian framework allows one to readily incorporate new scientific findings to update one’s belief as to the verity of any scientific hypothesis. Widespread endorsement of such a Bayesian system would mitigate the lack of confidence caused by the replication crisis – particularly in fields such as psychology, medicine, and the social sciences where contradictory findings abound. The replication crisis is at its heart an interdisciplinary scientific phenomenon, as all sciences place reproducibility among their most important desiderata. The status of reproducibility as a sine qua non of scientific hypotheses traces back to the requisite condition of all scientific hypotheses: falsifiability. As was first popularized by Popper (1962), a scientific hypothesis must make a prediction about what ought to occur given some initial conditions. For instance, a believer in the oxygen theory of combustion would predict that a piece of wood will not burn in an atmosphere of pure nitrogen gas. Such a statement would serve as a suitable scientific hypothesis in the Popperian sense. Similarly, a more outlandish person might predict that the piece of wood will burn in a nitrogen atmosphere, if they happen to subscribe to the phlogiston theory of combustion. This too would be a valid hypothesis. The two scientists with incongruent world views could then conduct a series of tests, by repeatedly attempting to set a piece of wood, situated in a chamber filled with pure nitrogen, aflame. According to Popper, the logical asymmetry between verification and falsification dictates that - 2 - no amount of “confirmatory” experimental outcomes will be sufficient to verify any scientific hypothesis. On the other hand, a single experimental outcome which contradicts the prediction made by a scientific hypothesis is sufficient grounds to refute that hypothesis (Popper, 1959). It follows that a single failure to set the piece of wood aflame would effectively refute the phlogiston theorist’s hypothesis. Such an outcome, however, would not confirm the oxygen theorist’s hypothesis, since the possibility the wood catching fire in a subsequent trial of the experiment cannot (and can never) be ruled out. Popper’s treatment of the topic is insufficient, because it does not hold true for statistical hypotheses. By definition, statistical hypotheses do not make deterministic predictions about what will occur given some initial conditions; instead, statistical hypotheses assign probabilities to different outcomes which may occur given some set of initial conditions. The case of statistical hypotheses severely complicates the desideratum of falsifiability. How can we determine