TWO PRINCIPLES OF EVIDENCE AND THEIR IMPLICATIONS FOR THE PHILOSOPHY OF SCIENTIFIC METHOD by Gregory Stephen Gandenberger BA, Philosophy, Washington University in St. Louis, 2009 MA, Statistics, University of Pittsburgh, 2014 Submitted to the Graduate Faculty of the Kenneth P. Dietrich School of Arts and Sciences in partial fulfillment of the requirements for the degree of Doctor of Philosophy University of Pittsburgh 2015 UNIVERSITY OF PITTSBURGH KENNETH P. DIETRICH SCHOOL OF ARTS AND SCIENCES This dissertation was presented by Gregory Stephen Gandenberger It was defended on April 14, 2015 and approved by Edouard Machery, Pittsburgh, Dietrich School of Arts and Sciences Satish Iyengar, Pittsburgh, Dietrich School of Arts and Sciences John Norton, Pittsburgh, Dietrich School of Arts and Sciences Teddy Seidenfeld, Carnegie Mellon University, Dietrich College of Humanities & Social Sciences James Woodward, Pittsburgh, Dietrich School of Arts and Sciences Dissertation Director: Edouard Machery, Pittsburgh, Dietrich School of Arts and Sciences ii Copyright © by Gregory Stephen Gandenberger 2015 iii TWO PRINCIPLES OF EVIDENCE AND THEIR IMPLICATIONS FOR THE PHILOSOPHY OF SCIENTIFIC METHOD Gregory Stephen Gandenberger, PhD University of Pittsburgh, 2015 The notion of evidence is of great importance, but there are substantial disagreements about how it should be understood. One major locus of disagreement is the Likelihood Principle, which says roughly that an observation supports a hypothesis to the extent that the hy- pothesis predicts it. The Likelihood Principle is supported by axiomatic arguments, but the frequentist methods that are most commonly used in science violate it. This dissertation advances debates about the Likelihood Principle, its near-corollary the Law of Likelihood, and related questions about statistical practice. Chapter2 provides a new axiomatic proof of the Likelihood Principle that avoids influential responses to previous proofs. Chapter3 exhibits the close connection between the Likelihood Principle and the Law of Likelihood and responds to three purported counterexamples to them. Chapter4 presents a new counterexample that is more difficult to avoid but argues that it does not speak against those principles in typical applications. The next two chapters turn to implications. It is motivated by tension among three desiderata for a method of evaluating hypotheses in light of data. We would like such a method to (1) respect the evidential meaning of data, (2) provide direct guidance for belief or action, and (3) avoid inputs that are not grounded in evidence. Unfortunately, frequentist methods violate (1) by violating the Likelihood Principle, likelihoodist methods violate (2) by directly addressing only questions about the evidential meaning of data, and Bayesian methods violate (3) by using prior probabilities. Chapter5 sharpens this tension by arguing that no method that satisfies likelihoodist strictures can provide a genuine rival to iv frequentist and Bayesian methodologies. Chapter6 argues that many frequentist violations of the Likelihood Principle are not required by basic frequentist commitments. Those that are may be permissible for the sake of enabling progress in the presence of highly indefinite prior beliefs, but they are not mandatory or even strongly motivated. These considerations support more widespread use of Bayesian methods despite difficulties in specifying prior probability distributions. v TABLE OF CONTENTS PREFACE ......................................... xiv 1.0 INTRODUCTION .................................1 1.1 Evidence and the Norms of Scientific Research................1 1.2 Statistical Inference and Scientific Inference.................5 1.3 The Likelihood Principle and Its Bearing on Methodology......... 10 1.4 The Contributions Made by This Dissertation................ 11 2.0 A NEW PROOF OF THE LIKELIHOOD PRINCIPLE ......... 13 2.1 Introduction................................... 13 2.2 The New Proof................................. 17 2.3 How the New Proof Addresses Proposals to Restrict Birnbaum's Premises. 30 2.4 A Response to Arguments that the Proofs are Fallacious.......... 36 2.5 Conclusion.................................... 39 3.0 NEW RESPONSES TO THREE COUNTEREXAMPLES TO THE LIKELIHOOD PRINCIPLE ........................... 41 3.1 Introduction................................... 41 3.2 Why a counterexample to the Law of Likelihood is a prima facie counterex- ample to the Likelihood Principle....................... 42 3.3 Response to Fitelson's counterexample.................... 44 3.3.1 Objection 1: Response conflicts with constraints on evidential favoring 46 3.3.2 Objection 2: Response fails to address tacking paradox........ 48 3.3.2.1 Response 1: Bite the bullet................... 49 3.3.2.2 Response 2: Regard Law as explicating \r-favoring"..... 50 vi 3.3.2.3 Response 3: Restrict Law of Likelihood to structurally iden- tical alternatives......................... 51 3.3.3 Objection 3: Response excludes cases of scientific interest...... 52 3.3.3.1 Cases involving competing causal claims............ 52 3.3.3.2 Cases involving nested models................. 54 3.4 Response to Armitage's counterexample................... 57 3.4.1 Two problems for attempts to use the Weak Repeated Sampling Prin- ciple as an objection to the Law of Likelihood............. 58 3.4.1.1 Problem 1: The Weak Repeated Sampling Principle is unrea- sonably strong.......................... 58 3.4.1.2 Problem 2: The Law of Likelihood itself cannot conflict with the Weak Repeated Sampling Principle............ 60 3.4.2 Why Armitage's example is no threat to the Law of Likelihood... 61 3.5 Response to Stein's counterexample...................... 63 3.6 Conclusion.................................... 68 4.0 A COUNTEREXAMPLE TO THE LAW OF LIKELIHOOD FOR PROBABILITY-ZERO HYPOTHESES ................... 69 4.1 Introduction................................... 69 4.2 A Counterexample to the Law of Likelihood for Probability-Zero Hypotheses 70 4.3 Responses to Worries About the Counterexample.............. 71 4.3.1 Worry 1: Each hypothesis omits two points on the relevant great circle 73 4.3.2 Worry 2: The likelihood ratio is small................. 73 4.3.3 Worry 3: Real measurement techniques have finite precision..... 74 4.3.4 Worry 4: The hypotheses belong to different models......... 74 4.3.5 Worry 5: No limiting operation is specified.............. 78 4.4 A No-Go Result for Addressing the Counterexample Through an Alternative Theory of Conditional Probability....................... 79 4.5 Addressing the Counterexample Within Existing Theories of Conditional Probability................................... 84 4.5.1 Restricting the Law of Likelihood.................... 85 vii 4.5.2 Regarding Evidential Favoring as Either Relative or Indeterminate. 89 4.6 Conclusion.................................... 92 5.0 WHY I AM NOT A METHODOLOGICAL LIKELIHOODIST ..... 93 5.1 Introduction................................... 93 5.2 Methodological Likelihoodism......................... 94 5.3 An Overview of My Argument Against Methodological Likelihoodism... 96 5.4 Claim 1: An adequate methodology provides a good norm of commitment. 97 5.5 Claim 2: A good purely likelihood-based norm of commitment would have a particular form................................ 99 5.5.1 Objection: A good norm of commitment could allow withholding judg- ment.................................... 101 5.5.2 Reply: Allowing withholding judgment does not help......... 102 5.6 Claim 3: A good norm of commitment would allow one to prefer chance hypotheses to maximally likely alternatives.................. 103 5.7 Claim 4: A good norm of commitment would treat alike pairs consisting of a hypothesis and its negation that are logically equivalent given one's evidence106 5.7.1 Objection 1: H1 and H2 Are Not Mutually Exclusive......... 107 5.7.2 Reply to Objection 1........................... 108 5.7.3 Objection 2: H1 and H2 are not statistical hypotheses........ 108 5.7.4 Reply to Objection 2........................... 108 5.8 Claim 5: A good norm of commitment would allow one both to obey a highly plausible disjunction rule and to treat pairs of logically equivalent hypotheses alike................................. 109 5.8.1 Objection: This argument relies on a strict notion of neutrality that is not clearly appropriate........................ 112 5.8.2 Reply: Making the notion of neutrality less strict does not result in a successful theory............................. 112 5.9 Conclusion.................................... 113 6.0 WHY FREQUENTIST VIOLATIONS OF THE LIKELIHOOD PRIN- CIPLE ARE PERMISSIBLE AT BEST ................... 115 viii 6.1 Introduction................................... 115 6.2 How Standard Frequentist Practice Conforms to Likelihoodist Principles Within Component Experiments........................ 117 6.3 How Standard Frequentist Practice Violates Likelihoodist Principles Across Component Experiments........................... 120 6.3.1 Frequentists generally use a common Type I error rate across experi- ments................................... 120 6.3.2 Frequentists Generally Adjust Their Standards on Individual Tests to Control \Familywise" Error Rates................... 124 6.3.3 Frequentists Generally Require That Hypotheses Be Predesignated. 126 6.4 Why Violating the Likelihoodist Principles Is Incompatible With Maximiz- ing Expected Utility.............................. 127 6.5 How Thinking of Frequentist Considerations as Tie-Breakers