How Surprised is the Skeptic? Simulating the Skeptic’s World STAT 113 Hypothesis Testing II The World According to the Null Hypothesis Colin Reimer Dawson October 19, 2020 1 / 17 How Surprised is the Skeptic? Simulating the Skeptic’s World How Surprised is the Skeptic? Simulating the Skeptic’s World Randomization Distribution 2 / 17 How Surprised is the Skeptic? Simulating the Skeptic’s World The Lady Tasting Tea At a 1920s party in Cambridge, UK, a lady (Dr. Muriel Bristol, a phycologist, specializing in algae) claimed she could tell whether a cup of tea had been prepared by adding milk before or after the tea was poured. A statistician, Ronald Fisher, also in attendance, proposed a blind taste test w/ 10 cups of tea, each prepared in random order. • How much success is enough to believe her? 3 / 17 How Surprised is the Skeptic? Simulating the Skeptic’s World The Null Hypothesis • R.A. Fisher: Formulate the “most boring” hypothesis about the world/process/population (“nothing to see here; moving along”) • Try to measure how surprising the data would have been if the “boring” thing were true. • Fisher called this boring “antihypothesis” the null hypothesis (abbreviated as H0) 4 / 17 How Surprised is the Skeptic? Simulating the Skeptic’s World The Alternative Hypothesis • Jerzy Neyman and Egon Pearson added the idea of a specific alternative hypothesis to this formulation • The “alternative” is usually the one that you started with H0: the new drug works no better than the old one H1: the new drug works better than the old one H0: there is no relationship between bill and tip percent H1: there is some relationship between bill and tip percent 5 / 17 How Surprised is the Skeptic? Simulating the Skeptic’s World Hypotheses Are About Parameters • We know what’s true about our dataset (the sample) • Our hypotheses propose possibilities involving the wider context (population/process) • Important: When formulating statistical hypotheses, they will always be about the population/process/phenomenon • Correct H1: A majority of all U.S. registered voters plan to vote for Biden in November • Incorrect H1: A majority of the registered voters in the poll plan to vote for Biden in November 6 / 17 • Data: She tastes 10 cups in a blind taste test. • Data: 20 mice are randomly split into two groups. One group is fed in the light, another in the dark. Their food intake in grams is measured. • Data: We measure pH and mercury levels in 50 random lakes in Florida. 2. Claim: Dr. Bristol can tell the difference between milk-first and tea-first preparations better than a coin flip 3. Claim: Lab mice eat more on average if the room is light at meal time than if it is dark at meal time. How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1) 1. Claim: There is a positive linear association between pH and mercury in Florida lakes. 7 / 17 • Data: She tastes 10 cups in a blind taste test. • Data: 20 mice are randomly split into two groups. One group is fed in the light, another in the dark. Their food intake in grams is measured. 2. Claim: Dr. Bristol can tell the difference between milk-first and tea-first preparations better than a coin flip 3. Claim: Lab mice eat more on average if the room is light at meal time than if it is dark at meal time. How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1) 1. Claim: There is a positive linear association between pH and mercury in Florida lakes. • Data: We measure pH and mercury levels in 50 random lakes in Florida. 7 / 17 • Data: 20 mice are randomly split into two groups. One group is fed in the light, another in the dark. Their food intake in grams is measured. • Data: She tastes 10 cups in a blind taste test. 3. Claim: Lab mice eat more on average if the room is light at meal time than if it is dark at meal time. How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1) 1. Claim: There is a positive linear association between pH and mercury in Florida lakes. • Data: We measure pH and mercury levels in 50 random lakes in Florida. 2. Claim: Dr. Bristol can tell the difference between milk-first and tea-first preparations better than a coin flip 7 / 17 • Data: 20 mice are randomly split into two groups. One group is fed in the light, another in the dark. Their food intake in grams is measured. 3. Claim: Lab mice eat more on average if the room is light at meal time than if it is dark at meal time. How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1) 1. Claim: There is a positive linear association between pH and mercury in Florida lakes. • Data: We measure pH and mercury levels in 50 random lakes in Florida. 2. Claim: Dr. Bristol can tell the difference between milk-first and tea-first preparations better than a coin flip • Data: She tastes 10 cups in a blind taste test. 7 / 17 • Data: 20 mice are randomly split into two groups. One group is fed in the light, another in the dark. Their food intake in grams is measured. How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1) 1. Claim: There is a positive linear association between pH and mercury in Florida lakes. • Data: We measure pH and mercury levels in 50 random lakes in Florida. 2. Claim: Dr. Bristol can tell the difference between milk-first and tea-first preparations better than a coin flip • Data: She tastes 10 cups in a blind taste test. 3. Claim: Lab mice eat more on average if the room is light at meal time than if it is dark at meal time. 7 / 17 How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1) 1. Claim: There is a positive linear association between pH and mercury in Florida lakes. • Data: We measure pH and mercury levels in 50 random lakes in Florida. 2. Claim: Dr. Bristol can tell the difference between milk-first and tea-first preparations better than a coin flip • Data: She tastes 10 cups in a blind taste test. 3. Claim: Lab mice eat more on average if the room is light at meal time than if it is dark at meal time. • Data: 20 mice are randomly split into two groups. One group is fed in the light, another in the dark. Their food intake in grams is measured. 7 / 17 How Surprised is the Skeptic? Simulating the Skeptic’s World Outline How Surprised is the Skeptic? Simulating the Skeptic’s World Randomization Distribution 8 / 17 • a skeptic who thinks there is nothing interesting going on (they believe H0) • a proponent who thinks there is something interesting there (they believe H1) • Ask which values of the statistic would surprise the proponent less than they would surprise the skeptic • Of those, sort them in descending order according to how much they favor the proponent • If the data yields a statistic which is sufficiently far up that list, the skeptic will change their mind How Surprised is the Skeptic? Simulating the Skeptic’s World Logic of Testing H0 • Imagine two observers: 9 / 17 • a proponent who thinks there is something interesting there (they believe H1) • Ask which values of the statistic would surprise the proponent less than they would surprise the skeptic • Of those, sort them in descending order according to how much they favor the proponent • If the data yields a statistic which is sufficiently far up that list, the skeptic will change their mind How Surprised is the Skeptic? Simulating the Skeptic’s World Logic of Testing H0 • Imagine two observers: • a skeptic who thinks there is nothing interesting going on (they believe H0) 9 / 17 • Ask which values of the statistic would surprise the proponent less than they would surprise the skeptic • Of those, sort them in descending order according to how much they favor the proponent • If the data yields a statistic which is sufficiently far up that list, the skeptic will change their mind How Surprised is the Skeptic? Simulating the Skeptic’s World Logic of Testing H0 • Imagine two observers: • a