Cuckoo Birds Case Study Cuckoo birds have a behavior in which they lay their eggs in other Analysis of Variance birds nests. The other birds then raise and care for the newly hatched cuckoos. Cuckoos return year after year to the same territory and lay their eggs Bret Hanlon and Bret Larget in the nests of a particular host species. Department of Statistics Furthermore, cuckoos appear to mate only within their territory. University of Wisconsin|Madison Therefore, geographical sub-species are developed, each with a dominant foster-parent species. November 22{November 29, 2011 A general question is, are the eggs of the different sub-species adapted to a particular foster-parent species? Specifically, we can ask, are the mean lengths of the cuckoo eggs the same in the different sub-species? ANOVA 1 / 59 ANOVA Case Studies 2 / 59 Cuckoo Bird Egg Length Distribution Comparing More than Two Populations We have developed both t and nonparametric methods for inference ● for comparing means from two populations. 25 ● ● What if there are three or more populations? ● ● ●●● ●● 24 ●● ● ● ● ● ● ● It is not valid to simply make all possible pairwise comparisons: ● ● ● ●● ●● ● ● ●●● ●●●● ● ●● ●●●●● with three populations, there are three such comparisons, with four 23 ● ●●● ● ● ● ● ● ● ● ●● ● there are six, and the number increases rapidly. ●●● ● ● ● ●●●●●●● ● ●●● ● ●●● 22 ●● ● ● ● ● The comparisons are not all independent: the data used to estimate ● ●● ● the differences between the pair of populations 1 and 2 and the pair ● ● ● ● ●●● Egg Length (mm) 21 ● ● ● ● of populations 1 and 3 use the same sample from population 1. ● ● When estimating differences with confidence, we may be concerned ● ● 20 ● ● about the confidence we ought to have that all differences are in their respective intervals. Robin TreePipet Wren HedgeSparrowMeadowPipet PiedWagtail For testing, there are many simultaneous tests to consider. Host Species What to do? ANOVA Case Studies 3 / 59 ANOVA The Big Picture 4 / 59 Hypotheses Illustrative Example The dot plots show two cases of three samples, each of size five. The sample means are respectively 180, 220, and 200 in both cases. The left plot appears to show differences in the mean; evidence for The common approach to this problem is based on a single null this in the right plot appears weaker. hypothesis ● H0 : µ1 = µ2 = ··· = µk 300 300 ● versus the alternative hypothesis that the means are not all the same ● (so that there are at least two means that differ) where there are k 250 250 ● ● ● groups. ● ● ● ● ● ● ● ●● ● If there is evidence against the null hypothesis, then further inference y 200 ● ● y 200 ● ● ● ● ● is carried out to examine specific comparisons of interest. ● ● ● 150 150 ● ● ● 100 100 ● A B C A B C ANOVA The Big Picture 5 / 59 ANOVA The Big Picture 6 / 59 Analysis of Variance ANOVA Table Concept To test the previous hypothesis, we construct a test statistic that is a ratio of two different and independent estimates of an assumed common variance among populations, σ2. The previous example suggests an approach that involves comparing The numerator estimate is based on sample means and variation variances; among groups. If variation among sample means is large relative to variation within The denominator estimate is based on variation within samples. samples, then there is evidence against H0 : µ1 = µ2 = ··· = µk . If the null hypothesis is true, then we expect this ratio to be close to If variation among sample means is small relative to variation within one (but with random sampling, it may be somewhat greater). samples, then the data is consistent with H0 : µ1 = µ2 = ··· = µk . If the null hypothesis is false, then the estimate in the numerator is The approach of testing H0 : µ1 = µ2 = ··· = µk on the basis of likely to be much larger than the estimate in the denominator, and comparing variation among and within samples is called Analysis of the test statistic may be much larger than one than can be explained Variance, or ANOVA. by chance variation alone. An ANOVA Table is simply an accounting method for calculating a complicated test statistic. The following several slides develop the notation underlying this theory. ANOVA The Big Picture 7 / 59 ANOVA ANOVA Table Variance 8 / 59 Notation Sample Mean and SD The sample mean for a group is the sum of all observations in the group divided by the number in the group. There are k populations. The notation i : j(i) = j means all i such that observation i is in group j. The ith observation is Yi which is in the j(i)th sample. The sample mean in group j is: We write j(i) to indicate the group associated with observation i. X We let i vary from 1 to n, the total number of observations. Yi i:j(i)=j j varies from 1 to k, the total number of populations/samples. Y¯ = j n There are a total of n observations with nj observations in sample j. j and the sample standard deviation in group j is n = n1 + ··· + nk v u X ¯ 2 u (Yi − Yj ) u t i:j(i)=j sj = nj − 1 ANOVA ANOVA Table Variance 9 / 59 ANOVA ANOVA Table Variance 10 / 59 Grand Mean Modeling Assumptions We make the following modeling assumptions: The grand mean Y¯ is the mean of all observations. All observations Yi are independent. Note that the grand mean E(Yi ) = µj(i), where µj(i) is the mean of population j from which observation i was drawn. k 2 2 X nj Var(Y ) = σ , where σ is the variance of population j. Y¯ = Y¯ i j(i) j(i) n j j=1 We will also often make the following two additional assumptions: is the weighted average of the sample means, weighted by sample size. 2 2 all population variances are equal: σj = σ for all j; all observations are normally distributed: Yi ∼ N(µj(i); σj(i)) ANOVA ANOVA Table Variance 11 / 59 ANOVA ANOVA Table Variance 12 / 59 Distributions of the Sample Means Variation Among Samples We use this formula for the variation among sample means: k X 2 nj (Y¯j − Y¯ ) j=1 With the first set of assumptions, note that 2 which is a weighted sum of squared deviations of sample means from σj E(Y¯j ) = µj and Var(Y¯j ) = nj the grand mean, weighted by sample size. and additionally, if the second set of assumptions are made, then Under the assumptions of independence and equal variances, σ Y¯j ∼ N µj ; p nj k k X 2 2 X 2 E nj (Y¯j − Y¯ ) = (k − 1)σ + nj (µj − µ) j=1 j=1 where Pk nj µj µ = j=1 n is the expected value of the grand mean Y¯ . ANOVA ANOVA Table Variance 13 / 59 ANOVA ANOVA Table Variance 14 / 59 Variation Among Samples (cont.) Variation Within Samples The sum For each sample, the sample variance k X ¯ ¯ 2 P (Y − Y¯ )2 nj (Yj − Y ) 2 i:j(i)=j i j j=1 sj = nj − 1 is called the group sum of squares. is an estimate of that population's variance, σ2. If the null hypothesis H0 : µ1 = µ2 = ··· = µk is true, then j Pk 2 Under the assumptions of equal variance and independence, each s2 is j=1 nj (µj − µ) = 0 and j then an independent estimate of σ2. k X ¯ ¯ 2 2 The formula E nj (Yj − Y ) = (k − 1)σ k j=1 X 2 (nj − 1)sj This suggests defining j=1 Pk 2 nj (Y¯j − Y¯ ) is the sum of all squared deviations from individual sample means and MS = j=1 groups k − 1 has expected value to be the group mean square. k 2 X 2 2 If the null hypothesis is true, then E(MSgroups) = σ ; otherwise, E (nj − 1)sj = (n − k)σ 2 Pk 2 2 j=1 E(MSgroups) = σ + j=1 nj (µj − µ) =(k − 1) > σ . ANOVA ANOVA Table Variance 15 / 59 ANOVA ANOVA Table Variance 16 / 59 Variation Within Samples (cont.) The F Test Statistic We have developed two separate formulas for variation among and within samples, each based on a different mean square: The mean square error formula I MSgroups measures variation among groups; I MSerror measures variation within groups. Pk 2 j=1(nj − 1)sj Define the ratio F = MS =MS to be the F -statistic (named in MS = groups error error n − k honor of R. A. Fisher who developed ANOVA among many other accomplishments). is a weighted average of the sample variances, weighted by degrees of freedom. When H0 : µ1 = µ2 = ··· = µk is true (and the assumption of equal 2 2 variances is also true), then both E(MSgroups) = σ and Notice that E(MSerror) = σ always: it is true when 2 E(MSerror) = σ and the value of F should then be close to 1. H0 : µ1 = µ2 = ··· = µk is true, but also when H0 is false. However, if the population mean are not all equal, then 2 E(MSgroups) > σ and we expect F to be greater than one, perhaps by quite a bit. ANOVA ANOVA Table Variance 17 / 59 ANOVA ANOVA Table Test Statistic 18 / 59 The F Distribution Sampling Distribution Definition If W and W are independent χ2 random variables with d and d 1 2 1 2 If we have k independent random samples and: degrees of freedom, then I the null hypothesis H0 : µ1 = µ2 = ··· = µk is true; W1=d1 2 2 F = I all population variances are equal σi = σ ; W2=d2 I individual observations are normal, Yi ∼ N(µ, σ); has an F distribution with d1 and d2 degrees of freedom.