Random Effects Lukas Meier, Seminar für Statistik New Philosophy… . Up to now: treatment effects were fixed, unknown parameters that we were trying to estimate. Such models are also called fixed effects models. Now: Consider the situation where treatments are random samples from a large population of potential treatments. Example: Effect of machine operators that were randomly selected from a large pool of operators. In this setup, treatment effects are random variables and therefore called random effects. The corresponding model will be a random effects model. 1 New Philosophy… . Why would we be interested in a random effects situation? . It is a useful way of thinking if we want to make a statement (conclusion) about the population of all treatments. In the operator example we shift the focus away from the individual operators (treatments) to the population of all operators (treatments). Typically, we are interested in the variance of the treatment population. E.g., what is the variation from operator to operator? 2 Examples of Random Effects Randomly select… …from… clinics …all clinics in a country. school classes …all school classes in a region. investigators …a large pool of investigators. series in quality control …all series in a certain time period. … … 3 Carton Experiment One (Oehlert, 2000) . Company with 50 machines that produce cardboard cartons. Ideally, strength of the cartons shouldn’t vary too much. Therefore, we want to have an idea about . “machine-to-machine” variation . “sample-to-sample” variation on the same machine. Perform experiment: . Choose 10 machines at random (out of the 50) . Produce 40 cartons on each machine . Test resulting cartons for strength ( response) 4 Carton Experiment One (Oehlert, 2000) . Model so far: 푌푖푗 = 휇 + 훼푖 + 휖푖푗, where 훼푖 is the (fixed) effect of machine 푖 and 휀푖푗 are the errors with the usual assumptions. However, this model does not reflect the sampling mechanism from above. If we repeat the experiment, the selected machines change and therefore also the meaning of the parameters: they typically correspond to a different machine! . Moreover, we want to learn something about the population of all machines. 5 Carton Experiment One (Oehlert, 2000) . New: Random effects model: Parameter 푌푖푗 = 휇 + 훼푖 + 휖푖푗, Random variable with effect of 2 machine . 훼푖 i. i. d. ∼ 푁 0, 휎훼 2 . 휖푖푗 i. i. d. ∼ 푁 0, 휎 . This looks very similar to the old model, however the 훼푖’s are now random variables! . That small change will have a large impact on the properties of the model and on our way to analyze such kind of data. 6 Carton Experiment One (Oehlert, 2000) . Properties of random effects model: 2 2 . Var 푌푖푗 = 휎훼 + 휎 variance components 0 푖 ≠ 푘 different machines . Cor 푌 , 푌 = 휎2/(휎2 + 휎2) 푖 = 푘, 푗 ≠ 푙 푖푗 푘푙 훼 훼 same machine 1 푖 = 푘, 푗 = 푙 intraclass correlation Reason: Observations from the same machine “share” the same random value 훼푖 and are therefore correlated. Conceptually, we could also put all the correlation structure into the error term and forget about the 훼푖’s, i.e. 푌푖푗 = 휇 + 휖푖푗 where 휖푖푗 has the appropriate correlation structure from above. Sometimes this interpretation is a useful way of thinking. 7 Random vs. Fixed: Overview . Comparison between random and fixed effects models Term Fixed effects model Random effects model fixed, unknown 훼 훼 i. i. d. ∼ 푁(0, 휎2) 푖 constant 푖 훼 Side constraint on 훼푖 needed not needed 퐸[푌푖푗] 휇 + 훼푖 휇, but 퐸 푌푖푗 훼푖 = 휇 + 훼푖 2 2 2 Var(푌푖푗) 휎 휎훼 + 휎 0 푖 ≠ 푘 2 2 2 Corr(푌푖푗, 푌푘푙) = 0 (푗 ≠ 푙) = 휎훼 /(휎훼 + 휎 ) 푖 = 푘, 푗 ≠ 푙 1 푖 = 푘, 푗 = 푙 . A note on the sampling mechanism: . Fixed: Draw new random errors only, everything else is kept constant. Random: Draw new “treatment effects” and new random errors (!) 8 Think of 3 specific Illustration of Correlation Structure machines Fixed case: 3 different fixed treatment levels 훼푖. We (repeatedly) sample 2 observations per treatment level: Think of 2 carton 푌푖푗 = 휇 + 훼푖 + 휖푖푗 samples 훼3 = 3.5 훼2 = 1 훼1 = −4.5 9 Illustration of Correlation Structure Think of 2 carton Random case: samples Whenever we draw 2 observations 푌푖1 and 푌푖2 we first have to draw a new (common) random treatment effect 훼푖. Think of a 8 random machine. 6 4 2 2 i Y 0 -2 -4 -6 -10 -5 0 5 10 10 Yi1 Carton Experiment Two (Oehlert, 2000) . Let us extend the previous experiment. Assume that machine operators also influence the production process. Choose 10 operators at random. Each operator will produce 4 cartons on each machine (hence, operator and machine are crossed factors). All assignments are completely randomized. 11 Carton Experiment Two (Oehlert, 2000) 2 푁 0, 휎2 푁 0, 휎2 2 . Model: 푁 0, 휎훼 훽 훼훽 푁 0, 휎 푌푖푗푘 = 휇 + 훼푖 + 훽푗 + 훼훽 푖푗 + 휖푖푗푘, main effect main effect interaction with machine operator machine×operator . 훼푖, 훽푗, 훼훽 푖푗, 휖푖푗푘 independent and normally distributed. 2 2 2 2 . Var 푌푖푗푘 = 휎훼 + 휎훽 + 휎훼훽 + 휎 (different variance components). Measurements from the same machine and / or operator are again correlated. The more random effects two observations share, the larger the correlation. It is given by sum of shared variance components sum of all variance components . E.g., correlation between two (different) observations from the same operator on different machines is given by 2 휎훽 2 2 2 2 휎훼 + 휎 + 휎 + 휎 훽 훼훽 12 Carton Experiment Two (Oehlert, 2000) . Hierarchy is typically less problematic in random effects models. 1) What part of the variation is due to general machine-to-machine 2 variation? 휎훼 2) What part of the variation is due to operator-specific machine 2 variation? 휎훼훽 Could ask question (1) even if interaction is present (question (2)). Extensions to more than two factors straightforward. 13 ANOVA for Random Effects Models (balanced designs) . Sums of squares, degrees of freedom and mean squares are being calculated as if the model would be a fixed effects model (!) . One-way ANOVA (퐴 random, 푛 observations per cell) Source df SS MS E[MS] 2 2 퐴 푔 − 1 … … 휎 + 푛휎훼 Error 푁 − 푔 … … 휎2 . Two-way ANOVA (퐴, 퐵, 퐴퐵 random, 푛 observations per cell) Source df SS MS E[MS] 2 2 2 퐴 푎 − 1 … … 휎 + 푏 ⋅ 푛 ⋅ 휎훼 + 푛 ⋅ 휎훼훽 2 2 2 퐵 푏 − 1 … … 휎 + 푎 ⋅ 푛 ⋅ 휎훽 + 푛 ⋅ 휎훼훽 2 2 퐴퐵 (푎 − 1)(푏 − 1) … … 휎 + 푛 ⋅ 휎훼훽 Error 푎푏(푛 − 1) … … 휎2 14 One-Way ANOVA with Random Effects . We are now formulating our null-hypothesis with respect 2 to the parameter 휎훼 . 2 2 . To test 퐻0: 휎훼 = 0 vs. 퐻퐴: 휎훼 > 0 we use the ratio 푀푆퐴 퐹 = ∼ 퐹푔−1,푁−푔 under 퐻0 푀푆퐸 Exactly as in the fixed effect case! . Why? Under the old and the new 퐻0 both models are the same! 15 Two-Way ANOVA with Random Effects 2 . To test 퐻0: 휎훼 = 0 we need to find a term which has identical 퐸[푀푆] under 퐻0. 푀푆퐴 . Use 푀푆퐴퐵, i.e. 퐹 = ∼ 퐹푎−1, 푎−1 푏−1 under 퐻0. 푀푆퐴퐵 2 . Similarly for the test 퐻0: 휎훽 = 0. The interaction will be tested against the error, i.e. use 푀푆퐴퐵 퐹 = ∼ 퐹 푎−1 푏−1 , 푎푏 푛−1 푀푆퐸 2 under 퐻0: 휎훼훽 = 0. In the fixed effect case we would test all effects against the error term (i.e., use 푀푆퐸 instead of 푀푆퐴퐵 to build 퐹-ratio)! 16 Didn’t look at this Two-Way ANOVA with Random Effects column when analyzing factorials . Reason: ANOVA table for fixed effects: Shorthand notation for a Source df E[MS] term depending on 훼′푠 퐴 푎 − 1 휎2 + 푏 ⋅ 푛 ⋅ 푄 훼 푖 퐵 푏 − 1 휎2 + 푎 ⋅ 푛 ⋅ 푄(훽) 퐴퐵 (푎 − 1)(푏 − 1) 휎2 + 푛 ⋅ 푄(훼훽) Error 푎푏(푛 − 1) 휎2 . E.g, 푆푆퐴 (푀푆퐴) is being calculated based on column-wise means. In the fixed effects model, the expected mean squares do not “contain” any other component. 17 Two-Way ANOVA with Random Effects . In a random effects model, a column-wise mean is “contaminated” with the average of the corresponding interaction terms. In a fixed effects model, the sum (or mean) of these interaction terms is zero by definition. In the random effects model, this is only true for the expected value, but not for an individual realization! . Hence, we need to check whether the variation from “column to column” is larger than term based on error and interaction term. 18 Point Estimates of Variance Components . We do not only want to test the variance components, we also want to have estimates of them. 2 2 2 2 . I.e., we want to determine 휎 훼 , 휎 훽 , 휎 훼훽, 휎 etc. Easiest approach: ANOVA estimates of variance components. Use columns “MS” and “E[MS]” in ANOVA table, solve the corresponding equations from bottom to top. Example: One-way ANOVA 2 . 휎 = 푀푆퐸 2 푀푆퐴−푀푆퐸 . 휎 = 훼 푛 19 Point Estimates of Variance Components . Advantage: Can be done using standard ANOVA functions (i.e., no special software needed). Disadvantages: . Estimates can be negative (in previous example if 푀푆퐴 < 푀푆퐸). Set them to zero in such cases. Not always as easy as here. This is like a method of moments estimator. More modern and much more flexible: restricted maximum-likelihood estimator (REML). 20 Point Estimates of Variance Components: REML . Think of a modification of maximum likelihood estimation that removes bias in estimation of variance components. Theory complicated (still ongoing research). Software implementation in R-package lme4 (or lmerTest) .