Statistics 203: Introduction to Regression and Analysis of Variance Fixed vs. Random Effects

Jonathan Taylor

- p. 1/19 Today’s class

● Today’s class ■ Random effects. ● Two-way ANOVA ● Random vs. fixed effects ■ ● When to use random effects? One-way random effects ANOVA. ● Example: sodium content in ■ beer Two-way mixed & random effects ANOVA. ● One-way random effects model ■ ● Implications for model Sattherwaite’s procedure. ● One-way random ANOVA table ● Inference for µ· ● 2 Estimating σµ ● Example: productivity study ● Two-way random effects model ● ANOVA tables: Two-way (random) ● Mixed effects model ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 2/19 Two-way ANOVA

● Today’s class ■ Second generalization: more than one grouping variable. ● Two-way ANOVA ● Random vs. fixed effects ■ ● When to use random effects? Two-way ANOVA model: observations: ● Example: sodium content in ≤ ≤ ≤ ≤ ≤ ≤ beer (Yijk), 1 i r, 1 j m, 1 k nij : r groups in first ● One-way random effects model grouping variable, m groups ins second and nij samples in ● Implications for model ● One-way random ANOVA (i, j)-“cell”: table ● Inference for µ· 2 ● 2 Estimating σµ Yijk = µ + αi + βj + (αβ)ij + εijk, εijk ∼ N(0, σ ). ● Example: productivity study ● Two-way random effects ■ model Constraints: ● ANOVA tables: Two-way ◆ r (random) αi = 0 ● i=1 Mixed effects model ◆ Pm ● Two-way mixed effects model j=1 βj = 0 ● ANOVA tables: Two-way Pm (mixed) ◆ (αβ) = 0, 1 ≤ i ≤ r ● Confidence intervals for j=1 ij variances ◆ Pr ● Sattherwaite’s procedure (αβ)ij = 0, 1 ≤ j ≤ m. Pi=1

- p. 3/19 Random vs. fixed effects

● Today’s class ■ In ANOVA examples we have seen so far, the categorical ● Two-way ANOVA ● Random vs. fixed effects variables are well-defined categories: below average fitness, ● When to use random effects? ● Example: sodium content in long duration, etc. beer ● One-way random effects ■ model In some designs, the categorical variable is “subject”. ● Implications for model ● One-way random ANOVA ■ Simplest example: repeated measures, where more than table ● Inference for µ· one (identical) measurement is taken on the same individual. ● Estimating σ2 µ ■ ● Example: productivity study In this case, the “group” effect αi is best thought of as ● Two-way random effects model random because we only sample a subset of the entire ● ANOVA tables: Two-way (random) population of subjects. ● Mixed effects model ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 4/19 When to use random effects?

● Today’s class ■ A “group” effect is random if we can think of the levels we ● Two-way ANOVA ● Random vs. fixed effects observe in that group to be samples from a larger population. ● When to use random effects? ● Example: sodium content in ■ beer Example: if collecting data from different medical centers, ● One-way random effects model “center” might be thought of as random. ● Implications for model ● One-way random ANOVA ■ Example: if surveying students on different campuses, table ● Inference for µ· “campus” may be a random effect. ● 2 Estimating σµ ● Example: productivity study ● Two-way random effects model ● ANOVA tables: Two-way (random) ● Mixed effects model ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 5/19 Example: sodium content in beer

● Today’s class ■ How much sodium is there in North American beer? How ● Two-way ANOVA ● Random vs. fixed effects much does this vary by brand? ● When to use random effects? ● Example: sodium content in ■ beer Observations: for 6 brands of beer, researchers recorded the ● One-way random effects model sodium content of 8 12 ounce bottles. ● Implications for model ● One-way random ANOVA ■ Questions of interest: what is the “grand mean” sodium table ● Inference for µ· content? How much variability is there from brand to brand? ● Estimating σ2 µ ■ ● Example: productivity study “Individuals” in this case are brands, repeated measures are ● Two-way random effects model the 8 bottles. ● ANOVA tables: Two-way (random) ● Mixed effects model ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 6/19 One-way random effects model

● Today’s class ■ Suppose we take n identical measurements from r subjects. ● Two-way ANOVA ● Random vs. fixed effects ■ ● When to use random effects? Yij ∼ µ· + αi + εij , 1 ≤ i ≤ r, 1 ≤ j ≤ n ● Example: sodium content in ■ 2 beer ε ∼ N(0, σ ), 1 ≤ i ≤ r, 1 ≤ j ≤ n ● One-way random effects ij model ■ 2 ● Implications for model αi ∼ N(0, σµ), 1 ≤ i ≤ r. ● One-way random ANOVA table ■ ● Inference for µ· We might be interested in the population mean, µ·: CIs, is it ● 2 Estimating σµ zero? etc. ● Example: productivity study ● Two-way random effects ■ Alternatively, we might be interested in the variability across model ● ANOVA tables: Two-way subjects, 2 : CIs, is it zero? (random) σµ ● Mixed effects model ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 7/19 Implications for model

● Today’s class ■ In random effects model, the observations are no longer ● Two-way ANOVA ● Random vs. fixed effects independent (even if ε’s are independent). In fact ● When to use random effects? ● Example: sodium content in beer 0 0 2 0 2 0 ● One-way random effects Cov(Yij, Yi j ) = σµδi,i + σ δj,j . model ● Implications for model ■ ● One-way random ANOVA In more complicated mixed effects models, this makes MLE table ● Inference for µ· more complicated: not only are there parameters in the ● 2 Estimating σµ ● Example: productivity study mean, but in the covariance as well. ● Two-way random effects ■ model In ordinary least squares regression, the only parameter to ● ANOVA tables: Two-way 2 2 (random) estimate is σ because the covariance matrix is σ I. ● Mixed effects model ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 8/19 One-way random ANOVA table

● Today’s class Source SS df E(MS) ● Two-way ANOVA r 2 Treatments SST R = n Y · − Y ·· r − 1 σ2 + nσ2 ● Random vs. fixed effects Pi=1 “ i ” µ ● When to use random effects? r n − 2 − 2 Error SSE = i=1 j=1(Yij Y i·) (n 1)r σ ● P P Example: sodium content in ■ beer Only change here is the expectation of SST R which reflects ● One-way random effects model randomness of α ’s. ● Implications for model i ● One-way random ANOVA ■ table ANOVA table is still useful to setup tests: the same F ● Inference for µ· ● 2 statistics for fixed or random will work here. Estimating σµ ● Example: productivity study ■ 2 ● Two-way random effects Under H0 : σµ = 0, it is easy to see that model ● ANOVA tables: Two-way (random) MST R ● Mixed effects model ∼ Fr−1,(n−1)r. ● Two-way mixed effects model MSE ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 9/19 Inference for µ·

● Today’s class ■ ● Two-way ANOVA We know that E(Y ··) = µ·, and can show that ● Random vs. fixed effects ● When to use random effects? 2 2 ● Example: sodium content in nσµ + σ beer Var(Y ··) = . ● One-way random effects rn model ● Implications for model ■ ● One-way random ANOVA Therefore, table ● Inference for µ· Y ·· − µ· ● 2 Estimating σµ ∼ tr−1 ● Example: productivity study SST R ● Two-way random effects q (r−1)rn model ● ANOVA tables: Two-way ■ (random) Why r − 1 degrees of freedom? Imagine we could record an ● Mixed effects model ● Two-way mixed effects model infinite number of observations for each individual, so that ● ANOVA tables: Two-way (mixed) Y i· → µi. ● Confidence intervals for variances ■ · ● Sattherwaite’s procedure To learn anything about µ we still only have r observations (µ1, . . . , µr). ■ Sampling more within an individual cannot narrow the CI for µ·.

- p. 10/19 2 Estimating σµ

● Today’s class ■ From the ANOVA table ● Two-way ANOVA ● Random vs. fixed effects ● When to use random effects? 2 E(SST R/(r − 1)) − E(SSE/((n − 1)r)) ● Example: sodium content in σ = . beer µ ● One-way random effects n model ● Implications for model ■ Natural estimate: ● One-way random ANOVA table ● Inference for µ· 2 SST R/(r − 1) − SSE/((n − 1)r) ● 2 Estimating σµ Sµ = ● Example: productivity study n ● Two-way random effects model ■ ● ANOVA tables: Two-way Problem: this estimate can be negative! One of the (random) ● Mixed effects model difficulties in random effects model. ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 11/19 Example: productivity study

● Today’s class ■ Imagine a study on the productivity of employees in a large ● Two-way ANOVA ● Random vs. fixed effects manufacturing company. ● When to use random effects? ● Example: sodium content in ■ beer Company wants to get an idea of daily productivity, and how ● One-way random effects model it depends on which machine an employee uses. ● Implications for model ● ■ One-way random ANOVA Study: take m employees and r machines, having each table ● Inference for µ· employee work on each machine for a total of n days. ● Estimating σ2 µ ■ ● Example: productivity study As these employees are not all employees, and these ● Two-way random effects model machines are not all machines it makes sense to think of ● ANOVA tables: Two-way (random) both the effects of machine and employees (and ● Mixed effects model ● Two-way mixed effects model interactions) as random. ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 12/19 Two-way random effects model

● Today’s class ■ ∼ ·· ≤ ≤ ≤ ≤ ≤ ● Two-way ANOVA Yijk µ + αi + βj + (αβ)ij + εij , 1 i r, 1 j m, 1 ● Random vs. fixed effects k ≤ n ● When to use random effects? ● Example: sodium content in ■ ∼ 2 ≤ ≤ ≤ ≤ ≤ ≤ beer εijk N(0, σ ), 1 i r, 1 j m, 1 k n ● One-way random effects 2 model ■ ∼ ≤ ≤ ● Implications for model αi N(0, σα), 1 i r. ● One-way random ANOVA ■ 2 table βj ∼ N(0, σβ), 1 ≤ j ≤ m. ● Inference for µ· ● Estimating σ2 ■ 2 µ (αβ)ij ∼ N(0, σ ), 1 ≤ j ≤ m, 1 ≤ i ≤ r. ● Example: productivity study αβ ● Two-way random effects ■ 0 0 0 0 2 0 2 0 0 2 0 0 0 2 model Cov(Yijk, Yi j k ) = δii σα +δjj σβ +δii δjj σαβ +δii δjj δkk σ . ● ANOVA tables: Two-way (random) ● Mixed effects model ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 13/19 ANOVA tables: Two-way (random)

● Today’s class SS df E(SS) ● Two-way ANOVA r 2 SSA = nm Y ·· − Y ··· r − 1 σ2 + nmσ2 + nσ2 ● Random vs. fixed effects Pi=1 “ i ” α αβ 2 ● When to use random effects? m − ··· − 2 2 2 SSB = nr j=1 “Y ·j· Y ” m 1 σ + nrσβ + nσαβ ● Example: sodium content in P r m 2 2 2 beer SSAB = n Y · − Y ·· − Y · · + Y ··· (m − 1)(r − 1) σ + nσ Pi=1 Pj=1 “ ij i j ” αβ ● One-way random effects r m n SSE = (Y − Y ·)2 (n − 1)ab σ2 model Pi=1 Pj=1 Pk=1 ijk ij ● Implications for model ■ 2 ● One-way random ANOVA To test H0 : σα = 0 use SSA and SSAB. table ● Inference for µ· ■ 2 To test H0 : σ = 0 use SSAB and SSE. ● 2 αβ Estimating σµ ● Example: productivity study ● Two-way random effects model ● ANOVA tables: Two-way (random) ● Mixed effects model ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 14/19 Mixed effects model

● Today’s class ■ In some studies, some factors can be thought of as fixed, ● Two-way ANOVA ● Random vs. fixed effects others random. ● When to use random effects? ● Example: sodium content in ■ beer For instance, we might have a study of the effect of a ● One-way random effects model standard part of the brewing process on sodium levels in the ● Implications for model ● One-way random ANOVA beer example. table ● Inference for µ· ■ Then, we might think of a model in which we have a fixed ● 2 Estimating σµ ● Example: productivity study effect for “brewing technique” and a random effect for beer. ● Two-way random effects model ● ANOVA tables: Two-way (random) ● Mixed effects model ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 15/19 Two-way mixed effects model

● Today’s class ■ ∼ ·· ≤ ≤ ≤ ≤ ≤ ● Two-way ANOVA Yijk µ + αi + βj + (αβ)ij + εij , 1 i r, 1 j m, 1 ● Random vs. fixed effects k ≤ n ● When to use random effects? ● Example: sodium content in ■ ∼ 2 ≤ ≤ ≤ ≤ ≤ ≤ beer εijk N(0, σ ), 1 i r, 1 j m, 1 k n ● One-way random effects 2 model ■ ∼ ≤ ≤ ● Implications for model αi N(0, σα), 1 i r. ● One-way random ANOVA ■ table βj , 1 ≤ j ≤ m are constants. ● Inference for µ· ● 2 ■ 2 Estimating σµ (αβ)ij ∼ N(0, (m − 1)σαβ/m), 1 ≤ j ≤ m, 1 ≤ i ≤ r. ● Example: productivity study ● Two-way random effects ■ Constraints: model ● ANOVA tables: Two-way ◆ m (random) j=1 βj = 0 ● Mixed effects model ◆ Pr ● ≤ ≤ Two-way mixed effects model i=1(αβ)ij = 0, 1 i r. ● ANOVA tables: Two-way ◆ P 0 0 2 (mixed) Cov ((αβ)ij, (αβ)i j ) = −σαβ/m ● Confidence intervals for variances ■ 0 0 0 ● Sattherwaite’s procedure Cov(Yijk, Yi j k ) = m−1 1 0 2 0 2 − − 0 2 0 0 2 δjj σβ + δii m σαβ (1 δii ) m σαβ + δii δkk σ 

- p. 16/19 ANOVA tables: Two-way (mixed)

● Today’s class SS df E(MS) ● Two-way ANOVA SSA r − 1 σ2 + nmσ2 ● Random vs. fixed effects α m 2 ● When to use random effects? j=1 βi SSB m − 1 σ2 + nr P + nσ2 ● Example: sodium content in m−1 αβ beer SSAB (m − 1)(r − 1) σ2 + nσ2 ● One-way random effects αβ SSE r m n Y − Y 2 n − ab σ2 model = i=1 j=1 k=1( ijk ij·) ( 1) ● Implications for model P P P ■ 2 ● One-way random ANOVA To test H0 : σα = 0 use SSA and SSE. table ● Inference for µ· ■ To test · · · use and . ● 2 H0 : β1 = = βm = 0 SSB SSAB Estimating σµ ● Example: productivity study ■ To test H : σ2 use SSAB and SSE. ● Two-way random effects 0 αβ model ● ANOVA tables: Two-way (random) ● Mixed effects model ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 17/19 Confidence intervals for variances

● Today’s class ■ Consider estimating 2 in the two-way random effects ● Two-way ANOVA σβ ● Random vs. fixed effects ● When to use random effects? ANOVA. A natural estimate is ● Example: sodium content in beer 2 ● One-way random effects σβ = nr(MSB − MSAB). model ● Implications for model b ● One-way random ANOVA ■ What about CI? table ● Inference for µ· ■ 2 2 ● 2 A linear combination of χ – but not χ . Estimating σµ ● Example: productivity study ■ 2 ● Two-way random effects To form a confidence interval for σβ we need to know model 0 ● ANOVA tables: Two-way distribution of a linear combinationb of MS · s, at least (random) ● Mixed effects model approximately. ● Two-way mixed effects model ● ANOVA tables: Two-way (mixed) ● Confidence intervals for variances ● Sattherwaite’s procedure

- p. 18/19 Sattherwaite’s procedure

● Today’s class ■ Given k independent MS·’s ● Two-way ANOVA ● Random vs. fixed effects ● When to use random effects? k ● Example: sodium content in beer L ∼ ciMSi ● One-way random effects X model i=1 ● Implications for model b ● One-way random ANOVA table ■ Then ● Inference for µ· ● 2 Estimating σµ dfT L 00 2 ● ∼ Example: productivity study “ χdfT . ● Two-way random effects E(Lb) model ● ANOVA tables: Two-way (random) where b ● Mixed effects model 2 ● Two-way mixed effects model k ● c MS ANOVA tables: Two-way  i=1 i i (mixed) P dfT = ● Confidence intervals for k 2 2 variances i=1 ci MSi /dfi ● Sattherwaite’s procedure P where dfi are the degrees of freedom of the i-th MS. ■ (1 − α) · 100% CI for E(L): b dfT × L dfT × L LL = 2 , LU = 2 χ − b χ b dfT ;1 α/2 dfT ;α/2 - p. 19/19