Alternative Approaches to Mixed Linear Models

George W. Pasdirtz, Hazleton Laboratories

The inclusion of fixed effects and random effects arrive at the computer laboratory, the associated (independent variables not under control) in the with each operator is an uncontrolled component of the require special SAS· procedures (pROC study's overall variance. There may then be a general V ARCOMP or PROC MIXED) to estimate the component of variance associated with these particular of the random components. When these procedures are males and females and a specific component of variation inconvenient or inappropriate for a particular application, associated with their use of a particular computer. alternative techniques are available. This paper presents a small-scale simulation study investigating the properties There are a number of diff'lCult problems associated with of two alternative approaches. the estimation of variance components: Introduction • Hand calculation is required when using PROC GLM 10 compute method-of-moments variance estimates. • The more elegant maximum likelihood (ML) techniques The treatment of random components in ordinary least can produce estimates which are outside the parameter squares (OLS) is one of the "messy· problems (Milliken space (e.g., negative variances), are biased downwards in and Johnson, 1984) in analysis. A first step in small samples, and can take excessive computing confronting mixed-model issues is an understanding of the times. sampling plan: • None of the techniques have good properties in unbalanced designs. • If the values of a classification factor are under experimental control or include all possible levels of Milliken and Johnson (1984, Chapters 18-23) and the population, the factor is afued effect. Schwarz (1993) document the standard tactical difficulties • If the values of a classification factor result from a of using mixed-model computer algorithms. There may be random process which is not under experimental a number of additional strategic problems. control, the factor is a random effect. In an applied research setting it is never clear whether The messiness is a result of added sources of variation equal variances exist, even within fixed factors. For introduced by the random components. The added variance example, it may be that one of the computing machines can obscure the fixed effects unless it is removed from the is more sophisticated and thus produces greater operator error term of the model. variability. Levene (1960) developed a test for homogeneity of variance that was generalized by Draper A standard example is an industrial in which and Hunter (1969). 11Ie Levene statistic estimates the different classes of computing machines are being tested variance of each independent variable using the absolute with different operators. The determination of whether values of the residuals from the initial model fit. machines and/or operators are random or fixed effects Although the Levene test is not asymptotically depends on the sampling plan and on the inference space distribution free (Miller, 1968), a Monte Carlo study 10 which the results are being projected (McLean, Sanders, conducted by Brown and Forsythe (1974) indicated that and Stroup, 1991). If comparisons are restricted to the with at least ten observations per cell. asymptotic computing machines of only two manufacturers and if normality is not a problem. Furthermore, increasing differences in manufacturing process are ignored, then the percentages of trimming (from ten to firty percent, which machines chosen are a fixed factor. If the results are to be is the ) convert the Levene estimate into a very generalized 10 all computing machines that have similar robust test. characteristics and the particular machines were only chosen for convenience, the variance introduced by the After the results of a robust Levene test are available, how specific machines needs to be controlled in order 10 make do we make use of the information? The mixed-model the wider inference. literature assumes that a variance component estimation should take place even without a prior test of A similar analysis of the sampling plan for operators homogeneity. However, Freedman and Peters (1984) argue yields a more complicated picture. Assume that each that complicated estimators of variance may be of little operator can be classified by observable physical value in small samples and may be worse than no characteristics. If a random sample is drawn from the estimate at all. For example, a number of textbook data population of all operators with those characteristics (say sets used to illustrate variance component· estimation male and female), operators are a fixed effect. If operators contain under twenty observations with as few as two are not sampled but rather assigned to machines as they observations per cell.

Proceedings of MWSUG '94 and Data Visualization 273 is conventionally assumed to be zero, but is not This paper investigates two simple estimators that can be guaranteed to be zero by the OLS estimation itself. applied to mixed-model problems: The SAS code to generate data for a 2 x 3 design with 'k • The Levene estimate of variance followed by Weighted replications per cell is: (WLS). • The simple average of OLS coefficients across do rep = 1 to 'k; replications. do op = 1 to 3; do machine = 1 to 2; Both estimators assume that the variance of each y = 'mu + (machine-I) * 'b * 'sigma independent variable is open to question, possibly for + op * &9 * 'sigma * rannor(O) different reasons. If no statistical differences in variances + &sigma * rannor(O) were detected using the Levene test, then an unweighted output; OLS estimate could be used regardless of the sampling end; end; end; plan. Notice that there are two random number streams and that, A small-scale Monte Carlo study was conducted to in this case, there is no between operators and investigate: machines. Notice also that each effect is scaled by &b and &s relative to &sigma. The scaling assumption will • Asymptotic bias (the convergence of parameter prove useful later in describing the WLS model. With estimates as a function of increasing sample size). minor modifications. this code was used in the Monte • Misspecification (the effect of erroneously applying a Carlo study presented below. WLS estimate to an equal-variance problem). The full for one with main The SAS code necessary to apply each estimator in effects and interactions is: ,. practice is included. " F, " F, " The Mixed Model ',. 110100100000 If, " 110010010000 , In the standard two-way linear model: " 'h. 1 1 0 0 0 1 0 0 I 0 0 0 If, + " Yiji =p+aj +Pj + rjj+ejjl 101100000100 FR" " 2 0 0 0 0 0 1 0 = PHlj + Pj + rij + NID(O,cr ) I 0 I I 0 FR. " I 0 I 0 0 I 0 0 0 0 0 1 FR,. " P is the grand . a j and Pj are fixed effects. rjj is FIf. '. the interaction term, and eijk is the error term such that FR. ". the Yjji are distributed independently and identically FIf. ". normal with zero mean and common variance, 2 N1D(O, cr ). The fixed machine effect (F) has two levels, the random operator effect (R) has three levels and the interaction In the two-way mixed model: (FR) has six levels.

Yiji = p+ a i + Pj + rij + eijl 2 Variance Estimation = Il + aj + NID(O,cr;)+ rij + NID(O,cr ) 2 =Il + aj + Pj + NID(O.cr~)+ N1D(O.cr ) In matrix terms, the linear model is:

y=Xb+e Il is the grand mean, aj is the fixed effect. Pj is the random effect, rij is the interaction term, and eijk is the where X is the n Xm design matrix. The usual OLS 2 error term distributed NID(O, cr ). The distributions of assumptions are: the random components are also NID with variances 1 2 depending on whether variance is introduced by the E(e) = 0, V(e)=lcr , and e-NID(O,lcr ) random factor, the random factor in combination with the fixed factor (e.g., different operator variances for each To test the homogeneity-of-variance assumption, Levene's machine), or both. The mean of each random component test reestimates the model:

274 Statistics and Data Visualization Proceedings of MWSUG '94 A disadvantage of the full WLS estimator is the n x n dimension of V and P. Laird and Ware (1982) have suggested an estimated weighted least squates (EWLS) using the absolute values of the residuals. The estimated procedure in which V is formed by iterative estimation of: coefficients, c, are a robust estimate of the relative associated with each column of the design matrix. The advantage of using lei is that the b = (tXiV-IX;)-I(tXiV-Iy;) ,-I .=1 distribution of e2 can be matkedly skewed (Carroll and Ruppert, 1988). l ' V=k- I I:(YI- X/b)(YI- X,";) The estimated standard deviation of each observation is: 1=1 where the dimension of V is now k x k and there ate k lei = Xc replications of the basic design matrix. Lindstrom and where lei is an n x I vector of estimated standatd Bates (1988) have compared EWLS to the standard ML deviations with the estimation error in u removed. estimators and found that convergence is slow. A natural extension of EWLS would be to apply the Levene weights to the estimate of V. Carroll and Ruppert (1988) suggest that as few as two iterations would then be needed for convergence. Weighted Least Squates (Draper and Smith, 1981: 108- 116) drops the homogeneity of variance assumption for a Gumpertz and Pantula (1989) have studied a further simplification: full n Xn variance- matrix, V, where: l

E(e)=O, V(e)=Vc:r, and e-NlD(O,Vc:r) b=k-lI,bj 1=1 If the are V becomes: 0, the mean of the individual regression coefficients across replications (BBAR). Each replication of the full design matrix and its associated regression curve is a random drawing from a larger population of response curves. By comparison, EWLS is a weighted mean of population estimates.

The coefficients in b are a random drawing from some k­ variate population with: where the w's are weights applied to the common variance, (72. Thus, observations that have larger l ' variances receive smaller weights. V(b) = (k _1)-1 I, (b; - iiXb; - ii) i=l The WLS estimator: l (il = [k(n- m)rl I, [yiY; -biXiYi] i=1

and are thus random coefficients. is then computed by decomposing the weighting matrix: In their Monte Carlo study, Gumpertz and Pantula found V-I = p-o.sp-o.s that BBAR had greater power in small samples than did EWLS. To cover a broad of situations, however. they recommend 50 replicates. Carroll and Ruppert and applying OLS 10 the reweighted X and y matrices. suggest, however, that effective WLS estimates can be obtained with far fewer observations.

As a result of reweighting, the WLS estimator has SAS Coding minimum variance. As long as the Levene weights are proportional to the true variances, the reweighted mixed­ model estimates will also have minimum variance. The WLS estimator can easily be coded in SAS using the WEIGHT command in PROC GLM.

Proceedings of MWSUG '94 Statistics and Data Visualization 275 proc reg data = a; * OLS step; * by rep; • Scaling for random factor variances [0.25. 1.0] which model 'dvar a 'model were taken as a direct linear function of operator output out=b r=yr: classification. data c; • Estimators [OLS. WLS, BBARJ. set bl • True models [OLS, MIXED]. yr a abs(yrl: proc reg data = C; * Levene step: model yr = 'model; Fifty Monte Carlo simulations were conducted for each output outed p=vhat; combination. A 2a difference (&b = 2) in machine data e; performance was assumed in all simulations. with a =1 set d: and JJ =SO. Three observations per cell provide adequate vhat - vhat**(-.S); power (p ~ 0.46) at conventional levels (p S 0.05) to proc reg data = e outest = f; * WLS Step; detect a 2a difference, while & k = 5 provides higher * by rep; power (p ~ 0.79). Fifty Monte Carlo simulations provide weight vhat: adequate power (p ~ 0.59) to detect a 0.25a difference model 'dvar = 'model; * proc data = f; between estimators. All power calculations were computed with JMP·.

Notice that negative or zero weights effectively eliminate The results are presented in Appendix L An analysis of the observation. To compute the BBAR estimator. covariance was conducted on the machine effect (dl) and comment out the weight statement, uncomment the by the average of the non-zero interaction terms (FR). rep; statements. and uncomment the proc means Controlling for & k and & s. there were no significant statement. Similar coding can be developed for PROC differences in estimators. There was a significant GLM. difference (p S Om) in root mean square error L RMSE_) between OLS and WLS (not displayed). with WLS being Monte Carlo Study closer to the true value. The BBAR estimator does not produce a _ RMSE_ in SAS for the full design matrix.

Simulation provides a way of comparing estimated An inspection of the regression coefficients. however, parameters with known values to resolve mathematical indicates that there were numerical differences. The Irue and theoretical problems. value of the dl effect in the simulation was -2. OLS came closest to the true value at -2.03, followed by • The Levene estimate is not asymptotically dislribution BBAR at -1.88 and finally WLS at -2.43. Notice also free (Miller. 1968) but might still produce relatively from the interaction term (N*MOOEL) that BBAR was accurate estimates of weights in V. least affected by sample size. Finally. the mixed model does indeed introduce bias (0.12), but the effect is not • Since is a concave function of V. WLS (X'V-txt statistically significant given the Monte Carlo sample estimators may have a downward bias. especially when size used in the simulation. V is variable (Freedman and Peters. 1984). • The WLS estimate is justified by asymptotic reasoning. In terms of the random components (FR) which were set meaning that the theoretical results hold as n ~ 00. at zero in the simulation. BBAR is the least biased Weighting with small sample size may be worse than (0.022). followed by WLS (0.053) and OLS (0.20761). no weighting at all. Again, none of these biases were significant. A small-scale Monte Carlo study will generate information about the behavior of estimators in finite Discussion samples. This paper investigated a classical "messy" problem in • Bias can be investigated by comparing observed statistical analysis: the mixed model. Mathematically. the parameter estimates with "true" values. presence of variance components introduces additional error that can obscure fixed effects. When comparing bias(8) =(8- 9). models estimated in OLS to WLS or EWLS, a reduction • Asymptotic convergence can be investigated by in RMSE and an increase in model sum of squares is increasing sample size and observing the effect on bias. often observed. Thus. a strong argument can be made both • Misspecification can be investigated by applying WLS mathematically and practically that more complicated to data generated with homogeneous variances. procedures must be applied to the mixed model.

The design of the Monte Carlo study thus involved Simple estimators such as WLS and BBAR did Dot different settings for the macro variables &n. &b. &s. introduce significant bias. Sophisticated ML techniques. &mu. and &sigma in the SAS code above: . on the other hand. can produce very biased estimates (Milliken and Johnson. 1984). especially in unbalanced • Sample sizes [3 and 5 observations per cellJ. models. Even though OLS does an acceptable job of estimating fixed effect parameters. the use of WLS or

276 Statistics and Data Visualization Proceedings of MWSUG '94 BBAR is indicated in situations where large variance Gumpertz, M. and S. G. Pantula (1989), "A Simple components relative to fIXed effects are expected. Approach to Inference in Random Coefficient Models," The American , 43. 203-210. This study did not investigate the effect on p-values and statistical decision making. The reduction in RMS E Laird, N. M. and J. H. Ware (1982), "Random-Effects observed for WLS suggests that it would be better than Models for Longitudinal Data," Biometrics, 38, 963-974_ OLS. The BBAR estimates of v(ii") and (72 suggested by Levene, H. (1960). "Robust Tests for Equality of Gumpertz and Pantula (1989) would require further coding Variances," in Contributions to Probability and Statistics. in PROC IML to produce p-values. (eds.) I. Olkin et. al.• Ch. 25 (Stanford, CA: Stanford University Press) 278-292-

Lindstrom. M. J. and D. M. Bates (1988), "Newton­ Raphson and EM Algorithms for Linear Mixed-Effects References Models for Repeated-Measures Data; Journal of the American Statistical Association. 83,1014-1022.

Brown, M. B. and A. B. Forsythe (1974), "Robust Tests McLean. R. A., W. L. Sanders, and W. W. Stroup for the Equality of Variances," Journal of the American (1991). "A Unified Approach to Mixed Linear Models: Statistical Association, 69,364-367. The American Statistician. 45.54-64.

Carroll. R.J. and D. Ruppert (1988). Trans/ormation and Miller, R. G. (1968). "Jackknifing Variances," Annals of Weighting in Regression, New York: Chapman and Hall. , 39, 567-582.

Conover, W. I., M. E. Johnson, and M. M. Johnson Milliken, G. A. and D. E. 10hnson (1984). Analysis of (1981), "A Comparative Study of Tests for Homogeneity Messy Data, (New York: Van Nostrand Reinhold). of Variances, with Applications to the Outer Continental SbelfBidding Data," Technometrics, 23, 351-361, Schwarz, C. J. (1993), "The Mixed-Model ANOVA: The Truth, the Computer Packages, the Books. Part I: Draper, N. R. and W. G. Hunter (1969), Balanced Data." The American Statistician. 47,48-59. "Transformations: Some Examples Revisited," Technometrics, 11.23-40. *SAS® and JMp® are registered trademarks or the SAS Institute. Inc., Cary, NC, USA. Draper, N. R. and H. Smith (1981). Applied , Second Edition. New York: Wiley. Author

Freedman, D. A. and S. C. Peters (1984), "Bootstrapping George W. Pasdirtz, Ph.D. a Regression Equation: Some Empirical Results," Journal Hazleton Laboratories of the American Statistical Association, 79, 97-106. 3301 Kinsman Boulevard Madison. WI 53704

Proceedings of MWSUG '94 Statistics and Data Visualization 277 Appendix 1. Estimation Results.

Number of observations in data set = 2401

Dependent Variable: FR Sum of Mean Source OF Squares Square F Value Pr > F

Model 8 3.32861071 0.41607634 0.55 0.8219

Error 2392 1820.98611018 0.16128182

corrected Total 2400 1824.31472149

R-Square c.v. Root MSE FR Mean

0.001825 -4214.967 0.8725147 -0.0207004

T for HO: Pr > ITI Std Error of Parameter Estimate Parameter=O Estimate

INTERCEPT -.0058395879 B -0.01 0.9947 0.87983568 S -.0365280816 -0.71 0.4419 0.04749368 N 0.0029943217 B 0.13 0.8943 0.02252823 MODEL OLS 0.2076187200 B 0.23 0.8112 0.89830882 WLS 0.0531939033 B 0.06 0.9528 0.89830882 BBAR 0.0215283188 B 0.02 0.9803 0.87331409 TRUE 0.0000000000 B N*MODEL OLS -.0423070234 B -0.86 0.3890 0.04909914 WLS -.0062355856 B -0.13 0.8990 0.04909914 BBAR 0.0000000000 B TRUE 0.0000000000 B TYPE MIXED -.0599499207 B -1.68 0.0925 0.03562026 OLS 0.0000000000 B START 0.0000000000 B

Dependent variable: 01 Sum of Mean Source OF Squares Square F Value Pr > F

Model 8 26.96206074 3.37025759 0.69 0.6991

Error 2392 11652.75431146 4.87155281

Corrected Total 2400 11679.71637220

R-Square C.V. Root MSE 01 Mean

0.002308 -111.8785 2.2071594 -1.9728177

T for HO: Pr > ITI std Error of Parameter Estimate 'Parameter=O Estimate

INTERCEPT -1.736938645 B -0.78 0.4352 2.22567911 5 0.173992998 1.45 0.1471 0.12014254 N -0.061311921 B -1.08 0.2821 0.05698861 MODEL OLS -0.294168402 B -0.13 0.8970 2.27240913 WLS -0.687172558 B -0.30 0.7624 2.27240973 BBIIR -0.138551713 B -0.06 0.9500 2.20918175 TRUE 0.000000000 B N*MODEL OLS 0.037574910 B 0.30 0.7623 0.12420380 WLS 0.122466802 B 0.99 0.3242 0.12420380 BBAR 0.000000000 B TRUE 0.000000000 B TYPE MIXED 0.120166233 B 1. 33 0.1825 0.09010691 OLS 0.000000000 B START 0.000000000 B

278 Statistics and Data Visualization Proceedings of MWSUG '94