A Review of Two Different Approaches for the Analysis of Growth Data Using

THE STATISTICAL SOFTWARENEWSLETTER 583

A review of two different approaches for the analysis of growth data using longitudinal mixed linear models: Comparing hierarchical linear regression (ML3, HLM) and repeated measures designs with structured covariance matrices (BMDP5V)

Rien van der Leeden 1), Karen Vrijburg 1) & Jan de Leeuw~)

1~Department of Psychometrics and Research Methodology, University of Leiden, Postbus 9555, 2300 RB Leiden,The Netherlands 2)UCLA Department of Statistics, University of California at Los Angeles, USA

Abstract: In this paper we review two approaches for method to handle growth data, theoretically, as well as the analysis of growth data by means of longitudinal in a practical sense. For the most part, shortcomings mixed linear models. In these models the individual are induced by the accompanying software, developed growth parameters, (most often) specifying polynomial within different scientific traditions. Applied to com- growth curves, may vary randomly across individuals. parable problems, the three programs produce This variation may in turn be accounted for by explain- equivalent results. ing variables. Keywords: Multilevel analysis, Mixed linear models, The first approach we discuss, is a type of multilevel MANOVA, Repeated mesaures, Growth curve models, model in which growth data are treated as having a hi- Structured covariance matrices. erarchical slructure: measurements are 'nested' within (SSNinCSDA 20, 583-605 (1996)) individuals. The second is a version of a MANOVA repeated measures model employing a structured Received: March 1995 Revised: February 1996 (error)covariance matrix. Of both approaches we examine the underlying statistical models and their inter- I. Introduction relations. Apart from this theoretical comparison we review software by which they can be applied for real In social science one frequently encounters re- data analysis: two multilevel programs, ML3 and search situations in which individuals are meas- HLM, and one repeated measures program, BMDP5V. ured repeatedly over a period of time. The main The programs are described and discussed with re- goal of such longitudinal research can be consid- spect to several more general criteria, such as data ered as the assesment of change. Often, in social setup and handling, implemented numerical routines science research, this change concerns a kind of and user friendliness, and, in particular, with respect to their application in longitudinal situations, i.e. their growth rate, e.g. biological, learning, etc. In that capabilities for the analysis of data on growth. Two case, we could call the measurements 'growth data sets are used to compare the results of analyses data'. performed by the three programs. Growth data, more specifically, can be considerd Although both ways of specifying growth curve models as measurements on one variable for the same show some shortcomings, each appears to be a fruitful (groups of) individuals on a number of consecu-

NOTE: Since this review was written, both multilevel programs have been updated. ML3 is now MLn and is able to fit multilevel models to data with up to 15 levels (Rasbash, Yang, Woodhouse & Goldstein, 1995). HLM is now combining software for the analysis of models with two or three levels. 584 THE STATISTICAL SOFTWARE NEWSLETTER five points in time. In this paper we will be con- tudinal data can be viewed as such: the meas- cerned with the analysis of individual growth as urements at multiple timepoints are 'nested' well as with the analysis of mean growth for a within the individuals. In longitudinal multilevel group of subjects. models, each individual is allowed to have his/her own growth curve, whereas the individual Because data on growth reflect a developmental growth parameters vary across individuals. For process as a function of time, a suitable method more details concerning the multilevel treatment to model them is describing the expected values of longitudinal data we refer to Laird and Ware of the observations as functions of time. It has (1982), Goldstein (1986a, 1986b, 1995), Bryk become very common to use polynomial func- and Raudenbush (1987, 1989) and Beck (1989). tions for this purpose, and to present the results Ware (1985) gives an overview of linear models in the form of (growth) curves. for longitudinal data. Generally, the analysis of longitudinal data is more complicated than that of cross sectional Longitudinal multilevel models can be consid- data, since repeated measurements for the same ered very appropriate for the modeling of growth subjects are likely to be serially dependent. A data, conceptually as well as statistically. It can useful property for growth curve models could be shown that they incorporate two important therefore be the possibility of handling more features of real growth (cf. Bryk and Rauden- complex, e.g. autoregressive, within-subject co- bush, 1987; Prosser, Rasbash and Goldstein, variance structures. 1990): observations on the different timepoints are correlated and the variance of the observations is a function of time (i.e. the between- 1.1 Models for growth data subject variation is a function of time). Longitu- Many authors have considered the analysis of dinal multilevel models allow for the description growth data from several points of view and a of the mean growth trajectory and for the estima- variety of techniques for fitting polynomial tion of individual variation around this mean. If growth curves has been proposed. An overview of grouping variables are involved (e.g. sex group), these developments, as well as a number of useful these can be used as explanatory variables to ac- references, is given by Visser (1982) and Van der count for the between-subject variation of indi- Leeden (1990). All the techniques are variations vidual growth parameters. In longitudinal multi- of a general linear model. Nevertheless, two ma- level models, differences in growth rate between jor approaches concerning models for growth groups of individuals will therefore be revealed data can be distinguished. On the one hand, a not only in different mean levels of growth rate. (more traditional) class of models can be ob- Moreover, differences between groups will be served based on the MANOVA repeated meas- modelled as different distributions of growth pa- ures design. In that case the consecutive meas- rameters within each group. Further, longitudinal ures on the growth variable are treated as sepa- multilevel models can treat unbalancedness due rate dependent variables corresponding with the to missing observations very well, since it is not different timepoints. The work of Potthoff and necessary for each individual to have the same Roy (1964) has inspired much research on this number of measurements. class of models. Some references include Rao Compared to longitudinal multilevel models, re- (1959, 1965, 1966, 1967), Khatri (1966), Grizzle peated measures models appear to be less attrac- and Allen (1969), Lee and Geiser (1972, 1975), tive for the analysis of growth data. Models of Timm (1980) and Geisser (1970, 1980). this kind are mostly concentrated upon On the other hand there is a class of growth (MANOVA-like) between-group designs and curve models emerging within the framework of thus upon the estimation of (fixed) group effects. multilevel analysis, which technique is also Hence, they are not as flexible as one should known as hierarchical linear regression analysis. wish, concerning the modeling of random varia- In the application of multilevel models, data are tion of individual growth parameters around the viewed as having a hierarchical structure. Longi- mean growth trajectory. Generally the results THE STATISTICAL SOFTWARE NEWSLE7TER 585 consist of mean growth curves estimated for Rabash and Goidstein. FILM is a program for two specified groups, found as deviations from the level analysis written by Bryk, Raudenbush and mean growth curve for the whole sample. In the Congdon. A third program we will discuss is ultimate case, groups may be reduced to indi- BMDPSV, designed to fit repeated measures viduals, and random error components could be models with structured covariance matrices and interpreted as the most basic individual variation. written by Jennrich and Schluchter. But since estimation procedures for repeated We will focuss our discussion upon models in measures model are generally not suited for N-I which the individual growth parameters may groups and assumptions for (in that case) within- vary randomly across individuals and in which subject (co)variances are fairly simple and these this variation may be, partially, explained by one (co)variances can not be modelled, this does not or more variables at the subject level. Models of provide an appealing approach. that kind go under various names, such as 'ran- However, recent developments include repeated dom coefficient regression models', 'longitudinal measures models with structured covariance ma- random effects models', etc. Because it can be trices, which are suited to handle incomplete data shown that these models can be split up into a too. Jennrich and Schluchter (1986) present a random part and a fixed part (cf. Bryk and very general model formulated at the level of the Raudenbush, 1987) we prefer the general term individual response vectors. Their model may be 'longitudinal mixed linear models'. considered the most advanced repeated measures It should be noted that the class of 'generalized model as it includes all other previously sug- linear models' (GLMs) (see e.g. McCullagh and gested approaches as special cases, such as uni- Nelder, 1989) could provide a common theoreti- variate mixed-model ANOVA and (versions of) cal framework to compare the two longitidinal MANOVA (for details see Jennrich and Schlu- data analysis approaches discussed here. Zeger, chter, 1986). In the model of Jennrich and Schlu- Liang and Albert (1988), for instance, study ex- chter, the within-subject (co)variances can be tensions to the generalized linear model for the modeled as arbitrary functions of a set of un- analysis of repeated measures data. They discuss known (co)variance parameters. The mean a 'mixed generalized linear model' of which the structure is specified by a set of regression pa- longitudinal multilevel model (with two-levels) rameters defined by a (fixed) between-subject and the generalized MANOVA repeated meas- design matrix. The general parametrization of ures model of Jennrich and Schluchter are special the within-subject covariance matrix makes it cases. possible to fit a numer of different structures, among which random effects and time-series The reason for not choosing this perspective is a structures are the most interesting for the analy- practical one. GLMs are usually fitted using the sis of growth data. GLIM package (Francis, Green and Payne, 1993), which has been designed as an interactive 1.2 Purpose of the paper 'tool kit' for statistical modeling. With GLIM, 'longitudinal mixed generalized linear models' In this paper we will review both approaches for should be developed (programmed) as 'macros'. the analysis of growth data described above. At As a result, the GLIM software is of a complete first, we will examine the underlying statistical different nature than the more or less 'tailored' models and their interrelations. Although they software discussed in this paper, and less suited are originating from different fields of applica- for a comparison. tion, their elementary concepts are basically alike. Secondly, we will give a review and com- The comparison of the three programs we discuss here will be limited in the following way: parison of several computer programs with which these models can be applied for data analysis. (1) Not all programs could be run on the same Concerning longitudinal multilevel models we computer system or in similar computer and will discuss 1~K,3 and HLM. ML3 is software for user environments. At the time of writing this two or three level analysis, written by Prosser, paper, ML3 and HLM were available as MS- 586 THE STATISTICAL SOFTWARE NEWSLETTER

DOS applications and ran on a 80286-based nomial vectors (see e.g. De Lury, 1950), or it PC, but BMDP5V was run on an AMDAHL may contain an explanatory variable, as well as 5870 mainframe; powers of that variable, corresponding to the T occasions, such as 'weight' or 'age'. (2) our discussion will be focussed on those features of ML3, HLM and BMDP5V relevant Model (2.2) is well known in the literature. It can for the modeling of growth data. A more be considered as a version of the polynomial re- complete discussion and comparison of both gression model with the special feature that the multilevel programs is given in Kreft, De observations, i.e. the repeated measurements l.aeuw and Van der Leeden (1994). BMDP5V contained by yj, are not indepently distributed. is extensively discussed in Dixon (1988). The problem of estimating ~ j can, for instance, be solved using generalized least squares (cf. II. Comparison of models Rao, 1973; Timm, 1975; Visser, 1982). Individual growth can often be modeled as a In the sequel we will take this model for individ- polynomial function of time. Suppose a subject j ual growth as a common basis to compare the is measured on some response variable Y on T formulation of longitudinal mixed linear models consecutive occasions, indicated by i. Then, the within a multilevel framework and within a gen- basic model to fit these responses with a poly- eralized MANOVA repeated measures design. In nomial growth curve looks like this kind of models, the individual growth parameters vary randomly across a group of sub- Yij" 130j + 131j tij + 132jt2 ,ij + "" + eij" (2.1) jects and this variation may be accounted for by variables on the group level. Using more convenient matrix notation, model (2.1) is written as 2.1 Longitudinal mixed linear models in a multilevel framework yj - TI3j + ej, (2.2) Multilevel models are designed for the analysis of data with hierarchical structure. The term where yj is the (T x 1) response vector containing 'multilevel' points out that we study nested mem- the repeated measurements for subject j, T is a bership relations among the units of observation. (T x r, r < T) matrix of known constants, 13j_is a Models for the analysis of hierarchical data and (r x 1) vector of individual parameters specitying their corresponding estimation procedures have the shape of the growth curve for person j and ej been extensively discussed in the literature. is a (T x 1) vector of random error components. Among others we mention Goldstein (1986a; The matrix T can be defined as 1995), Raudenbush (1988), Bock (1989), Bryk and Raudenbush (1992) and Longford (1993).

T-I" Usually, in multilevel model building, individu- • 1 tj t2~ ... t l als are considered as the lowest (level 1) level T-I units. For instance, students are members of 1 t2 t 2 ... t 2 classes, and classes (level 2) are nested within T =. (2.3) .o, schools. Growth data can be interpreted as hav- T-I ing a hierarchical structure too. Data from indi- 1 tT t 2 ... t T viduals that are measured at a number of consecutive points in time can be interpreted as having a two-level structure: individuals are Since T is of order (T x r, r < T), the degree of considered as level 2 units and observations as the polynomial fitted for person j is r-1, and T-1 level 1 units, that is, measurements are nested at most. Note that in this formulation, it is not within individuals. In that way, a longitudinal necessary for each individual to have T meas- mixed linear model is easily formulated within a urements. Matrix T may contain a set of poly- multilevel framework. Equation (2.2) serves as THE STATISTICAL SOFTWARE NEWSLETTER 587 the within-subject (level 1) model for person j multilevel (longitudinal) model as a 'mixed' (which is a level 2 unit). Clearly, each person has model. Some useful references concerning the its own set of growth parameters, i.e. each per- multilevel treatment of longitudinal data can be son's growth curve is unique. The variation of the found in section 1. individual growth parameters across the total The variance of the random part, i.e. the group is modeled by the between-subject (level 2) (co)variances among the elements of y. that are J model (2.4b). Hence, the complete two-level not accounted for by the fixed part, is given by model for growth data is given by Var (Tuj + ej) - Var (yjITZT) - yj - T[3j + ej, (2.4a) gj - TROT' + o2I. (2.6) [3j - Z T + uj. (2.4b) The assumptions described above, especially that In equation (2.4b), Z is a (r x q) between-subject ej - N(0, o2I), lead to a relatively simple covari- design matrix with known, fixed elements, T is a ance structure on the first (individual) level: the (q x 1) vector of fixed coefficients and uj is a (r x error terms are equal and uncorrelated across the 1) vector of random error components.-The be- timepoints. Although it is possible to formulate a tween-subject model (2.4b) is formulated in a more general model incorporating a complex general way. For instance, if Z - I, we have a level 1 covaxiance structure, Bryk and Rauden- model of simple variation in which T is a vector busch (1987) argue that this simple structure is containing parameters indicating the mean sufficient for most longitudinal applications with growth curve over all individuals, whereas the a limited number of timepoints. elements of u.- denote the departures from this J mean curve for each individual j. More elabo- 2.2 Longitudinal mixed linear models in a rated models emerge if Z contains dummy vari- MANOVA repeated measures context ables coding subgroups of individuals and/or other explanatory variables that could account for There exists a huge amount of literature concern- the variation of the ~j's. In that way it is possible ing MANOVA and MANOVA repeated meas- to study the variability of the growth curve ccef- ures models. They are discussed extensively e.g. ficients as a function of the differences between in Kirk (1982) in an experimental framework, the individuals on some characteristic that re- and in many textbooks on multivariate data mains fixed across occasions (e.g. sex). Another analysis, such as Timm (1975). Relevant for our extension to the model would be to include time present discussion is the following. Typically, varying covadates to T at level 1. MANOVA models are focussed upon the between-group~subject side of the analysis. The to- The within- and between-subject models can be tal variance to be explained in the dependent combined into one single equation model that is variables, is accounted for by differences in group given by membership as good as possible. Main effects, as well as interaction effects are estimated by look- yj ffi TZ T + Tuj + ej. (2.5) ing at differences between the overall mean and the group means. Usually, this takes place under Usually it is assumed that the elements of ej are fairly simple assumptions for the within- independently distributed as ej ~ N(0, o~I), that group~subject covariances. the level 2 random terms in uj are distributed as MANOVA models therefore typically emphasize ~ N(0, f~), and that the level 1 random terms, the fixed part of the modeling. An 'explaining' ~ e elements of e j, are distributed independently structure of the means is estimated and the from the level 2 random terms in uj. within-group~subject covariances are considered In the combined model (2.5) the term TZT can be as random error, that is, they are the final part of called the fixed part and the term Tuj + ej the the variance in the dependent variables that can random part. This explains why we can view the not be explained by group membership. 588 THE STATISTICAL SOFTWARE NEWSLETTER

In repeated measures applications the fixed part where vjis as before and Z is a known (T x r) of the model is expanded. To fit polynomial matrix • The (r x 1) bj and (T x 1) r.j are inde- growth curves of a certain degree, the set of ex- pendent random vectors. It is assumed that the bj planatory variables, first indicating group mem- and rj are independently, identically distributed bership alone, is supplemented with within- as bj - N(0,~) and rj - N(0, o2I), respectively. If subject variables (and powers of that variables) Z is specified accordingly, b, contains the corresponding to the different timepoints, such as (random) growth parameters forJ subject j. This 'age'. On the individual level, we can write for holds if its columns contain the same (within- such a model subject) explaining variables (and powers of that variables). The rj contain the remaining random yj - X15 + vj. (2.7) error which can not be accounted for by the fixed part modeled with X and the random part mod- Again, y.. contains the T repeated measurements • J eled with Z. As in the previous section both for a single subject j, X is a known (T x q) design models (2.7) and (2.8) can be combined into one matrix and 1~ is a (q x 1) vector of regression single equation model coefficients• The. v..J are vectors of random error components. Since the repeated measures are yj - X15 + Zbj + rj. (2.9) likely to be dependent, it is assumed that the v.. are independently distributed as N(0,~j). Thd As it holds for model (2.5), this model can be columns of X correspond to the different terms in split up into a fixed part and a random part too. the model, i.e. the elements of 15. Normally the The variance of the random part in (2.9) is given first column is a vector of ones, neccesary to by specify the intercept. Further columns code (between-subjec0 grouping variables and vat (Zbj + rj ) - var (yjlX15) = (within-subject) explaining variables (and powers of that variables). The fixed parameters in 15 can ~j - Z~Z' + 02I. (2.10) be combined so that we find the parameters of a mean growth curve for the whole group and of A comparison of models (2.5) and (2.6) with separate (mean) curves for the subgroups, found (2.9) and (2.10) shows that both approaches, i.e. as deviations from this mean curve. the multilevel and the MANOVA repeated meas- Clearly, model (2.7) is conceptually not aimed at ures framework, if specified accordingly, may the modeling of individual growth. Only if X lead to the same longitudinal mixed linear model would be specified in such way that each individ- (see also Vrijburg (1991) for a comprehensive ual forms a subgroup, we could estimate individ- discussion on this comparison). If we let X =, X v, ual growth curves. However, this option is not Z - T R, b. - u. and rj - e. in (2.9), if we very attractive from a statistical point of view, let15 TTZ' - X F and 41" ffi ~R in (2.5)J (with the sub- not in the least because we would be confronted scripts F and R indicating fixed and random, re- with the problem of incidental parameters: each spectively), and if we let ~ - f~ in (2.10), we new individual adds another set of parameters to can write this longitudinal mixed linear model as the model (see e.g. Kendall & Stuart, 1979; Van der Leeden, 1990). yj -. XFY + TRuj + ej, (2.11) So, even though the fixed part of the repeated measures model contains both between- and where the uj and ej are independently, identically within-subject elements it still remains a fixed distributed as uj ~ N(0,[2) and e i ~ N(0, o21), re- part. To include random variation of the individ- spectively. The-variance of the r-andom part, Var ual growth parameters in model (2.7), we have to (TRuj + ej), is then written as TRg2Tn' + o21. In give the random part of it a more complex struc- the literature, models employing this specific ture. This can be accomplished by specifying within-subject covariance structure generally go under the name of 'random coefficient models' vj - Zbj + q, (2.8) (De I.e~uw and Kreft, 1986; Prosser et al., 1990). THE STATISTICAL SOF'IWARE NEWSLETTER 589

In the sequel we will adopt the notation described Program ML3 above. Hence, fixed effects will be indicated with T' s and the variances of the random parameters A. Design philosophy with oYs, the elements of f~. The variance of the The ML3 program is designed for multilevel elements of ej (which is constant in our case) will analysis with two or three levels. One is able to be denoted by 0 -2. fit mixed linear models to data with a two or Concluding, one could say that when the error three-level hierarchical structure, in which the covariance structure of a MANOVA model is coefficients of the explanatory variables may be specified in an appropriate way, the model ap- random. The program is written by Presser, Ras- proaches or becomes identical to a random coef- bash and Goldstein. It is described and docu- ficient model. In that way, we could also con- mented in Presser, Rasbash and Goldstein elude that MANOVA models can be derived as (1990), the ML3 user's guide. ML3 is combined special cases of a multilevel model. Apparently, with the more general statistical package the concept of hierarchically structured data NANOSTAT written by Healy (Healy, 1989), could provide a common framework for studying which makes a variety of utility functions avail- the interrelations among both approaches. able to the program. The theory on which ML3 is based is described in Goldstein (1987) and IH. Description of the programs Goidstein and Rasbash (1989). Models that can In this paper we review two different approaches be estimated using the program include random for the analysis of growth data using longitudinal coefficient regression models, growth curve mixed linear models. In the previous section we models and hierarchical legit models. Fitting have examined the underlying statistical models growth curve models with ML3 is basically a and their interralations. In this section we review two-level analysis of longitudinal data. It means and compare three computer programs with that the repeated measurements 0evei 1) are which these models can be applied for data viewed as being nested within individuals (level analysis. The programs are ML3, HLM and 2). If necessary, the individuals may be grouped BMDP5V. They are described on the basis of the on level 3. The handling of longitudinal data in a following topics: multilevel framework is discussed in Goldstein • Design philosophy. (1986a, 1987, 1989, 1995) and in Bryk and • Implementation details. Raudenbush (1987, 1992). • Implemented model. B. Implementation details • Routines. • Data setup and data handling. ML3 is available as an executable program only. • Output and results. That is, it is shipped without the program source. • User friendliness. It runs on 80X86- to Pentium-based personal • Special features. computers under DOS/OS2. Proper operation of the program requires 640 Kb of RAM and a hard In this paper we are directing our attention to disk. A CGA, EGA or Hercules graphics card is longitudinal applications, i.e. to the analysis of growth data. In the descriptions of the programs, required if one wishes to use the program's high- we have tried to balance the emphasis given to resolution graphics capabilities. ML3 is available aspects relevant for those applications, and to either as a math coprocessor version or as a non- more general features. A certain quantity of gen- coprocessor version.. It is shipped on a single eral remarks is very valuable and unavoidable diskette in compressed form. Installation is sim- too. Nonetheless, some aspects are treated briefly. ple as the files are easily expanded and tranfered For a very detailed, extensive discussion of ML3 to a subdirectory on the hard disk. According to and HLM we refer to Kreft, De Leeuw and Van the ML3 User's guide, insufficient memory is the der Leeden (1994). In their paper, the application most common problem that new users experi- of BMDP5V for longitudinal mixed models is ence. For this paper we ran ML3 Version 2.1 on discussed too. a 80286 PC with math coprocessor. 590 THE STATISTICAL SOFTWARE NEWSLETTER

There are two versions of ML3 available for According to the ML3 User's guide, RIGLS and analyses involving very large data sets. ML3/E is IGLS estimates may differ considerably for snail especially compiled for personal computers with data sets. IGLS appears to be the default method expanded RAM (386-based and higher). It lets of estimation. However, neither the reason why, you work with all the extended memory avail- nor further information concerning these estima- able. ML3/V is a special version of ML3 for tion procedures, are given in the manual. VAX computers running under VMS. The number of iterations may range from 1 to 999, but the default number is set to 5. According C. Implemented model to the manual this number is sufficient to reach convergence if several criteria are met, such as a In ML3 the basic model implemented is the number of observations per unit large enough for mixed linear model for data with a two or three- stable estimation, a small number of parameters level nested structure. At any level of the hierar- to be estimated, and so on (further details are chy, the coefficient of any explanatory variable given in the ML3 manual). may be defined random or fixed. The random coefficients at each level may have any E. Data setup and data handling (co)variance pattem. Concerning models for ML3 stores data in a memory segment called the longitudinal data, ML3 can fit polynomial worksheet. This worksheet is divided into 100 growth curve models, as well as growth models columns which can be refered to as C 1, C2, etc. fitting functions with free coefficients. In the When a data file is read in, these columns will manual, the latter models go under the name of hold the variables. On the worksheet there are fixed occasion models. 100 boxes too, refered to as K1, K2, etc., which are capable of holding single numbers, for in- D. Routines stance, the result of some computation. A third section of the worksheet is meant for the specifi- In ML3 the model parameters are estimated us- cation of the model. For convience, the data in a ing an iterative generalized least squares (IGLS) worksheet can be stored in a file for use in future algorithm. This procedure yields consistent esti- work sessions. mates of model parameters and maximum likeli- The input file needs to be a standard, numeric hood estimates if the assumption of multivariate ASCII file. File reading may be unformatted or normality is valid. The program can also com- formatted, using a FORTRAN format statement. pute unbiased or restricted (RIGLS) estimates. Small quantities of data, for instance starting Under the assumption of multivariate normality, values, may be entered by keyboard. these estimates are called restricted maximum likelihood (REML) estimates. IGLS is described Records in the input file are considered individ- by Goldstein (1986b, 1987, 1995). RIGLS is de- ual cases as usual. In regular multilevel analysis, individuals are nested within groups. Hence, scribed by Goldstein and Rasbash (1989). The ML3 expects each case to be labeled with a group RIGLS procedure is comparable to REML in the identifier (ID2 -> level 2) and a case identifier other multilevel program discussed in this paper, (ID1 -> level 1). However, when analyzing re- HLM by Bryk, Raudenbush and Congdon. The peated measures data within a two-level context, RIGLS procedure adds bias correction terms to the individuals are considered as level 2 units, the IGLS estimates on each iteration. This im- and the measurements-within-individuals are the plies that during the estimation of the random level 1 units. Hence, the usual structure of re- parameters, the fixed coefficients are treated as peated measures data needs to be transformed ac- quantities which have uncertainty "built in". In cordingly: each occasion should be given a sepa- IGLS, the fixed coefficients are treated as known rate record. For the Potthoff-Roy data (Potthoff when estimating the random parameters. Starting and Roy, 1964; see also section 4), in the case of values for both IGLS and RIGLS procedures are a quadratic model where growth is modeled as a provided by an ordinary least squares (OLS) re- function of age (and age2), with one explanatory gression using all cases and ignoring grouping. grouping variable (sex), this may look like: THE STATISTICAL SOFTWARE NEWSLETTER 591

1 01 21 8 64 1 The NANOSTAT package provides the user with 2 01 20 10 100 I several graphic commands. Plots such as stem- 3 01 21.5 12 144 1 and-leaf plots, boxplots ansd scatterplots may be 4 01 23 14 196 1 obtained. 1 02 21 8 64 1 2 02 21.5 10 100 1 For a more elaborate discussion of the options mentioned above, as well as for further features, we refer to Kreft, De Leeuw and Van der Leeden 3 27 23.5 12 144 -1 (1994). 4 27 25 14 196 -1 G. User friendliness occasion person response age age2 sex The program is extensively documented, the hr. ID var. (expi. variables) manual is clearly written and complete. An entire chapter with examples is included in which the Fortunately, ML3 contains a utility for convert- log files of six example runs (with different sub- ing a "one record per individual file" into a "one models) are given and discussed. The manual record per occasion" file. provides a nice introduction to multilevel modeling theory too, with many references. This intro- On the whole, data handling is very flexible in duction may be of great help to a novice in this ML3. A variety of data operations can be per- kind of data analysis. formed within the NANOSTAT package, such as computational and statistical transformations, The interactive capabilities of ML3 are very coding and recoding, generation of dummy vari- powerful. Users have much freedom in making ables, selection and sorting of cases, and so on. adjustments during the run. The specification of Auxiliary files can be used. Even vectors and a model is very flexible. One proceeds by enter- matrices can be input. ing commands that tell the program the roles of the input, or created, variables and which pa- Missing data can be handled if they are indicated rameters are to be estimated. A screen displaying with a numerical code chosen by the user. Sev- all current specifications and choices may be eral options are available including the imputa- called up at any time entering the command tion of estimated values to replace missing data SETI'. An online help program is built in too, and listwise deletion. which shows the commands and their syntax.

F. Output and results The ML3 program offers many possibilities to advanced users of multilevel analysis. There are The default output is none, program convergence many options, not in the least because of the ex- is the only information reported. All special out- tensive NANOSTAT package which is merged put can be acquired by commands. "Rand" and with the program. The program is probably not "fixe", for instance, generate the estimates of the an easy one to start out with for an unexperi- random coefficients and the fixed coefficients, re- enced user. spectively, as well as their accompanying standard errors. Although not documented in the H. Special features manual, the command "like" produces the - 2 log likelihood value (also called "deviance") of the There are a number of special features. We men- estimated model. Commands enable the user to tion the possibility to enter starting values for pa- compute predicted values and residuals. This is rameters other than default OLS estimates. Fur- of particular interest in the analysis of growth ther, estimation of certain, slow converging, pa- curve data if one is interested in the individual rameters can be 'frozen' during the estimation growth trajectories. proces, to speed up convergence. Special is the ability to fit multilevel logit and log-linear mod- All output may be printed on the screen or writ- els. This allows the analysis of data with propor- ten to an output file specified by the user. tions or binary variables as dependent variables. 592 THE STATISTICAL SOFTWARE NEWSLETTER

Program HLM For this paper we used HLM version 2.1 running on a 80286 PC with math coprocessor. A. Design philosophy The abbreviation HLM refers to the Hierarchical C. Implemented model Linear Model. The program FILM, implementing The basic model implemented in HLM is the this model is written by Bryk, Raudenbusch and two-level hierarchical model. By default, the Congdon (1988) and is designed for the analysis program assumes that the coefficients of the of multi-level data with a two level hierarchical within-unit model are to be treated as random, structure. The program is described and docu- but this can be changed. In the interactive proc- mented in Bryk, Raudenbush, Seltzer and Cong- ess during the run, the user may apply additional don (1988), the HLM user's guide. According to constraints to the coefficients. Either the variance this manual the program is aimed at two broad or the mean of one or more of the within-unit classes of applications: the familiar analysis of coefficients can be set to zero. The result is that nested data, as in the study of school effects, for we can have models in which a within-unit vari- instance, and the analysis of growth curve data. able has a fixed effect or a random effect only, or In both cases we have a within-unit and between- both. unit representation: within and between class Also by default, both the within and between analysis and within-subject (time-series observa- model have a base term (intercept). This can be tions) and between-subject analysis, respectively. overruled too. For instance in growth curve Recently, a version of HLM has been released analysis it may be of interest to fit models with- which can deal with three-level data as well. out a within-unit intercept. B. Implementation details Concerning models for growth data, HLM has roughly the same possibilities as ML3. A third HLM is available in two versions, one for work- level in the data can be handled by the recent stations or mainframes and one for the PC op- three-level version of HLM. erating under MS-DOS. The PC version of the program is an executable only; there is no source D. Routines code available. The other version comes with the FILM implements an EM algorithm for its pa- source code too (the source of HI_hi used to be in rameter estimation (Dempster, Laird and Rubin, the public domain, but this appears to have come 1977). The algorithm is supplemented by an to an end with version 2.1). HLM runs on Aitkin accelerator (Laird, Lange and Strom, 80X86- (with or without a math coprocessor) to 1987). According to the manual, using this ac- Pentium-based personal computers under DOS/ celerator a 30 to 40 percent reduction in compu- OS2. The computational kernel of the program is tation time can be achieved. The number of it- written in FORTRAN 77. Some additional rou- erations is optional. The manual suggests that 10 tines have been written in C. For the PC version iterations are often sufficient for explanatory we used, program limitations (see the manual, p. analysis, but recognizes the fact that generally 39) are: a maximum of 10 within-unit variables this number will not be high enough to reach (either fixed or random) in a model, a maximum convergence in practice. In fact, the EM algo- of 15 between-unit variables and a maximum of rithm is well known for its notoriously slow con- 35 fixed effects in all equations. HLM uses the vergence, especially when variance estimates are input data to create a file of "sufficient statistics". close to zero (close to a boundary of the parame- The current maximums for this sufficient staffs- ter space). In those cases, a number of over 100 tics file are 25 within-unit variables, 25 between- iterations is easily reached. unit variables, and a total of 300 units. For the non-PC version, these maximums are slightly A second algorithm provided by I-ILM is the so higher, although they can all be changed if the called "V-known" routine. This program is in parameter statements in the source code are fact a very general, multivariate regression rou- modified. tine for cases in which the sampling variances THE STATISTICAL SOFTWARE NEWSLETTER 593 and covariances are (assumed) known for each sample variances and covarianees (see the man- unit. The V-known routine is mainly used in ual, p. 57 for further details). meta-analysis. Further details concerning this The program can handle missing data in the topic are given by Raudenbush and Bryk (1985). within-unit file only. It is assumed that the be- E. Data setup and data handling tween-unit file contains no missing data. If so, the user has to take action such as delete the case If a user wants to analyze a particular data set for or impute a value. Handling of missing data can the first time, FILM needs two input files: a be listwise or pairwise deletion. within-unit and a between-unit file. From these files, the program computes a file of sufficient F. Output and results statistics, which can be used in all subsequent By default HLM produces a complete and com- analyses of the same data. For growth data the prehensive output file, for which name the user is within-unit file has the familar "one record per prompted. Standard errors for the fixed parame- occasion" structure (see the section on ML3). ter estimates are given as well as a table with t- Subjects are identified by an ID number in the statistics and p-values. The variance estimates first field of each record, which must be read in (the random parameters) are tested in a chi- text format. This also holds for the between-unit square table with which one can decide if a suf- file which uses these ID numbers to link the be- ficient part of variance is explained by variables tween-unit information for a unit to its within- on the second level. Reliability estimates of these unit data. ID numbers in both files must therefore parameters are provided too. Model fit can be as- be completely consistent. For the Potthoff-Roy sessed with a deviance statistic. Options control data (see section 4), in the case of a linear model the printing of OLS estimates for each unit. Ei- where growth is modeled as a function of age, ther for ten or for all units the within-unit re- these files may look like: gression parameters are printed. For growth curve applications this is of interest: it provides within-unit file between-unit file the user with the individual growth trajectories, wheras the average growth trajectory for the 01 21 8 01 1 whole sample is given too. The output concern- 01 20 10 02 1 ing the history of iterations is controlled by the 01 21.5 12 03 1 user too. The value of the likelihood function is 01 23 14 ...... given either for all iterations or only for the first 02 21 8 16 1 and the last one. 02 21.5 10 17 -1 For an extensive discussion of the options men- ...... 18 -1 tioned above, as well as for further features, we

... refer to Kreft, De Leeuw and Van der Leeden 27 23.5 12 27 -1 (1994). 27 25 14 Upon request HLM produces a so called residual file, providing a means of checking the fit and person response age person sex distributional assumptions of the between-unit ID variable ID model. This file can be read in into SAS, SPSS or SYSTAT if the user wants to compute various A HLM utility, PRINTSSM, can be used to print diagnostics. and thereafter inspect the sufficient statistics file when it is first created. G. User friendliness HLM will also accept a SYSTAT system file as The program is extremely interactive: it starts by input (SYSTAT is a statistical package for use on asking the user the first of a long series of ques- personal computers). Using the V-known routine tions, which guides the user step-by-step through requires a different input file structure. In that the model specification process. This makes the case the input includes estimates of within-unit program fairly easy to use. 594 THE STATISTICAL SOFTWARE NEWSLETTER

The manual is definitely of great help to an un- be estimated, the program becomes expensive to experienced user. It is build up along the lines of use. the annotated output of a number of example runs. A serious disadvantage of this approach is B. Implementation details that relevant information is scattered throughout BMDP5V is provided at any site where the the manual. Specific information is mixed with BMDP package is running. Usually this package example output, instead of organized under has been implemented on mainframes and mini specific headings. It is therefore not always easy computers, so there are versions working under to find. different operating systems such as MVS, VM, VAX/VMS and so on. More recently, PC ver- H. Special features sions of the package have become available for Compared to the other two programs, HLM is the use under MS-DOS or OS/2. only one that provides a variety of tests. We The BMDP procedures can be operated either in mention: a t-test for significance of the fixed pa- batch mode, or in interactive mode, depending on rameters, a chi-square test for residual unex- the user's preferences and/or the available com- plained variance in the first level parameters, a puter facilities. reliability estimate of the first level variables, a multivariate hypothesis test involving the fixed BMDP5V shows no limitations concerning the effects, a homogeneity of variance test for the re- number of individuals and variables. However, sidual variation in each unit and a likelihood- the memory needed for data storage will increase ratio test for the variance-covariance components with these numbers, and will also depend on the model. specified model. Probable implementation re- strictions could therefore arise dependent on the Although the program is meant to be totally in- computer system used for running BMDP. teractive, it can also be executed in a batch mode using an extended default file (or "personal" de- C. Implemented model fault file). This can be of great use to bypass ir- relevant prompts and other cumbersome interac- The models that can be fitted in BMDP5V belong tive features as one becomes a more experienced to a general class of multivariate linear models. user. In these models a set of regression parameters describes the structure of the expected values of the observations, and a set of covariance parame- Program BMDP5V ters provide for a general parametrization of the within-subject covariances. A number of special A. Design philosophy structures for these covariances are built-in. The BMDP5V program is designed by Jennrich User-defined covariance structures can be speci- and Schluchter and described in Schluchter fied too. (1988). The theory on which the program is When repeated measures data are analyzed using based is given by Jennrich and Schluchter (1986). a generalized multivariate linear model such as Being a procedure of the BMDP software pack- in BMDP5V, the individuals are considered as age, the program is extensively documented and data records and the observations on multiple oc- illustrated in the BMDP user's guide (Dixon, casions are interpreted as separate dependent 1988). variables. In a multilevel context, this makes in- BMDP5V is especially meant for the analysis of dividuals the second level within which the (first repeated measures with special emphasis on un- level) observations are nested. balanced situations, including unbalance caused D. Routines by missing data. Although the program has many possible applications, it is developed with small BMDP5V uses three different algorithms to com- experimental data sets in mind, especially in a pute maximum likelihood estimates for all model biological context. When there are many obser- parameters: a Newton-Raphson, a Fisher scoring vations and/or a large number of parameters to and a generalized EM algorithm. Producing THE STATISTICAL SOFTWARE NEWSLETTER 595 identical results, these methods differ with re- LEVEL = 4. spect to the number of iterations required and the DPNAME = DISTANCE. DPVAR = DISTI, DIST2, DIST3, costs per iteration. Generally, EM will converge DIST4. slowly, but with low costs per iteration, whereas CONTRAST (TIME) ~ AGE. Newton-Raphson will require the smallest num- AGE ~ 8, i0, 12, 14. ber of iterations, but has the highest cost. BMDP /MODEL DISTANCE = 'SEX + AGE + SEX. AGE'. advises to use Newton-Raphson if the number of /STUCTURE TYPE=UNSTRUC. covariance parameters is 15 or less, otherwise EM is preferable. BMDP5V offers two different In this example (see section 4), data are defined algorithms to compute restricted maximum like- and modeled consisting of measurements of some lihood estimates for the covariance parameters: dental distance for a group of children, obtained generalized EM and a quasi-scoring algorithm. at ages 8, 10, 12 and 14 (DISTI, DIST2, DIST3 Results will be identical but convergence is faster and DIST4). As a characteristic of interest, re- for the scoring method with higher cost and vice maining fixed over occassions, we have the versa for the restricted ML method. For any of grouping variable sex (SEX). In the example these procedures, the maximum number of itera- setup, the within-subject factor is defined to be tions is set to 15 by default. If neccessary, this TIME. The CONTRAST statement creates a number can be altered by the user. variable, AGE, taking the values 8, 10, 12 and 14 respectively, for each of the values of TIME. The E. Data setup and data handling model specified here fits linear growth curves to The data input file must be organized so that in- the data. The curves are found as a regression of dividuals are the records and the repeated meas- distance on age. The coefficients of the individ- urements are the variables. Note that this is dif- ual curves vary randomly across the individuals. ferent with respect to the way in which hierarchi- A mean growth curve for the whole group, as cal data are usually organized. well as mean growth curves for boys and girls We have run the program in batch mode. The separately, are easily produced by combining the BMDP instructions are given by keyword- fixed coefficients of the model in the correct way. statements, which look much alike usual SPSS The expression SEX.AGE adds an interaction statements. Input data and BMDP instructions term to the model. We will return to this data in can be stored in the same file, or we may use section 4 where the results of the different pro- separate files for both. Keywords control the grams are compared. names of the variables, their number, format, In the STRUCTURE statement one of the built-in tranformations, and so on. The actual modeling structures for the within-subject covariances can of the data is controlled by three keywords- be specified. In the example setup above, these statements: covariances are contained in an arbitrary matrix DESIGN, MODEL and STRUCTURE. (fully parametrized). Other built-in special The DESIGN statement specifies the structure of structures include first order autoregressive the data. It identifies variables classifying sub- models, compound symmetry and banded or gen- jects and variables measured repeatedly. The eral autoregressive structures. For some struc- models itself are specified in the MODEL state- tures additional input is needed. A factor analytic ment, using variable names. The STRUCTURE structure requires the number of factors to be statement specifies the structure of the within- specified. Random effects requires the input of a subject covariances. An example of a set of (known) matrix containing the variables that are BMDP instructions for the Potthoff-Roy data, is to be used for the specification of random coeffi- given below. cients. For the Potthoff-Roy data the additional input in that case may look like:

/PROBLEM TITLE='POTTHOFF-ROY DATA'. /INPUT FILE = DENTAL.DAT. /STUCTURE TYPE=RANDOM. /DESIGN GROUPING = SEX. ZCOLI z i, I, I, i. REPEATED = TIME. ZCOL2 = 8, i0, 12, 14. 596 THE STATISTICAL SOFTWARE NEWSLETTER

A general linear structure requires the number of subject covariances can be very useful. However, parameters to be specified as well as a set of to include a user-defined covariance structure by known matrices. Finally a user-defined covari- adding a FORTRAN 77 subroutine to the pro- ance structure which is not built-in and not lin- gram may be quite difficult for regular users. ear, can be specified by adding a FORTRAN 77 subroutine to the program input. This routine be- H. Special features comes a part of the program, called in each it- A unique option of BMDP5V is the ability to eration. analyze incomplete data. Using an EM algorithm, values for missing data can be imputed. In F. Output and results that case it is assumed that there is an underlying The output of BMDP5V is controlled by the set of complete data from which the observations PRINT statement and can be very comprehen- of each subject form a subset. This property is sive. There are no special output files. Program very useful because it is been known to improve instructions, model specifications and (default) the reliability of the data analysis (cf. Little and values of various program parameters are exten- Rubin, 1987). An option like this is certainly not sively listed. An extended output presents, for very common in statistical software. each iteration, a table with the log likelihood of Another special feature is that BMDP5V can deal the specified model, the values of the regression with time-varying covariates. For instance, in the parameters and the values of the covariance pa- case that we have multivariate repeated observa- rameters. Akaike's information criterion is pro- tions, that is two or more dependent variables vided to check the appropriateness of a chosen measured repeatedly, we could take one (or covariance structure. Asymptotic standard errors more) of these variables as a covariate. This fea- and z-values are given for all maximum likeli- ture is built-in and easy to use. hood estimates of the model parameters. For the terms of the regression model Wald (chi-square) tests of significance are included. Within-subject IV. Comparison of results and all-pairs within-subject covariance and corre- In this section we present comparisons of the lation matrices are given. A very useful option, outcomes of growth curve analyses with ML3, especially when analyzing individual growth HLM and BMDP5V. In doing this, two different trajectories, is a listing of individual responses (at datasets will be used: a small data set called the different timepoints), their predicted values, DENTAL data, and a large data set called residuals and standardized residuals. With this SCHOOL data (for a description see below). listing, Mahalanobis distances are provided. These distances can be helpful in detecting cases First, a description of the data will be given, fol- which may be outliers. lowed by the models which we have analysed. Each model will be accompanied by a table with G. User friendliness results comparing the three programs. For ML3 we have included results from two estimation The BMDP user's guide is written very clearly. methods available within this program: full in- BMDP5V, as any BMDP procedure, is exten- formation and restricted maximum likelihood, sively discussed and commented. The different indicated by ML3F and ML3R respectively. As options are very well illustrated presenting a va- mentioned before we have concentrated upon riety of real data examples. Most technical details models in which the individual growth parame- can be found in the appendix. ters vary randomly across individuals and in The program is easy to use. As far as the key- which this variation is (partially) explained by word command structure, and the ability to run variables on the group level. BMDP in interactive mode is considered user- DENTAL data: The DENTAL data were al- friendly, the program is too. ready briefly mentioned in Section 3, where data When repeated measures are to be analyzed, the setup and handling in the three programs were built-in autoregressive structures for the within- discussed. The data are taken from Potthoff and THE STATISTICAL SOFTWARE NEWSLETTER 597

Roy (1964). They were collected at the University tween several subpopulations. The dimensions of North Carolina Dental School and concern the along which these subpopulations are divided, measurements of the distance (DIST) from the are considered the independent variables. In this centre of the pituitary to the pteryomaxillary fis- example the independent variables are (SEX) and sure for eleven gifts and sixteen boys, at ages of (AGE). In other words, one wants to study the 8, 10, 12 and 14. joint effect of the independent variables on one Both approaches for the analysis of growth data (here DIST) or more dependent variables. In the using a random coefficient model have been MANOVA tradition these are all fixed effects. applied to these data: a multilevel analysis and a Random variation of the individual growth pa- generalized MANOVA. Conceptually, the ran- rameters can be achieved by a modeling of the dom coefficient model as a general case of the error structure, as was explained in Section 2. multilevel model is equivalent to the random The within-subject covariance matrix, Ej, is coefficient model as a special case of the general- given the 'random coefficient' structure X~X' + ized MANOVA-model of Jennrich and Schlu- o2I. In that way we can derive a single equation chter (Jennrich and Schluchter, 1986). Conse- model which can be interpreted in the same way quently, we will compare the results of the two as that of the multilevel model described above. multilevel programs, ML3 and HLM, with the For the DENTAL data model, Xj is thus build up results of the MANOVA program of Jennrich en from four covariance terms: the variances of the Schluchter, BMDP5V. intercept and the linear component, their covari- The analysis consists of fitting linear growth ance and the variance of the (remaining) within- curves found as a regression of distance in age. subject error. The former three are the same ele- In a multilevel framework the repeated meas- ments that are contained in the second level vari- urements are considered nested within the sub- anoe-covariance matrix (fl) of the multilevel jects. On the first level the individual growth model. The latter term is equal to the only ele- curve coefficients are treated as random vari- ment of which the first level varianoe-covariance ables. On the second level (SEX), defined as a matrix consist in the multilevel model (02). dummy variable with values -1 for gifts and 1 for boys, is used as an explanatory variable of ran- Results of the analysis are presented in Table 1. dom variation across the indiviual growth trajec- Refering to the model described above, we find tories. the following entries in Table I. First we have The within-subject model can be written as the fixed coefficients, T00 and Tl0, which denote the fixed intercept and slope of the mean growth (DIST)ij = 130j + [3 lj (AGE)j + eij, curve for the whole group. Deviations from this mean intercept and slope for the gifts' group and and the between-subject model is for the boys' group, are given by T01 and Tll respectively. Hence, the fixed intercept and slope of [~0j ""Yoo + ~'Ol (SEX)j + u0j, the mean curve for the gifts are given by (T00 - ~lj "'~/10 + Tll (SEX)j + Ulj. T01) and (TI0 - Yl l)" Equivalently, the fixed intercept and slope of the mean curve for the boys are The single equation version of the model is written as given by (Too + To l) and (TlO + Tll). (DIST)ij" ~'oo+ ~/Ol (SEX)j + ~'lO (AGE)j + The random coefficients to0o and tOll denote the variances of the random (within-subject) inter- TIl (AGE)j(SEX)j + uij(AGE)j + U0j+ eij. cept and the random (within-subject) slope, i.e. Jennrich and Schluchter define the random coef- the variance not accounted for by (AGE) and ficient model in a slightly different way. Accord- (SEX). to01 gives the covariance between both ing to the MANOVA tradition one typically random coefficients. The variance of the first wants to research whether one or more dependent level disturbances is contained in o 2. Note that variables (or the mean score on it) differs be- regardless of the fact that the measurements are 598 THE STATISTICAL SOFTWARE NEWSLETTER

Table I: Results of the analyses of the DENTAL data, N = 27.

BMDP5V HLM ML3F ML3R

~/00 16.8567 16.8993 16.8570 16.8570 TI0 0.6320 0.6287 0.6320 0.6320 ~/01 -0.5161 -0.3824 -0.5161 -0.5161 '~11 0.1524 0.1419 0.1524 0.1524

02 1.7162 1.7651 1.7162 1.7162 o~00 4.5569 5.7509 4.5569 5.7861 6010 -0.1983 -0.3082 -0.1983 -0.2896 0)11 0.0238 0.0356 0.0238 0.0325

# iterations 2 60 2 4

deviance 427.80 43 i .73 427.81 427.96 time related here, this random coefficient model "One-Minute Test". This test measures the speed does not include a special covariance structure at which a pupil can decode words. Scores of 436 for the error components. pupils are used which are measured after respectively 1, 8, 11, 18 and 22 months, on 25 Dutch Comparing solutions in Table 1, we see that the Primary Schools (for further details we refer to three programs produce similar, though not Guldenmund, 1991). Two explanatory variables identical results. Estimates for fixed and random will be used on the second level, the variable coefficients are the same for BMDP5V and (SEX) and a score on a test measuring fonemic ML3F. However, results from HLM differ from analysis ability, (FA). The first model to be ana- these to a certain extent. This holds for the lysed is a random coefficient model in which estimates of the random parameters in the ML3R (TR) is regressed on (TIME) as a quadratic func- solution too. The major part of the differences in tion. (SEX) is used as an explanatory variable on the solutions can be explained by the use of dif- the second level. This model looks similar to the ferent estimation methods: HLM and ML3R use model which we used to analyse the DENTAL restricted maximum likelihood, whereas data. However, now we have a much larger BMDP5V and ML3F use full information maxi- sample and a quadratic instead of a linear func- mum likelihood. Differences between both likeli- tion is fitted. hood procedures are strengthened in this case be- The within-subject model can be written as cause we use a small data set. Results from analyses on larger data sets reported in the sequel (TR)ij - 130j + 131j (TIME)j + as well as results from Kreft, De l_geuw and Van der Leeden (1994) support this finding. 132j (TIME)j 2 +.eij, If we compare the number of required iterations, and the between-subject model is Table 1 clearly demonstrates the (well known) slow convergence of the EM algorithm employed 13oj = ~'0o + ~'01 (SEX)j + u0j, by HLM, when parameter estimates come close ~lj " TI0 + Tll (SEX)j + Ulj. tO zero. [~2j " T20 + T21 (SEX)j + u2j. SCHOOL data: For the next illustration we use part of the results stemming from an educational A single equation version of this model is given study. De data are derived from a longtitudinal by study (Voeten, 1991) concerning the individual (TR)ij "~/00 + Tol(SEX)j + ]tI0(TIME)j + development of technical reading (TR). Techni- cal reading is measured as a score on a so-called 720(TIME)j2 + T1 I(SEX)j(TIME)j + THE STATISTICAL SOFTWARE NEWSLETTER 599

Table 2: Results of the analyses of the SCHOOL data, N - 300, explanatory variable is (SEX), measurement times are centered, except for BMDP5V.

BMDP5V HLM ML3F ML3R

700 45.9339 60.2410 60.2800 60.2800 Y10 0.7522 0.8009 0.7522 0.7522 Y20 0.0288 0.0258 0.0288 0.0288 Y01 1.6653 1.4756 1.4580 1.4580 T11 0.0187 0.0291 0.0187 0.0187 Y21 -0.0023 -0.0025 -0.0023 -0.0023

02 37.2859 36.9274 31.6500 31.7300 co00 158.9831 179.5018 182.0000 183.2000 COl0 7.6650 2.4385 5.4620 5.4710 °~11 -1.0276 0.0520 0.0000 0.0000 o)20 -0.3977 -0.1551 -0.2824 -0.2831 co21 0.0426 -0.0019 0.0016 0.0016 ¢o22 -0.0017 0.0002 0.0000 0.0000

# iterations 3 158 6 6

deviance 10511.88 10574.60 10549.30 10549.30

721 (SEX)j(TIME)j 2 + u0j + Ulj(TIME)j + viduals are treated as level 2 units (see also Sec- tion 3). Since we did not have immediate acces to u2j(TIME)j 2 + eij. the HLM Fortran source, we were not capable of The analysis of these data were troubled by some extending this maximum. program limitations. First, HLM appeared to run The results of our analysis are shown in Table 2. only if the independent variable (TIME) was centered, otherwise singularity problems occured. The entries in Table 2 are interpreted in a similar In order to make an as adequate comparison of way as in Table 1. Again, the two multilevel pro- the results we had to center the data using ML3 grams produce similar though not identical re- and BMDP5V too. In ML3 this requires some suits. Comparing ML3F and ML3R, the esti- data manipulation, in BMDP5V however, this mates of the random parameters are relatively appeared to be impossible. It is caused by the fact more alike than those in the first example analy- that the program does not allow the input of sis. This is explained by the fact that we use a noninteger values in its CONTRAST statement much larger data set here. (see Section 3), necessary for the specification of (1) The more general differences between HLM and centered variables . ML3 can be attributed to their different estima- Secondly, we were forced to analyse the data us- tion procedures. The slow convergence of the EM ing only 300 cases. This number is the default algorithm employed by HLM is shown dramati- maximum number of cases in HLM when indi- cally here.

(1) Personal communication with BMDP has indicated that there is indeed a prohibition against the input of other than integer parameters in the CONTRAST statement. However, BMDP suggested a so-called work-around approach based on the use of covariates to make use of centered variables. Moreover, it was stated that in the PC-90 version of the BMDP statistical software one has the possibility of introducing non-integer elements for the matrix Z (see ZCOL1 and ZCOL2 in the example above), and for the independent variables. 600 THE STATISTICAL SOFTWARE NEWSLEIT'ER

The parameter estimates of the noncentered so- Obviously, this model is similar to the model lution of BMDP5V are not quite comparable to using (SEX) as a second level explanatory vari- the estimates of the other solutions. As might be able, only now we use (FA) for this purpose. expected, the largest difference in parameter However, besides the aforementioned program value can be observed regarding the intercept limitations we now stumble upon what can be term Too. Centering of the measurement times, seen as a fundamental difference between the i.e. the within-unit independent variable, can be analysis of variance and regression analysis. This computationally convenient, since it causes the difference lies in the assumption of the measure- sampling covariance of the ordinary least squares ment level which is allowed for the independent intercept and slope to be equal to zero. At the variable. While in the analysis of variance only same time it changes the way in which the independent variables of no more then nominal within-unit coefficients are interpreted. It is level are assumed, in regression analysis the in- therefore probably the most informative to look at dependent variables are assumed to be of at least the deviance only. Comparing values of this sta- interval level. Since (FA) consists of a scale with tistic, we observe considerable differences among the programs. According to the results in both scores ranging from 1 to 29, it can be considered Table 1 and 2, it seems as if HLM overestimates as being measured on an interval scale. Hence, the deviance. It is not quite clear to us if the dif- different results will occur if multilevel analysis ferences in deviance value among all programs in considers the variable (FA) as numerical, general, and between BMDP5V and ML3(F and whereas BMDP5V considers it a categorical R) in particular, are acceptable. variable. As a consequence, the only way this program can deal with such explanatory variable, Notice the negative variance estimates in the so- is to code the categories by means of a set of 28 lutions of HLM and BMDP5V. Apparently, these parameters are at the boundary of the parameter dummy variables. In other words, in BMDP5V space (equal to zero). Both programs do not pro- this problem is considered as one in which ex- vide a proper solution for this problem. ML3, perimental conditions or groups are involved: the however, does, by simply setting those estimates scores on (FA) are conceived as a groupcode with to zero. 29 possible effects. In the next analysis, instead of (SEX) we use This problem is highlighted comparing another explanatory variable, fonemic analysis BMDP5V with ML3 only. Results are given in (FA). Now, the within-subject model can be Table 3. written as In Table 3, the entries indicated by FAk, FAk* (ZR)ij - [~0j + [~lj (TIME)j + Lin and FAk* Qua, each consists of 28 separate ~2j (TIME)j2 + eij" values denoting interactions of (FA) with the intercept, linear and quadratic component, respec- and the between-subject model is tively. Hence, the amount of parameters to be estimated in BMDP5V is much larger. A compari- ~0j " ~00 + "~0[ (FA)j + u0j, son with the parameter values of ML3 is therefore problematic. ~lj = T10 + Tll (FA)j + Ulj. In Table 3, the entries indicated by FAk, FAk* ~2j = 720 + T21 (FA)j + u2j. Lin and FAk* Qua, each consists of 28 separate The single equation version of this model equals values denoting interactions of (FA) with the intercept, linear and quadratic component, respec- (TR)ij = ~00 + ~01(FA)j + ~I0(TIME)j + tively. Hence, the amount of parameters to be estimated in BMDPSV is much larger. A compari- ~20(TIME)j2 + ~/l 1(FA)j(TIME)j + son with the parameter values of ML3 is therefore problematic. T21 (FA)j(TIME)j2 + Uoj + Ulj(TIME)j + Concerning ML3F and ML3R results, Table 3 u2j(TIME)j2 + eij • shows that the differences between the random THE STATISTICAL SOFTWARE NEWSLETTER 601

Table 3: Results of the analyses of the SCHOOL data, N - 384, explanatory variable is (FA), measurement times not centered.

ML3F ML3R BMDP5V

Y00 38.0600 38.0600 44.8670 YI 0 0.9890 0.9890 0.7067 Y20 0.0247 0.0247 0.0320 ~01 0.4459 0.4459 FAk 71 1 -0.0147 -0.0147 FAk* Lin Y21 0.0003 0.0003 FAk* Qua

02` 32.1200 32.1800 38.1900 co00 155.1000 156.1000 130.0900 C°l 0 4.5250 4.5280 5.8400 oo! ! 0.0000 0.0000 -0.8200 oo20 -0.2742 -0.2749 -0.0300 0)2 ! 0.0015 0.0015 0.0300 OO22 0.0000 0.0000 -0.0010

# iterations 5 5 4

deviance 13547.10 13547.1 13327.84 parameters are even less substantial here, with N tures are fitted: banded, or general autoregres-

- 384, as in the previous analysis. sive, first-order autoregressive, completely unconstrained (unstructured) and the already famil- So far, we have been comparing random coeffi- iar random coefficient structure. Again the cient models to analyse data on growth, using school data have been analyzed. The fitted model ML3, HLM and BMDP5V. As mentioned earlier is identical to the first model described in this one can expect growth data to be serially depend- section, with (SEX) as a grouping variable and ent. If this is the case, one can consider the employing a quadratic growth curve. The scores modeling of a characteristic structure for the of 384 pupils were used. The results are given in within-subject covariance matrix. In multilevel Table 4. Only the fixed terms of the model and models for longitudinal data, usually the as- the deviance are listed. Estimates of the random sumption of uncorrelated error components with terms are omitted because there number varies equal variance is made. Although the manual de- across the different covariance structures. scribes the possibility to model complex level-1 covariance structures, it is not straightforward If we assume that the two models which apply to understand how this should be accomplished autoregressive covariance structures, as well as in practice for serial dependent structures (see the one with the random coefficient structure, are also Discussion). 'nested' within the model with unconstrained covariance matrix, likelihood-ratio tests can be Contrary to both multilevel programs, BMDP5V computed to decide which model is the most ap- explicitly provides several possibilities to model propriate. Differences between relevant deviance the within-subject covariance structure. A set of values are chi-square distributed with degrees of frequently encountered special structures are freedom equal to the difference between the build into the program, including autoregressive number of independent parameters in the corre- structures (see also Section 3). sponding models. Hence, from Table 4 it can be In the next analysis, an illustration is given of a concluded that the models applying autoregres- model in which several within-covariance struc- sive covariance structures are neither favorable 602 THE STATISTICAL SOFTWARE NEWSLETTER

Table 4: Results of the analyses of the SCHOOL data using BMDP5V, N - 384, explanatory variable is (SEX), different structures for f.

E structure ~ unconstrained Y~-TFt~TF'+O2I oij-~k,k=li-jl+l oij-o2®li-jl

TOO 49.5493 46.2603 44.7326 48.1164 T10 0.8338 0.6877 1.0413 0.5681 ~'20 0.0163 0.0314 0.0152 0.0304 ~/01 -1.0327 -0.9486 -1.0553 -0.9652 T11 -0.0308 -0.0455 -0.0196 -0.0349 T21 0.OO22 0.OO27 0.OO17 0.OO21

# parameters i 5 7 5 2

deviance 13270.098 13412.618 13431.496 13859.856 over the random coefficient model (~2 _ 447.2, flexibility specifying the coefficients either as df-- 5 for the first-order autoregressive model random or as fixed on any level. One has also and X2 - 18.9, df i 2 for the general autoregres- much freedom in choosing which explanatory sive model), nor over the unconstrained model variable explains what variation on which level (~2, 589.8, df - 13 for the first-order autore- of the model. In BMDP5V however, either all or gressive model and X2 - 161.4, df - 10 for the non of the (level 1) individual growth coefficients general autoregressive model). In fact, Table 4 are specified as random. Coefficients on the sec- shows that the random coefficient model is less ond, between-subject, level are always fixed. Ex- appropriate than the completely unconstrained planatory variables can therefore only operate on the second level to account for individual growth model (X2 - 142.5, df-- 8). curve coefficient variation. Further, in BMDP5V V. Conclusions there is no build in way to specify interaction between fixed variables at level 2 and random In this paper we have reviewed two approaches coefficients at level 1. Inflexibilities of this kind for the analysis of growth data by means of longi- in BMDP5V are caused by the fact that the pro- tudinal mixed linear models. The first is a type of gram is in fact a variance component model, de- multilevel model in which growth data are composing the total variance into a between part treated as hierarchically structured. The second and a within part. The between model uses fixed approach is a version of a MANOVA repeated coefficients always. The random coefficient measures model employing a structured (error) growth curve model emerges by modeling the covariance matrix. From our theoretical and within-subject error components with an appro- practical comparisons, it can be concluded that priate covariance structure, which brings up the both ways of specifying 'random coefficient limitations mentioned above. models' are fruitful for the analysis of growth data. Although they originate from different sta- The smooth way in which the within-subject er- tistical and data-analytic traditions, the growth ror covariances can be modeled in BMDP5V is a curve models they provide are basically alike. very useful feature analyzing time dependent Nonetheless, both approaches show a few short- data. Especially when the number of time points comings which are not really on a theoretical enlarges, the modeling of serial dependency be- level, but most strongly connected to the associ- comes increasingly valuable. In BMDP5V this is ated computer programs. Several of these limita- easily accomplished: several relevant covariance tions are listed below. structures are build into the program and ready In multilevel modeling, and within the accompa- foi" use. However, the user has to be aware of the nying multilevel programs, one has optimal fact that applying those structures not necessarily THE STATISTICAL SOF'IWARE NEWSLETTER 603 implies a random coefficient structure for the cases, the results of BMDP5V versus those of the individual growth parameters. Reversely, the multilevel programs are somewhat difficult to multilevel programs do not offer easily applicable compare, although the amount of variance ac- ways in which the within-subject (level I) covari- counted for by the model, should be identical for ances can be modeled. In theory, complex level 1 both approaches. covariance structures can be formulated: Gold- As far as parameter estimates are concerned, we stein, Healy and Rasbash (1994) give a detailed have observed that the three programs produce discussion of multilevel time series models. similar, though not identical results. Differences Centering of the within-unit explanatory vari- are mainly caused by the different estimation ables brings up another point of difference. Al- procedures employed. In HLM and BMDP5V though it changes the way in which the within- variance estimates may become slightly negative unit coefficients are interpreted, as was men- in case these parameters are at the boundary of tioned before, centering can be computationaily the parameter space (equal to or close to zero). convenient. Concerning the multilevel programs, ML3 simply sets these estimates to zero (but this in HLM centering is offered as an option, option can be turned off). The programs show a large difference between EM and the other im- whereas in ML3 it is realized by simple data plemented estimation procedures, in the number manipulation using NANOSTAT commands. of iterations required for convergence. It is However, HLM refused to run with uncentered clearly illustrated that when parameter estimates variables, because of singularity problems. ML3 come close to zero, the EM algorithm is slowly on the other hand, provided centered as well as converging. uncentered solutions. In contrast~ it seemed impossible to compute a centered solution in A minor point of concern is the following. When BMDP5V. For this purpose, the program re- growth data are analyzed in HLM, the number of quires additional input of non-integer values. cases is limited to 300. This restriction is caused Unfortunately, in the BMDP version currently by the fact that the number of groups is limited available to us, this option was not available. analyzing growth data. Groups refer to the sec- Meanwhile we now from personal communica- ond level, so in a usual hierarchically organized tion with BMDP that this feature is implemented data set, where individuals are considered level 1 in the PC-90 version of the BMDP software. As units, this is not a problem in particular. In prac- an alternative, a 'work-around' approach using tice, we will seldom want to make use of more covariates was suggested but we were not able to than 300 groups. However, when growth data are further elaborate on this topic. treated as hierarchical, measurements are the level 1 units and individuals become the second Another major point of difference is the handling level. Hence, we have a restriction on the number of numerical, explanatory variables in the be- of cases. It may be overcome by changing the tween-subject model. In HLM and ML3, numeri- defaults in the HLM source code and having it cal variables at this second level are treated as recompiled. As far as we know one has access to predictor variables measured at an interval level, this source code in the non-PC versions of HLM just as is the case in common regression analysis. only. BMDP5V, on the other hand, is developed as a general version of the analysis of variance model. References Hence, variables acting in the between-subject model are considered as categorical 'grouping' Bock, R.D. (Ed.) (1989): Multilevel analysis of edu- variables. The categories of these variables are cational data. San Diego, CA: Academic Press coded by a set of dummy variables, as if they are Bryk, A.S., and Raudenbush, S.W. (1987): Application indicating separate experimental conditions. It of hierarchical linear models to assessing change. Psy- means that for each dummy variable, or separate chological Bulletin 101,147-158 group code, an additional model parameter is Bryk, A.S., and Raudenbush, S.W. (1989): Towards a created. Hence, the variance that can be ex- more appropriate conceptualization of research on plained by the model is decomposed into single school effects: A three-level hierarchical linear model. degree of freedom terms. Therefore, in those In: R.D. Bock (Ed.): Multilevel Analysis of Educa- 604 THE STATISTICAL SOFTWARENEWSLETTER tional Data (pp. 159-204): San Diego (CA): Academic peated measures data. Statistics in Medicine 13, 1643- Press 1655 Bryk, A.S., & Raudenbush, S.W. (1992): Hierarchical Grizzle, J.E., and Allen, D.M. (1969): Analysis of linear models: Applications and data analysis meth- growth and dose response curves. Biometrika 25, 357- ods. Newbury Park, CA: Sage Publications, Inc. 382 Bryk, A.S., Raudenbush, S.W., Seltzer, M., and Cong- Guldenmund, F.W. (1991): Single and two step ap- don, R.T. (1988): An Introduction to HLM: Computer proaches to modeling individual psychological Program and User's Guide. University of Chicago growth. Unpublished MA thesis. Department of Psy- chometrics and Research Methodology, University of Demster, A.P., Laird, N.M., and Rubin, D.B. (1977): Leiden Maximum likelihood from incomplete data via the EM algorithm (with discussion): Journal of the Royal Sta- Healy, M. (1989): NANOSTAT users' guide. London: tistical Society, Series B 39, 1-38 AlphaBridge Ltd, De Leeuw, J., and Kreft, I.G.G. (1986): Random coef- Jennrich, R.I., and Schluchter, M.D. (1986): Unbal- ficient models for multilevel analysis. Journal of Edu- anced repeated-measures models with structured co- cational Statistics 11, 57-85 variance matrices. Biometrics 42,805-820 De Lury, D.B. (1950): Values and integrals of the or- Kendall, M.G., and Stuart, A. (1979): The advanced thogonal polynomials up to n = 26. Toronto: Univer- theory of statistics, Vol. 2, Inference and relationship, sity of Toronto Press 4th ed. London: Griffin Dixon, W.J. (Ed.) (1988): BMDP Statistical software Khatri, O.G. (1966): A note on a MANOVA model manual, Vol. 2. Berkeley, CA: University of California applied to problems in growth curve. Annals of the Press Institute of Statistical Mathematics 18, 75-86 Francis, B., Green, M., and Payne, C. (Ed.) (1993): Kirk, R.E. (1982): Experimental design: Procedures The GLIM system, Release 4 manual. Oxford: Claren- for the behavioral sciences, 2nd ed. Monterey, CA: don Press Brooks/Cole Geisser, S. (1970): A Bayesian analysis of growth Kreft, I.G.G., De Leeuw, J., and Van der Leeden, R. curves. Sankhya A, 32, 53-64 (1994): Review of five multilevel analysis programs: BMDP-5V, GENMOD, HLM, ML3, VARCL. The Geisser, S. (1980): Growth curve analysis. In: P.R. American Statistician 48, 324-335 Krishnaiah (Ed.). Handbook of statistics, Vol. 1 (pp. 89-115): Amsterdam: North Hollan. Laird, N.M., Lange, N., and Strom, D. (1987): Maxi- mum likelihood estimation with repeated measures: Goldstein, H. (1986a): Multilevel mixed linear model applications to the EM algorithm. JASA, 82, 97-105 analysis using iterative generalised least squares. Laird, N.M., and Ware, J.H. (1982): Random-effects Biometrika 73, 43-56 models for longitudinal data. Biometrics 38, 963-974 Goldstein, H. (1986b): Efficient statistical modelling Lee, J., and Geisser, S. (1972): Growth curve predic- of longitudinal data. Annals of Human Biology 13, tion. Sankhya A 34, 393-412 129-141 Lee, J., and Geisser, S. (1975): Applications of growth Goldstein, H. (1987): Multilevel models in educational curve prediction. Sankhya A 37, 239-256 and social research. London: Griffin Little, RJ.A., and Rubin, D.B. (1987): Statistical Goldstein, H. (1989): Models for multilevel response analysis with missing data. New York: John Wiley variables with an application to growth curves. In: D. Book (Ed.). Multilevel Analysis of Educational Data Longford, N.T. (1993): Random coefficient models. (pp. 107-125): San Diego (CA): Academic Press Oxford: Clarendon Press Goldstein, H. (1995): Multilevel statistical models McCuilagh, P., and Nelder, J.A. (1989): Generalized (2nd ed.): London: Edward Arnold linear models (2nd ed.): London: Chapman & Hall Goidstein, H., and Rasbash, J. (1989): Efficient com- Potthoff, R.F., and Roy, S.N. (1964): A generalized putational procedures for the estimation of parame- multivariate analysis of variance model useful espe- ters in multilevel models, based on iterative general- cially for growth curve problems, Biometrika 51,313- ized least squares. Manuscript submitted for publica- 326 tion Prosser, R., Rasbash, J., and Goldstein, H. (1990): Goldstein, H., Healy, M.J.R., and Rasbash, J. (1994): ML3 Software for Three-level Analysis User's Guide. Multilevel time series models with applications to re- Institute of Education, University of London THE STATISTICAL SOFTWARE NEWSLETTER 605

Rao, C.R. (1959): Some problems involving linear hy- analysis with applications in education and psychol- potheses in multivariate analysis. Biometrika 46, 49- ogy. Belmont, CA: Wadsworth 58 Timm, N.H. (1980): Multivariate analysis of variance Rao, C.R. (1965): The theory of least squares when of repeated measurements. In: P.R. Krishnaiah (Ed.). the parameters are stochastic and its application to the Handbook of statistics, Vol. 1 (pp. 41-87): Amster- analysis of growth curves. Biometrika 52, 447-458 dam: North Holland Rao, C.R. (1966): Covariance adjustment and related Van der Leeden, R. (1990): Reduced rank regression problems in multivariate analysis. In P.R. Krishnaiah with structured residuals. Leiden: DSWO Press (Ed.). Multivariate analysis H, pp. 87-103. New York: Academic Press Visser, R.A. (1982): On quantitative longitudinal data in psychological research. Ph.D. dissertation: Univer- Rao, C.R. (1967): Least squares theory using an esti- sity of Leiden. Published in 1985 as: Analysis of longi- mated dispersion matrix and its application to meas- tudinal data in behavioral and social research. Leiden: urement of signals. Proceedings of the 5th Berkeley DSWO Press Symposium on Mathematical Statistics 1,355-372 Voeten, M.J.M. (1991): Beschrijving van de indi- Rao, C.R. (1973): Linear statistical inference and its viduele ontwikkeling van leesvorderingen [Description applications, 2rid ed. New York: John Wiley of the individual development of reading achieve- Rasbash, J., Yang, M., Woodhouse, G., and Goldstein, ment]. In: J. Hoogstraten, & W.J. van der Linden H. (1995): MLn: command reference guide. London: (Eds.). Methodologie. Onderwijsresearchdagen 91 Institute of Education [Methodology. Educational research days 91] (pp. 79- 88): Amsterdam: SCO Raudenbush, S.W. (1988): Educational applications of hierarchical linear models: A review. Journal of Edu- Vrijburg, K.E. (1991): Het modelleren van groeidata cational Statistics 13, 85-118 [The modeling of growth data]. Unpublished MA thesis. Department of Psychometrics and Research Meth- Raudenbush, S.W., and Bryk, A.S. (1985): Empirical odology, University of Leiden Bayes meta-analysis. Journal of Educational Statistics 10, 75-98 Ware, J.H. (1985): Linear models for the analysis of longitudinal studies. JASA 80, 95-101 Schluchter, M.D. (1988): Unbalanced repeated measures models with structured covariance matrices. In: Zeger, S.L., Liang, K.-Y., and Albert, P.S. (1988): W.J. Dixon (Ed.). BMDP Statistical software manual, Models for longitudinal data: a generalized estimating Vol. 2 (pp. 1081-1114): Berkeley, CA: University of equation approach. Biometrics 44, 1049-1060 California Press. Timm, N.H. (1975): Multivariate