Grobner¨ Basis and Structural Equation Modeling by Min Lim A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Graduate Department of Statistics University of Toronto Copyright c 2010 by Min Lim Abstract Gr¨obnerBasis and Structural Equation Modeling Min Lim Doctor of Philosophy Graduate Department of Statistics University of Toronto 2010 Structural equation models are systems of simultaneous linear equations that are gener- alizations of linear regression, and have many applications in the social, behavioural and biological sciences. A serious barrier to applications is that it is easy to specify models for which the parameter vector is not identifiable from the distribution of the observable data, and it is often difficult to tell whether a model is identified or not. In this thesis, we study the most straightforward method to check for identification – solving a system of simultaneous equations. However, the calculations can easily get very complex. Gr¨obner basis is introduced to simplify the process. The main idea of checking identification is to solve a set of finitely many simultaneous equations, called identifying equations, which can be transformed into polynomials. If a unique solution is found, the model is identified. Gr¨obner basis reduces the polynomials into simpler forms making them easier to solve. Also, it allows us to investigate the model-induced constraints on the covariances, even when the model is not identified. With the explicit solution to the identifying equations, including the constraints on the covariances, we can (1) locate points in the parameter space where the model is not iden- tified, (2) find the maximum likelihood estimators, (3) study the effects of mis-specified models, (4) obtain a set of method of moments estimators, and (5) build customized parametric and distribution free tests, including inference for non-identified models. ii Contents 1 Introduction 1 1.1 Structural Equation Models . 1 1.2 Special Cases . 5 1.3 The Importance of Model Identification . 7 1.4 Normality . 9 1.5 Intercepts . 10 1.6 Summary of Thesis . 12 2 Model Identification 15 2.1 Various Types of Identification . 15 2.2 Identification for Structural Equation Models . 20 2.3 The Available Methods . 22 2.3.1 Explicit Solution . 22 2.3.2 Counting Rule . 23 2.3.3 Methods for Surface Models . 26 2.3.4 Methods for Factor Analysis Models . 31 2.3.5 Methods for General Models . 40 2.3.6 Methods for Special Models . 41 2.3.7 Tests of Local Identification . 44 2.3.8 Empirical Identification Tests . 45 iii 3 Theory of Gr¨obnerBasis 47 3.1 Background and Definitions . 48 3.2 Gr¨obnerBasis . 58 3.3 Applications of Gr¨obnerBasis in Model Identification . 67 3.3.1 Roots of the Identifying Equations . 67 3.3.2 Equality Constraints on the Covariances . 73 3.3.3 Checking Model Identification . 75 3.3.4 Introduce Extra Constraints to Identify a Model . 86 3.3.5 Identifying a Function of the Parameter Vector . 88 3.3.6 Non-Recursive Models . 90 4 The Explicit Solution 98 4.1 Points where the Model is not Identified . 98 4.2 Maximum Likelihood Estimators . 101 4.3 Effects of Mis-specified Models . 104 4.4 Method of Moments Estimators . 107 4.5 Customized Tests . 110 4.5.1 Goodness-of-Fit Test . 110 4.5.2 Other Hypothesis Tests . 121 5 Examples 125 5.1 Body Mass Index Health Data . 125 5.2 The Statistics of Poverty and Inequality . 147 5.2.1 First Proposed Model - One Latent Variable . 148 5.2.2 Second Proposed Model - Two Latent Variables . 156 6 Discussion 176 6.1 Contributions . 176 6.2 Limitations and Possible Difficulties . 178 iv 6.3 Directions for Future Research . 180 A Buchberger’s Algorithm 182 B Sample Mathematica Code 210 C Sample SAS and R Code 212 Bibliography 236 v List of Tables 1.1 Definition of Symbols in Path Diagrams . 3 3.1 Summary Results of the Buchberger’s Algorithm . 62 5.1 Body Mass Index Health Data - Goodness-of-Fit Tests for Initial Model . 136 5.2 Body Mass Index Health Data - Follow-Up Tests based on Likelihood Ratio142 5.3 Body Mass Index Health Data - Identification Residuals of Test 3 - Normal Theory . 143 5.4 Body Mass Index Health Data - Identification Residuals of Test 5 - Normal Theory . 143 5.5 Body Mass Index Health Data - Test for γ12 = γ22 = 0 . 145 5.6 Body Mass Index Health Data - Inference for the Regression Coefficients 147 5.7 Poverty Data - One Latent Variable - 5 Largest Asymptotically Standard- ized Residuals of Initial Test . 150 5.8 Poverty Data - Two Latent Variables - Goodness-of-Fit Tests for Initial Model . 162 5.9 Poverty Data - Two Latent Variables - Identification Residuals . 163 5.10 Poverty Data - Two Latent Variables - Goodness-of-Fit Tests for First Improved Model . 163 5.11 Poverty Data - Two Latent Variables - Goodness-of-Fit Tests for Second Improved Model . 164 vi 5.12 Poverty Data - Two Latent Variables - Estimates of the Variances . 166 5.13 Poverty Data - Two Latent Variables - Tests for the Variances . 166 5.14 Poverty Data - Two Latent Variables - Test for λY 3 = λY 4 . 168 5.15 Poverty Data - Two Latent Variables - Goodness-of-Fit Tests for Groups A, B and C based on Likelihood Ratio . 170 5.16 Poverty Data - Two Latent Variables - Test Statistics of Pairwise Com- parison Tests for Groups B and C based on Likelihood Ratio . 171 5.17 Poverty Data - Two Latent Variables - Changes in Pairwise Comparisons after Introducing γ3 .............................. 171 5.18 Poverty Data - Two Latent Variables - Goodness-of-Fit Tests after Intro- ducing γ3 and γ4 ............................... 173 5.19 Poverty Data - Two Latent Variables - Inference for the Variances - Normal Theory . 174 vii List of Figures 1.1 Path Diagram of General LISREL Model . 4 1.2 Path Diagram for Language Study . 5 1.3 Path Diagram of General Multivariate Regression Model . 6 1.4 Path Diagram of General Path Analysis Model . 7 1.5 Path Diagram of General Factor Analysis Model . 7 2.1 The Covariance Function . 22 2.2 Path Diagram of a Recursive Model . 27 2.3 Path Diagram of Duncan’s Just-Identified Non-Recursive Model . 27 3.1 Mathematica Output - Gr¨obner Basis . 67 3.2 Path Diagram for a Just-Identified Non-Recursive Model . 92 3.3 Path Diagram for an Over-Identified Non-Recursive Model . 95 5.1 Path Diagram for Body Mass Index Health Data - Initial Model . 128 5.2 Path Diagram for Body Mass Index Health Data - Adopted Model . 144 5.3 Path Diagram for Poverty Data - One Latent Variable - Initial Model . 149 5.4 Path Diagram for Poverty Data - One Latent Variable - Improved Model 151 5.5 Path Diagram for Poverty Data - Two Latent Variables - Initial Model . 157 5.6 Path Diagram for Poverty Data - Two Latent Variables - Adopted Model 167 5.7 Path Diagram for Poverty Data - Two Latent Variables - A More Reason- able Model . 168 viii 5.8 Path Diagram for Poverty Data - Two Latent Variables - Second Adopted Model . 175 ix Chapter 1 Introduction 1.1 Structural Equation Models Structural equation models [Blalock 1971], [Goldberger 1972], [Goldberger and Duncan 1973], [Aigner and Goldberger 1977], [Bielby and Hauser 1977], [Bentler and Weeks 1980], [Aigner, Hsiao, Kapteyn and Wansbeek 1984], [J¨oreskog and Wold 1982], [Bollen 1989] are extensions of the usual linear regression models potentially involving unobservable random variables that are not error terms. Also, a random variable may appear as an independent variable in one equation and a dependent variable in another. A random variable in a structural equation model can be classified in two ways. It may be either latent or manifest, and either exogenous or endogenous. As these terms may be unfamiliar, the definitions are provided below. Latent Variable A random variable that is not observable. Manifest Variable A random variable that is observable. It is part of the data set. Exogenous Variable A random variable that is not written as a function of any other variable in the model. 1 Chapter 1. Introduction 2 Endogenous Variable A random variable that is written as a function of at least one other variable in the model. In this thesis, the notation of the LISREL Model (also called the JKW model [J¨oreskog 1973], [Keesling 1972], [Wiley 1973]) is employed. Independently for i = 1, ..., N, ηi = α + βηi + Γξi + ζi Xi = νX + ΛX ξi + δi (1.1) Yi = νY + ΛY ηi + i, where ηi is an m × 1 vector of latent endogenous variables; ξi is an n × 1 vector of latent exogenous variables; Xi is a q × 1 vector of manifest indicators for ξi; Yi is a p × 1 vector of manifest indicators for ηi; ζi, δi and i are vectors of error terms, independent of each other and of ξi; E(ξi) = κ, E(ζi) = E(δi) = E(i) = 0; V (ξi) = Φ, V (ζi) = Ψ, V (δi) = Θδ, V (i) = Θ; α, β, Γ, νX , ΛX , νY and ΛY are matrices of constants, with the diagonal of β zero. The first equation in Model 1.1 is usually called the structural model while the second and third equations combined is usually called the measurement model. To reduce notational clutter, the subscript i is dropped for the rest of the thesis. Chapter 1. Introduction 3 Sometimes it is easier to present the model in a pictorial form, the path diagram. To understand these diagrams, it is necessary to define the symbols involved. A summary of these definitions can be found in Table 1.1. Note that path diagrams do not show intercepts. Table 1.1: Definition of Symbols in Path Diagrams A rectangular or square box signifies an observed X or manifest variable.