DOCSLIB.ORG

Gov 2000: 9. Multiple Regression in Matrix Form

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Gov 2000: 9. Multiple Regression in Matrix Form Matthew Blackwell November 19, 2015 1 / 62 1. Matrix algebra review 2. Matrix Operations 3. Writing the linear model more compactly 4. A bit more about matrices 5. OLS in matrix form 6. OLS inference in matrix form 2 / 62 Where are we? Where are we going? • Last few weeks: regression estimation and inference with one and two independent variables, varying effects • This week: the general regression model with arbitrary covariates • Next week: what happens when assumptions are wrong 3 / 62 Nunn & Wantchekon • Are there long-term, persistent effects of slave trade on Africans today? • Basic idea: compare levels of interpersonal trust (푌푖) across different levels of historical slave exports for a respondent’s ethnic group • Problem: ethnic groups and respondents might differ in their interpersonal trust in ways that correlate with the severity of slave exports • One solution: try to control for relevant differences between groups via multiple regression 4 / 62 Nunn & Wantchekon • Whaaaaa? Bold letter, quotation marks, what is this? • Today’s goal is to decipher this type of writing 5 / 62 Multiple Regression in R nunn <- foreign::read.dta("Nunn_Wantchekon_AER_2011.dta") mod <- lm(trust_neighbors ~ exports + age + male + urban_dum + malaria_ecology, data = nunn) summary(mod) ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 1.5030370 0.0218325 68.84 <2e-16 *** ## exports -0.0010208 0.0000409 -24.94 <2e-16 *** ## age 0.0050447 0.0004724 10.68 <2e-16 *** ## male 0.0278369 0.0138163 2.01 0.044 * ## urban_dum -0.2738719 0.0143549 -19.08 <2e-16 *** ## malaria_ecology 0.0194106 0.0008712 22.28 <2e-16 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 0.978 on 20319 degrees of freedom ## (1497 observations deleted due to missingness) ## Multiple R-squared: 0.0604, Adjusted R-squared: 0.0602 ## F-statistic: 261 on 5 and 20319 DF, p-value: <2e-16 6 / 62 Why matrices and vectors? 7 / 62 8 / 62 Why matrices and vectors? • Here’s one way to write the full multiple regression model: 푦푖 = 훽0 + 푥푖1훽1 + 푥푖2훽2 + ⋯ + 푥푖푘훽푘 + 푢푖 • Notation is going to get needlessly messy as we add variables • Matrices are clean, but they are like a foreign language • You need to build intuitions over a long period of time 9 / 62 Quick note about interpretation 푦푖 = 훽0 + 푥푖1훽1 + 푥푖2훽2 + ⋯ + 푥푖푘훽푘 + 푢푖 • In this model, 훽1 is the effect of a one-unit change in 푥푖1 conditional on all other 푥푖푗. • Jargon “partial effect,” “ceteris paribus,” “all else equal,” “conditional on the covariates,” etc 10 / 62 1/ Matrix algebra review 11 / 62 Matrices and vectors • A matrix is just a rectangular array of numbers. • We say that a matrix is 푛 × 푘 (“푛 by 푘”) if it has 푛 rows and 푘 columns. • Uppercase bold denotes a matrix: 푎 푎 ⋯ 푎 ⎡ 11 12 1푘 ⎤ ⎢ 푎 푎 ⋯ 푎 ⎥ 퐀 = ⎢ 21 22 2푘 ⎥ ⎢ ⋮ ⋮ ⋱ ⋮ ⎥ ⎣ 푎푛1 푎푛2 ⋯ 푎푛푘 ⎦ • Generic entry: 푎푖푗 where this is the entry in row 푖 and column 푗 • If you’ve used Excel, you’ve seen matrices. 12 / 62 Examples of matrices • One example of a matrix that we’ll use a lot is the design matrix, which has a column of ones, and then each of the subsequent columns is each independent variable in the regression. 1 exports age male ⎡ 1 1 1 ⎤ ⎢ 1 exports age male ⎥ 퐗 = ⎢ 2 2 2 ⎥ ⎢ ⋮ ⋮ ⋮ ⋮ ⎥ ⎣ 1 exports푛 age푛 male푛 ⎦ 13 / 62 Design matrix in R head(model.matrix(mod), 8) ## (Intercept) exports age male urban_dum malaria_ecology ## 1 1 855 40 0 0 28.15 ## 2 1 855 25 1 0 28.15 ## 3 1 855 38 1 1 28.15 ## 4 1 855 37 0 1 28.15 ## 5 1 855 31 1 0 28.15 ## 6 1 855 45 0 0 28.15 ## 7 1 855 20 1 0 28.15 ## 8 1 855 31 0 0 28.15 dim(model.matrix(mod)) ##[1]20325 6 14 / 62 Vectors • A vector is just a matrix with only one row or one column. • A row vector is a vector with only one row, sometimes called a 1 × 푘 vector: 휶 = [ 훼1 훼2 훼3 ⋯ 훼푘 ] • A column vector is a vector with one column and more than one row. Here is a 푛 × 1 vector: 푦 ⎡ 1 ⎤ ⎢ 푦 ⎥ 퐲 = ⎢ 2 ⎥ ⎢ ⋮ ⎥ ⎣ 푦푛 ⎦ • Convention we’ll assume that a vector is column vector and vectors will be written with lowercase bold lettering (퐛) 15 / 62 Vector examples • One really common vector that we will work with are individual variables, such as the dependent variable, which we will represent as 퐲: 푦 ⎡ 1 ⎤ ⎢ 푦 ⎥ 퐲 = ⎢ 2 ⎥ ⎢ ⋮ ⎥ ⎣ 푦푛 ⎦ 16 / 62 Vectors in R • Vectors can come from subsets of matrices: model.matrix(mod)[1, ] ## (Intercept) exports age male ## 1.00 854.96 40.00 0.00 ## urban_dum malaria_ecology ## 0.00 28.15 • Note, though, that R always prints a vector in row form, even if it is a column in the original data: head(nunn$trust_neighbors) ##[1]330011 17 / 62 2/ Matrix Operations 18 / 62 Transpose • The transpose of a matrix 퐀 is the matrix created by switching the rows and columns of the data and is denoted 퐀′. • 푘th column of 퐀 becomes the 푘th row of 퐀′: 푎11 푎12 ⎡ ⎤ ′ 푎11 푎21 푎31 퐀 = ⎢ 푎21 푎22 ⎥ 퐀 = [ ] 푎12 푎22 푎32 ⎣ 푎31 푎32 ⎦ • If 퐀 is 푛 × 푘, then 퐀′ will be 푘 × 푛. • Also written 퐀퐓 19 / 62 Transposing vectors • Transposing will turn a 푘 × 1 column vector into a 1 × 푘 row vector and vice versa: 1 ⎡ ⎤ ⎢ 3 ⎥ 휔 = ⎢ ⎥ 휔′ = [ 1 3 2 −5 ] ⎢ 2 ⎥ ⎣ −5 ⎦ 20 / 62 Transposing in R a <- matrix(1:6, ncol = 3, nrow = 2) a ## [,1] [,2] [,3] ## [1,] 1 3 5 ## [2,] 2 4 6 t(a) ## [,1] [,2] ## [1,] 1 2 ## [2,] 3 4 ## [3,] 5 6 21 / 62 Write matrices as vectors • A matrix is just a collection of vectors (row or column) • As a row vector: ′ 푎11 푎12 푎13 퐚1 퐀 = [ ] = [ ′ ] 푎21 푎22 푎23 퐚2 with row vectors ′ ′ 퐚1 = [ 푎11 푎12 푎13 ] 퐚2 = [ 푎21 푎22 푎23 ] • Or we can define it in terms of column vectors: 푏 푏 ⎡ 11 12 ⎤ 퐁 = ⎢ 푏21 푏22 ⎥ = [ 퐛ퟏ 퐛ퟐ ] ⎣ 푏31 푏32 ⎦ where 퐛ퟏ and 퐛ퟐ represent the columns of 퐁. • 푗 subscripts columns of a matrix: 퐱푗 ′ • 푖 and 푡 will be used for rows 퐱푖 . 22 / 62 Addition and subtraction • How do we add or subtract matrices and vectors? • First, the matrices/vectors need to be comformable, meaning that the dimensions have to be the same • Let 퐀 and 퐁 both be 2 × 2 matrices. Then, let 퐂 = 퐀 + 퐁, where we add each cell together: 푎 푎 푏 푏 퐀 + 퐁 = [ 11 12 ] + [ 11 12 ] 푎21 푎22 푏21 푏22 푎 + 푏 푎 + 푏 = [ 11 11 12 12 ] 푎21 + 푏21 푎22 + 푏22 푐 푐 = [ 11 12 ] 푐21 푐22 = 퐂 23 / 62 Scalar multiplication • A scalar is just a single number: you can think of it sort of like a 1 by 1 matrix. • When we multiply a scalar by a matrix, we just multiply each element/cell by that scalar: 푎 푎 훼 × 푎 훼 × 푎 훼퐀 = 훼 [ 11 12 ] = [ 11 12 ] 푎21 푎22 훼 × 푎21 훼 × 푎22 24 / 62 3/ Writing the linear model more compactly 25 / 62 The linear model with new notation • Remember that we wrote the linear model as the following for all 푖 ∈ [1, … , 푛]: 푦푖 = 훽0 + 푥푖훽1 + 푧푖훽2 + 푢푖 • Imagine we had an 푛 of 4. We could write out each formula: 푦1 = 훽0 + 푥1훽1 + 푧1훽2 + 푢1 (unit 1) 푦2 = 훽0 + 푥2훽1 + 푧2훽2 + 푢2 (unit 2) 푦3 = 훽0 + 푥3훽1 + 푧3훽2 + 푢3 (unit 3) 푦4 = 훽0 + 푥4훽1 + 푧4훽2 + 푢4 (unit 4) 26 / 62 The linear model with new notation 푦1 = 훽0 + 푥1훽1 + 푧1훽2 + 푢1 (unit 1) 푦2 = 훽0 + 푥2훽1 + 푧2훽2 + 푢2 (unit 2) 푦3 = 훽0 + 푥3훽1 + 푧3훽2 + 푢3 (unit 3) 푦4 = 훽0 + 푥4훽1 + 푧4훽2 + 푢4 (unit 4) • We can write this as: 푦 1 푥 푧 푢 ⎡ 1 ⎤ ⎡ ⎤ ⎡ 1 ⎤ ⎡ 1 ⎤ ⎡ 1 ⎤ ⎢ 푦2 ⎥ ⎢ 1 ⎥ ⎢ 푥2 ⎥ ⎢ 푧2 ⎥ ⎢ 푢2 ⎥ ⎢ ⎥ = ⎢ ⎥ 훽0 + ⎢ ⎥ 훽1 + ⎢ ⎥ 훽2 + ⎢ ⎥ ⎢ 푦3 ⎥ ⎢ 1 ⎥ ⎢ 푥3 ⎥ ⎢ 푧3 ⎥ ⎢ 푢3 ⎥ ⎣ 푦4 ⎦ ⎣ 1 ⎦ ⎣ 푥4 ⎦ ⎣ 푧4 ⎦ ⎣ 푢4 ⎦ • Outcome is a linear combination of the the 퐱, 퐳, and 퐮 vectors 27 / 62 Grouping things into matrices • Can we write this in a more compact form? Yes! Let 퐗 and 휷 be the following: 1 푥1 푧1 ⎡ ⎤ 훽0 ⎢ 1 푥2 푧2 ⎥ ⎡ ⎤ 퐗 = ⎢ ⎥ 휷 = ⎢ 훽1 ⎥ (4×3) ⎢ 1 푥3 푧3 ⎥ (3×1) ⎣ 훽2 ⎦ ⎣ 1 푥4 푧4 ⎦ 28 / 62 Matrix multiplication by a vector • We can write this more compactly as a matrix (post-)multiplied by a vector: 1 푥 푧 ⎡ ⎤ ⎡ 1 ⎤ ⎡ 1 ⎤ ⎢ 1 ⎥ ⎢ 푥2 ⎥ ⎢ 푧2 ⎥ ⎢ ⎥ 훽0 + ⎢ ⎥ 훽1 + ⎢ ⎥ 훽2 = 퐗휷 ⎢ 1 ⎥ ⎢ 푥3 ⎥ ⎢ 푧3 ⎥ ⎣ 1 ⎦ ⎣ 푥4 ⎦ ⎣ 푧4 ⎦ • Multiplication of a matrix by a vector is just the linear combination of the columns of the matrix with the vector elements as weights/coefficients. • And the left-hand side here only uses scalars times vectors, which is easy! 29 / 62 General matrix by vector multiplication • 퐀 is a 푛 × 푘 matrix • 퐛 is a 푘 × 1 column vector • Columns of 퐀 have to match rows of 퐛 • Let 퐚푗 be the 푗th column of 퐴. Then we can write: 퐜 = 퐀퐛 = 푏1퐚1 + 푏2퐚2 + ⋯ + 푏푘퐚푘 (푛×1) • 퐜 is linear combination of the columns of 퐀 30 / 62 Back to regression • 퐗 is the 푛 × (푘 + 1) design matrix of independent variables • 휷 be the (푘 + 1) × 1 column vector of coefficients. • 퐗휷 will be 푛 × 1: 퐗휷 = 훽0 + 훽1퐱1 + 훽2퐱2 + ⋯ + 훽푘퐱푘 • Thus, we can compactly write the linear model as the following: 퐲 = 퐗휷 + 퐮 (푛×1) (푛×1) (푛×1) ′ • We can also write this at the individual level, where 퐱푖 is the 푖th row of 퐗: ′ 푦푖 = 퐱푖 휷 + 푢푖 31 / 62 4/ A bit more about matrices 32 / 62 Matrix multiplication • What if, instead of a column vector 푏, we have a matrix 퐁 with dimensions 푘 × 푚.
Recommended publications
  • Lecture 13: Simple Linear Regression in Matrix Format

    Lecture 13: Simple Linear Regression in Matrix Format

    11:55 Wednesday 14th October, 2015 See updates and corrections at http://www.stat.cmu.edu/~cshalizi/mreg/ Lecture 13: Simple Linear Regression in Matrix Format 36-401, Section B, Fall 2015 13 October 2015 Contents 1 Least Squares in Matrix Form 2 1.1 The Basic Matrices . .2 1.2 Mean Squared Error . .3 1.3 Minimizing the MSE . .4 2 Fitted Values and Residuals 5 2.1 Residuals . .7 2.2 Expectations and Covariances . .7 3 Sampling Distribution of Estimators 8 4 Derivatives with Respect to Vectors 9 4.1 Second Derivatives . 11 4.2 Maxima and Minima . 11 5 Expectations and Variances with Vectors and Matrices 12 6 Further Reading 13 1 2 So far, we have not used any notions, or notation, that goes beyond basic algebra and calculus (and probability). This has forced us to do a fair amount of book-keeping, as it were by hand. This is just about tolerable for the simple linear model, with one predictor variable. It will get intolerable if we have multiple predictor variables. Fortunately, a little application of linear algebra will let us abstract away from a lot of the book-keeping details, and make multiple linear regression hardly more complicated than the simple version1. These notes will not remind you of how matrix algebra works. However, they will review some results about calculus with matrices, and about expectations and variances with vectors and matrices. Throughout, bold-faced letters will denote matrices, as a as opposed to a scalar a. 1 Least Squares in Matrix Form Our data consists of n paired observations of the predictor variable X and the response variable Y , i.e., (x1; y1);::: (xn; yn).
  • Introducing the Game Design Matrix: a Step-By-Step Process for Creating Serious Games

    Introducing the Game Design Matrix: a Step-By-Step Process for Creating Serious Games

    Air Force Institute of Technology AFIT Scholar Theses and Dissertations Student Graduate Works 3-2020 Introducing the Game Design Matrix: A Step-by-Step Process for Creating Serious Games Aaron J. Pendleton Follow this and additional works at: https://scholar.afit.edu/etd Part of the Educational Assessment, Evaluation, and Research Commons, Game Design Commons, and the Instructional Media Design Commons Recommended Citation Pendleton, Aaron J., "Introducing the Game Design Matrix: A Step-by-Step Process for Creating Serious Games" (2020). Theses and Dissertations. 4347. https://scholar.afit.edu/etd/4347 This Thesis is brought to you for free and open access by the Student Graduate Works at AFIT Scholar. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of AFIT Scholar. For more information, please contact [email protected]. INTRODUCING THE GAME DESIGN MATRIX: A STEP-BY-STEP PROCESS FOR CREATING SERIOUS GAMES THESIS Aaron J. Pendleton, Captain, USAF AFIT-ENG-MS-20-M-054 DEPARTMENT OF THE AIR FORCE AIR UNIVERSITY AIR FORCE INSTITUTE OF TECHNOLOGY Wright-Patterson Air Force Base, Ohio DISTRIBUTION STATEMENT A APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED. The views expressed in this document are those of the author and do not reflect the official policy or position of the United States Air Force, the United States Department of Defense or the United States Government. This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States. AFIT-ENG-MS-20-M-054 INTRODUCING THE GAME DESIGN MATRIX: A STEP-BY-STEP PROCESS FOR CREATING LEARNING OBJECTIVE BASED SERIOUS GAMES THESIS Presented to the Faculty Department of Electrical and Computer Engineering Graduate School of Engineering and Management Air Force Institute of Technology Air University Air Education and Training Command in Partial Fulfillment of the Requirements for the Degree of Master of Science in Cyberspace Operations Aaron J.
  • Stat 5102 Notes: Regression

    Stat 5102 Notes: Regression

    Stat 5102 Notes: Regression Charles J. Geyer April 27, 2007 In these notes we do not use the “upper case letter means random, lower case letter means nonrandom” convention. Lower case normal weight letters (like x and β) indicate scalars (real variables). Lowercase bold weight letters (like x and β) indicate vectors. Upper case bold weight letters (like X) indicate matrices. 1 The Model The general linear model has the form p X yi = βjxij + ei (1.1) j=1 where i indexes individuals and j indexes different predictor variables. Ex- plicit use of (1.1) makes theory impossibly messy. We rewrite it as a vector equation y = Xβ + e, (1.2) where y is a vector whose components are yi, where X is a matrix whose components are xij, where β is a vector whose components are βj, and where e is a vector whose components are ei. Note that y and e have dimension n, but β has dimension p. The matrix X is called the design matrix or model matrix and has dimension n × p. As always in regression theory, we treat the predictor variables as non- random. So X is a nonrandom matrix, β is a nonrandom vector of unknown parameters. The only random quantities in (1.2) are e and y. As always in regression theory the errors ei are independent and identi- cally distributed mean zero normal. This is written as a vector equation e ∼ Normal(0, σ2I), where σ2 is another unknown parameter (the error variance) and I is the identity matrix. This implies y ∼ Normal(µ, σ2I), 1 where µ = Xβ.
  • Uncertainty of the Design and Covariance Matrices in Linear Statistical Model*

    Uncertainty of the Design and Covariance Matrices in Linear Statistical Model*

    Acta Univ. Palacki. Olomuc., Fac. rer. nat., Mathematica 48 (2009) 61–71 Uncertainty of the design and covariance matrices in linear statistical model* Lubomír KUBÁČEK 1, Jaroslav MAREK 2 Department of Mathematical Analysis and Applications of Mathematics, Faculty of Science, Palacký University, tř. 17. listopadu 12, 771 46 Olomouc, Czech Republic 1e-mail: [email protected] 2e-mail: [email protected] (Received January 15, 2009) Abstract The aim of the paper is to determine an influence of uncertainties in design and covariance matrices on estimators in linear regression model. Key words: Linear statistical model, uncertainty, design matrix, covariance matrix. 2000 Mathematics Subject Classification: 62J05 1 Introduction Uncertainties in entries of design and covariance matrices influence the variance of estimators and cause their bias. A problem occurs mainly in a linearization of nonlinear regression models, where the design matrix is created by deriva- tives of some functions. The question is how precise must these derivatives be. Uncertainties of covariance matrices must be suppressed under some reasonable bound as well. The aim of the paper is to give the simple rules which enables us to decide how many ciphers an entry of the mentioned matrices must be consisted of. *Supported by the Council of Czech Government MSM 6 198 959 214. 61 62 Lubomír KUBÁČEK, Jaroslav MAREK 2 Symbols used In the following text a linear regression model (in more detail cf. [2]) is denoted as k Y ∼n (Fβ, Σ), β ∈ R , (1) where Y is an n-dimensional random vector with the mean value E(Y) equal to Fβ and with the covariance matrix Var(Y)=Σ.ThesymbolRk means the k-dimensional linear vector space.
  • The Concept of a Generalized Inverse for Matrices Was Introduced by Moore(1920)

    The Concept of a Generalized Inverse for Matrices Was Introduced by Moore(1920)

    J. Japanese Soc. Comp. Statist. 2(1989), 1-7 THE MOORE-PENROSE INVERSE MATRIX POR THE BALANCED ANOVA MODELS Byung Chun Kim* and Ha Sik Sunwoo* ABSTRACT Since the design matrix of the balanced linear model with no interactions has special form, the general solution of the normal equations can be easily found. From the relationships between the minimum norm least squares solution and the Moore-Penrose inverse we can obtain the explicit form of the Moore-Penrose inverse X+ of the design matrix of the model y = XƒÀ + ƒÃ for the balanced model with no interaction. 1. Introduction The concept of a generalized inverse for matrices was introduced by Moore(1920). He developed it in the context of linear transformations from n-dimensional to m-dimensional vector space over a complex field with usual Euclidean norm. Unaware of Moore's work, Penrose(1955) established the existence and the uniqueness of the generalized inverse that satisfies the Moore's condition under L2 norm. It is commonly known as the Moore-Penrose inverse. In the early 1970's many statisticians, Albert (1972), Ben-Israel and Greville(1974), Graybill(1976),Lawson and Hanson(1974), Pringle and Raynor(1971), Rao and Mitra(1971), Rao(1975), Schmidt(1976), and Searle(1971, 1982), worked on this subject and yielded sev- eral texts that treat the mathematics and its application to statistics in some depth in detail. Especially, Shinnozaki et. al.(1972a, 1972b) have surveyed the general methods to find the Moore-Penrose inverse in two subject: direct methods and iterative methods. Also they tested their accuracy for some cases.
  • Kriging Models for Linear Networks and Non‐Euclidean Distances

    Kriging Models for Linear Networks and Non‐Euclidean Distances

    Received: 13 November 2017 | Accepted: 30 December 2017 DOI: 10.1111/2041-210X.12979 RESEARCH ARTICLE Kriging models for linear networks and non-Euclidean distances: Cautions and solutions Jay M. Ver Hoef Marine Mammal Laboratory, NOAA Fisheries Alaska Fisheries Science Center, Abstract Seattle, WA, USA 1. There are now many examples where ecological researchers used non-Euclidean Correspondence distance metrics in geostatistical models that were designed for Euclidean dis- Jay M. Ver Hoef tance, such as those used for kriging. This can lead to problems where predictions Email: [email protected] have negative variance estimates. Technically, this occurs because the spatial co- Handling Editor: Robert B. O’Hara variance matrix, which depends on the geostatistical models, is not guaranteed to be positive definite when non-Euclidean distance metrics are used. These are not permissible models, and should be avoided. 2. I give a quick review of kriging and illustrate the problem with several simulated examples, including locations on a circle, locations on a linear dichotomous net- work (such as might be used for streams), and locations on a linear trail or road network. I re-examine the linear network distance models from Ladle, Avgar, Wheatley, and Boyce (2017b, Methods in Ecology and Evolution, 8, 329) and show that they are not guaranteed to have a positive definite covariance matrix. 3. I introduce the reduced-rank method, also called a predictive-process model, for creating valid spatial covariance matrices with non-Euclidean distance metrics. It has an additional advantage of fast computation for large datasets. 4. I re-analysed the data of Ladle et al.
  • Linear Regression with Shuffled Data: Statistical and Computational

    Linear Regression with Shuffled Data: Statistical and Computational

    3286 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 64, NO. 5, MAY 2018 Linear Regression With Shuffled Data: Statistical and Computational Limits of Permutation Recovery Ashwin Pananjady ,MartinJ.Wainwright,andThomasA.Courtade Abstract—Consider a noisy linear observation model with an permutation matrix, and w Rn is observation noise. We refer ∈ unknown permutation, based on observing y !∗ Ax∗ w, to the setting where w 0asthenoiseless case.Aswithlinear d = + where x∗ R is an unknown vector, !∗ is an unknown = ∈ n regression, there are two settings of interest, corresponding to n n permutation matrix, and w R is additive Gaussian noise.× We analyze the problem of∈ permutation recovery in a whether the design matrix is (i) deterministic (the fixed design random design setting in which the entries of matrix A are case), or (ii) random (the random design case). drawn independently from a standard Gaussian distribution and There are also two complementary problems of interest: establish sharp conditions on the signal-to-noise ratio, sample recovery of the unknown !∗,andrecoveryoftheunknownx∗. size n,anddimensiond under which !∗ is exactly and approx- In this paper, we focus on the former problem; the latter imately recoverable. On the computational front, we show that problem is also known as unlabelled sensing [5]. the maximum likelihood estimate of !∗ is NP-hard to compute for general d,whilealsoprovidingapolynomialtimealgorithm The observation model (1) is frequently encountered in when d 1. scenarios where there is uncertainty in the order in which = Index Terms—Correspondenceestimation,permutationrecov- measurements are taken. An illustrative example is that of ery, unlabeled sensing, information-theoretic bounds, random sampling in the presence of jitter [6], in which the uncertainty projections.
  • Week 7: Multiple Regression

    Week 7: Multiple Regression

    Week 7: Multiple Regression Brandon Stewart1 Princeton October 24, 26, 2016 1These slides are heavily influenced by Matt Blackwell, Adam Glynn, Jens Hainmueller and Danny Hidalgo. Stewart (Princeton) Week 7: Multiple Regression October 24, 26, 2016 1 / 145 Where We've Been and Where We're Going... Last Week I regression with two variables I omitted variables, multicollinearity, interactions This Week I Monday: F matrix form of linear regression I Wednesday: F hypothesis tests Next Week I break! I then ::: regression in social science Long Run I probability ! inference ! regression Questions? Stewart (Princeton) Week 7: Multiple Regression October 24, 26, 2016 2 / 145 1 Matrix Algebra Refresher 2 OLS in matrix form 3 OLS inference in matrix form 4 Inference via the Bootstrap 5 Some Technical Details 6 Fun With Weights 7 Appendix 8 Testing Hypotheses about Individual Coefficients 9 Testing Linear Hypotheses: A Simple Case 10 Testing Joint Significance 11 Testing Linear Hypotheses: The General Case 12 Fun With(out) Weights Stewart (Princeton) Week 7: Multiple Regression October 24, 26, 2016 3 / 145 Why Matrices and Vectors? Here's one way to write the full multiple regression model: yi = β0 + xi1β1 + xi2β2 + ··· + xiK βK + ui Notation is going to get needlessly messy as we add variables Matrices are clean, but they are like a foreign language You need to build intuitions over a long period of time (and they will return in Soc504) Reminder of Parameter Interpretation: β1 is the effect of a one-unit change in xi1 conditional on all other xik . We are going to review the key points quite quickly just to refresh the basics.
  • Stat 714 Linear Statistical Models

    Stat 714 Linear Statistical Models

    STAT 714 LINEAR STATISTICAL MODELS Fall, 2010 Lecture Notes Joshua M. Tebbs Department of Statistics The University of South Carolina TABLE OF CONTENTS STAT 714, J. TEBBS Contents 1 Examples of the General Linear Model 1 2 The Linear Least Squares Problem 13 2.1 Least squares estimation . 13 2.2 Geometric considerations . 15 2.3 Reparameterization . 25 3 Estimability and Least Squares Estimators 28 3.1 Introduction . 28 3.2 Estimability . 28 3.2.1 One-way ANOVA . 35 3.2.2 Two-way crossed ANOVA with no interaction . 37 3.2.3 Two-way crossed ANOVA with interaction . 39 3.3 Reparameterization . 40 3.4 Forcing least squares solutions using linear constraints . 46 4 The Gauss-Markov Model 54 4.1 Introduction . 54 4.2 The Gauss-Markov Theorem . 55 4.3 Estimation of σ2 in the GM model . 57 4.4 Implications of model selection . 60 4.4.1 Underfitting (Misspecification) . 60 4.4.2 Overfitting . 61 4.5 The Aitken model and generalized least squares . 63 5 Distributional Theory 68 5.1 Introduction . 68 i TABLE OF CONTENTS STAT 714, J. TEBBS 5.2 Multivariate normal distribution . 69 5.2.1 Probability density function . 69 5.2.2 Moment generating functions . 70 5.2.3 Properties . 72 5.2.4 Less-than-full-rank normal distributions . 73 5.2.5 Independence results . 74 5.2.6 Conditional distributions . 76 5.3 Noncentral χ2 distribution . 77 5.4 Noncentral F distribution . 79 5.5 Distributions of quadratic forms . 81 5.6 Independence of quadratic forms . 85 5.7 Cochran's Theorem .
  • Fitting Differential Equations to Functional Data: Principal Differential Analysis

    Fitting Differential Equations to Functional Data: Principal Differential Analysis

    19 Fitting differential equations to functional data: Principal differential analysis 19.1 Introduction Now that we have fastened a belt of tools around our waists for tinker- ing with differential equations, we return to the problems introduced in Chapter 17 ready to get down to some serious work. Using a differential equation as a modelling object involves concepts drawn from both the functional linear model and from principal com- ponents analysis. A differential equation can certainly capture the shape features in both the curve and its derivative for a single functional datum such as the oil refinery observation shown in Figure 1.4. But because the set of solutions to a differential equation is an entire function space, it can also model variation across observations when N>1. In this sense, it also has the flavor of principal components analysis where we find a subspace of functions able to capture the dominant modes of variation in the data. We have, then, a question of emphasis or perspective. On one hand, the data analyst may want to capture important features of the dynamics of a single observation, and thus look within the m-dimensional space of solutions of an estimated equation to find that which gives the best account of the data. On the other hand, the goal may be to see how much functional variation can be explained across multiple realizations of a process. Thus, linear modelling and variance decomposition merge into one analysis in this environment. We introduce a new term here: principal differential analysis means the fitting of a differential equation to noisy data so as to capture either the 328 19.
  • Linear Regression in Matrix Form

    Linear Regression in Matrix Form

    Statistics 512: Applied Linear Models Topic 3 Topic Overview This topic will cover • thinking in terms of matrices • regression on multiple predictor variables • case study: CS majors • Text Example (KNNL 236) Chapter 5: Linear Regression in Matrix Form The SLR Model in Scalar Form iid 2 Yi = β0 + β1Xi + i where i ∼ N(0,σ ) Consider now writing an equation for each observation: Y1 = β0 + β1X1 + 1 Y2 = β0 + β1X2 + 2 . Yn = β0 + β1Xn + n TheSLRModelinMatrixForm Y1 β0 + β1X1 1 Y2 β0 + β1X2 2 . = . + . . . . Yn β0 + β1Xn n Y X 1 1 1 1 Y X 2 1 2 β0 2 . = . + . . . β1 . Yn 1 Xn n (I will try to use bold symbols for matrices. At first, I will also indicate the dimensions as a subscript to the symbol.) 1 • X is called the design matrix. • β is the vector of parameters • is the error vector • Y is the response vector The Design Matrix 1 X1 1 X2 Xn×2 = . . 1 Xn Vector of Parameters β0 β2×1 = β1 Vector of Error Terms 1 2 n×1 = . . n Vector of Responses Y1 Y2 Yn×1 = . . Yn Thus, Y = Xβ + Yn×1 = Xn×2β2×1 + n×1 2 Variance-Covariance Matrix In general, for any set of variables U1,U2,... ,Un,theirvariance-covariance matrix is defined to be 2 σ {U1} σ{U1,U2} ··· σ{U1,Un} . σ{U ,U } σ2{U } ... σ2{ } 2 1 2 . U = . . .. .. σ{Un−1,Un} 2 σ{Un,U1} ··· σ{Un,Un−1} σ {Un} 2 where σ {Ui} is the variance of Ui,andσ{Ui,Uj} is the covariance of Ui and Uj.
  • Some Matrix Results

    Some Matrix Results

    APPENDIX Some matrix results The main matrix results required for this book are given in this Appendix. Most of them are stated without proof as they are relatively straightforward and can be found in most books on matrix algebra; see for instance Searle (1966), Basilevsky (1983) and Graybill (1983). The properties of circulant matrices are considered in more detail as they have particular relevance in the book and are perhaps not widely known. A detailed account of circulant matrices can be found in Davis (1979). A.1 Notation An n x m matrix A = ((aij)) has n rows and m columns with aij being the element in the ith row andjth column. A' will denote the transpose of A. A is symmetrie if A = A'. If m = 1 then the matrix will be called a eolumn veetor and will usually be denoted by a lower ca se letter, a say. The transpose of a will be a row veetor a'. A diagonal matrix whose diagonal elements are those ofthe vector a, and whose ofT-diagonal elements are zero, will be denoted by a6• The inverse of a6 will be denoted by a- 6• More generally, if each element in a is raised to power p then the corresponding diagonal matrix will be denoted by a P6• Some special matrices and vectors are: In' the n x 1 vector with every element unity In = 1~, the n x n identity matrix Jnxm = I nl;", the n x m matrix with every element unity J n = J nxn Kn = (l/n)Jn the n x n matrix with every element n -1.