Matrix Algebra
Total Page:16
File Type:pdf, Size:1020Kb
Appendix Matrix Algebra This review consists of a brief exposition of the principal results of matrix algebra, scattered with a few exercises and examples. For a more detailed account we recom mend the book by STRANG [318]. Data table In multivariate data analysis we handle tables of numbers (matrices) and the one to start with is generally the data table Z Variables (columns) Zl,l Zl,i Zl,N (1.1) Samples [Zni] = Z. (rows) Zn,l Za,i Zo:,N Zn,l Zn,i Zn,N The element Zni denotes a numerical value placed at the crossing of the row num ber a (index of the sample) and the column number i (index of the variable). Matrix, vector, scalar A matrix is a rectangular array of numbers, (1.2) with indices i = 1, ... ,n, j = 1, ... ,m. (1.3) We speak of a matrix of order n x m to designate a matrix of n rows and m columns. A vector (by convention: a column vector) of dimension n is a matrix of order n x 1, i.e. a matrix having only one column. A scalar is a number (one could say: a matrix of order 1 xl). Concerning notation, matrices are symbolized by bold capital letters and vectors by bold small caps. 320 Appendix Sum Two matrices of the same order can be added (1.4) The sum of the matrices is performed by adding together the elements of the two matrices having the same index i and j. Multiplication by a scalar A matrix can be multiplied equa11y from the right or from the left by a scalar (1.5) which amounts to multiply a11 elements aij by the scalar A. Multiplication of two matrices The product AB of two matrices A and B can only be performed if B has as many rows as A has columns. Let A be of order n x m and B of order m x [. The multiplication of A by B produces a matrix C of order n x [ A . B = [ f aij bjk] = [eik] = C , (1.6) (nxm) (mxl) j=1 (nxl) where i = 1, ... , n and k = 1, ... , [. The product BA of these matrices is only possible if n is equal to [ and the result is then a matrix of order m x m. Transposition The transpose AT of a matrix A of order n x m is obtained by inverting the sequence of the indices, such that the rows of A become the columns of AT (1.7) The transpose of the product of two matrices is equal to the product of the trans posed matrices in reverse order (1.8) EXERCISE 1.1 Let 1 be the vector of dimension n whose elements are a11 equal to 1. Carry out the products 1Tl and 11T . Matrix Algebra 321 EXERCISE 1.2 Ca1culate the matrices resulting tram the products and (1.9) where Z is the n x N matrix oi the data. Square matrix A square matrix has as many rows as columns. Diagonal matrix A diagonal matrix D is a square matrix whose only non zero elements are on the diagonal du o D = ( 0 o ) . (1.10) o o dnn In particular we mention the identity matrix (1.11) which does not modify a matrix of the same order when multiplying it from the right or from the left AI IA=A. (1.12) Orthogonal matrix A square matrix A is orthogonal if it verifies (1.13) Symmetrie matrix A square matrix A is symmetrie if it is equal to its transpose (1.14) 322 Appendix EXAMPLE I.3 An example of a symmetrie matrix is the variance-eovariance matrix V, containing the experimental variances Sii on the diagonal and the experimental eovarianees Sij off the diagonal 1 T V = [Sij] = - (Z - M) (Z - M) , (1.15) n where M is the reetangular n x m matrix of the means (solution of the exercise 1.2), whose eolumn elements are equal to the mean of the variable corresponding to a given eolumn. EXAMPLE 1.4 Another example of a symmetrie matrix is the matrix R ofeorrelations (1.16) where D r1 is a diagonal matrix containing the inverses of the experimental standard deviations Si = .jSii of the variables 1 0 0 y'SU D.-l 0 0 (1.17) 1 0 0 v'SNN Linear independence A set of vectors {al,' .. ,a.n} is said to be linearly independent, if there exists no trivial set of scalars {Xl, ... , xm } such as m Lajxj = O. (1.18) j=l In other words, the linear independence of the columns of a matrix A is acquired, if only the nul vector x = 0 satisfies the equation Ax = O. Rank of a matrix A rectangular matrix can be subdivided into the set of column vectors which make it up. Similarly, we can also consider a set of "row vectors" of this matrix, which are defined as the column vectors of its transpose. The rank of the columns of a matrix is the maximum number of linearly inde pendent column vectors of this matrix. The rank of the rows is defined in an analog manner. It can be shown that the rank of the rows is equal to the rank of the columns. The rank of a rectangular n x m matrix A is thus lower or equal to the smaller of its two dimensions rank(A) ~ min(n, m). (1.19) Matrix Algebra 323 The rank of the matrix A indicates the dimension of the vector spaces 1 M (A) and N(A) spanned by the columns and the rows of A M(A) = {y: y = Ax}, N(A) = {x: x = yT A}, (L20) where x is a vector of dimension m and y is a vector of dimension n. Inverse matrix A square n x n matrix A is singular, if rank ( A) < n, and non singular, if ranke A) = n. If Ais non singular, an inverse matrix A -1 exists, such as (L21) and A is said to be invertible. The inverse Q-l of an orthogonal matrix Q is its transpose QT. Determinant of a matrix The determinant of a square n x n matrix A is n det(A) = Z)_1)N(k1 , ••• ,kn ) rr aiki' (L22) i=1 where the sum is taken over all the permutations (k 1 , ... , kn ) of the integers (1, ... , n) and where N(k1, ... ,kn ) is the number of transpositions of two integers necessary to pass from the starting set (1, ... , n) to a given permutation (k 1, ... , kn ) of this set. In the case of a 2 x 2 matrix there is the well-known formula A = det(A) = ad - bc. (L23) (acd' b) A non-zero determinant indicates that the corresponding matrix is invertible. Trace The trace of a square n x n matrix is the sum of its diagonal elements, n tr(A) Laii' (L24) i=l Ido not mix up: the dimension of a vector and the dimension of a vector space. 324 Appendix Eigenvalues Let A be square n x n. The eharaeteristie equation det(AI - A) = 0 (1.25) has n in general eomplex solutions A ealled the eigenvalues of A. The sum of the eigenvalues is equal to the traee of the matrix n tr(A) = LAp • (1.26) p=l The produet of the eigenvalues is equal to the determinant of the matrix n det(A) = rr Ap • (1.27) p=l If A is symmetrie, all its eigenvalues are real. Eigenvectors Let A be a square matrix and A an eigenvalue of A. Then veetors x and y exist, whieh are not equal to the zero veetor 0, satisfying (AI - A)x = 0, (1.28) l.e. Ax = AX, (1.29) The veetors x are the eigenveetors of the eolumns of A and the y are the eigen veetors of the rows of A. When A is symmetrie, it is not neeessary to distinguish between the eigenveetors of the eolumns and of the rows. EXERCISE I.5 Show that the eigenvaIues of a squared symmetrie matrix A 2 = AA are equaI to the square of the eigenvaIues of A. And that any eigenveetor of A is an eigenveetor of A 2 • Positive definite matrix Definition: asymmetrie n x n matrix A is positive definite, iff (if and only it) for any non zero veetor x the quadratic form xT Ax > O. (1.30) Similarly A is said to be positive semi-definite (non negative definite), iff xT Ax ~ 0 for any vector x. Furthermore, A is said to be indefinite, iff xT Ax > 0 for some x and xT Ax < 0 for other x. Matrix Algebra 325 We notice that this definition is similar to the definition of a positive definite func tion, e.g. the covariance function C(h). We now list three very useful criteria for positive semi-definite matrices. First Criterion: A is positive semi-definite, iff a matrix W exists such as A WTW. In this context W is sometimes written VA. Second Criterion: A is positive semi-definite, iff all its n eigenvalues Ap ~ o. Third Criterion: A is positive semi-definite, iff all its principal minors are non neg ative. A principal minor is the determinant of a principal submatrix of A. A principal submatrix is obtained by leaving out k columns (k = 0,1, ... ,n - 1) and the corre sponding rows which cross them at the diagonal elements. The combinatorial of the principal minors to be checked makes this criterion less interesting for applications with n > 3.