Chapter 2 a Short Review of Matrix Algebra

Chapter 2

A short review of matrix algebra

2.1 Vectors and vector spaces

Deﬁnition 2.1.1. A vector a of dimension n is a collection of n elements typically written as

 a2  a = = (a ) .  .  i n  .       an      Vectors of length 2 (two-dimensional vectors) can be thought of points in the plane (See ﬁgures). A vector with all elements equal to zero is known as a zero vector and is denoted by 0.

31 BIOS 2083 Linear Models Abdus S. Wahed Figure 2.1: Vectors in two and three dimensional spaces

(-1.5,2)

(1, 1)

(1, -2)

(2.5, 1.5, 0.95)

(0, 1.5, 0.95)

Chapter 2 32 BIOS 2083 Linear Models Abdus S. Wahed

A vector whose elements are stacked vertically is known as column • vector whereas a vector whose elements are stacked horizontally will be referred to as row vector. (Unless otherwise mentioned, all vectors will be referred to as column vectors).

A row vector representation of a column vector is known as its trans- • T pose. We will use the notation ‘′’ or ‘ ’ to indicate a transpose. For

 2  a T instance, if a = and b = (a1 a2 ... a ), then we write b = a  .  n  .       an    or a = bT .  

Vectors of same dimension are conformable to algebraic operations such • as additions and subtractions. Sum of two or more vectors of dimension n results in another n-dimensional vector with elements as the sum of the corresponding elements of summand vectors. That is,

(ai)n (bi)n = (ai bi)n. ± ±

Vectors can be multiplied by a scalar. •

c(ai)n = (cai)n.

Chapter 2 33 BIOS 2083 Linear Models Abdus S. Wahed

Product of two vectors of same dimension can be formed when one of • them is a row vector and the other is a column vector. The result is called a1 b1

 a2   b2  inner, dot or scalar product. if a = and b = , then  .   .   .   .           an   bn          T a b = a1b1 + a2b2 + ... + anbn.

Deﬁnition 2.1.2. The length, magnitude, or Euclidean norm of a vector is deﬁned as the square root of the sum of squares of its elements and is denoted by . . For example, || || n 2 T a = (ai)n = a = √a a. || || || || v i u i=1 uX t The length of the sum of two or more vectors is less than or equal to the • sum of the lengths of each vector. (Cauchy-Schwarz Inequality).

a + b a + b || || ≤ || || || ||

Chapter 2 34 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.1.3. A set of vectors a1, a2,..., am is linearly dependent { } if at least one of them can be written as a linear combination of the others.

In other words, a1, a2,..., am are linearly dependent if there exists at { } least one non-zero cj such that

cjaj = 0. (2.1.1) j=1 X In other words, for some k,

ak = (1/ck) cjaj. − j=k X6 Deﬁnition 2.1.4. A set of vectors are linearly independent if they are not linearly dependent. That is, in order for (2.1.1) to hold, all cj’s must be equal to zero.

Chapter 2 35 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.1.5. Two vectors a and b are orthogonal if their scalar product is zero. That is, aT b = 0, and we write a b. ⊥ Deﬁnition 2.1.6. A set of vectors is said to be mutually orthogonal if members of any pair of vectors belonging to the set are orthogonal.

If vectors are mutually orthogonal then they are linearly independent. •

Chapter 2 36 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.1.7. Vector space. A set of vectors which are closed under addition and scalar multiplication is known as a vector space.

Thus if is a vector space, for any two vectors a and b from , (i) V V caa + cbb , and (ii) caa for any two constants ca and cb. ∈V ∈V

Deﬁnition 2.1.8. Span. All possible linear combinations of a set of linearly independent vectors form a Span of that set.

Thus if A = a1, a2,..., am is a set of m linearly independent vectors, { } then the span of A is given by

span(A) = a : a = cjaj , ( j=1 ) X for some numbers cj,j = 1, 2,...,m. Viewed diﬀerently, the set of vectors A generates the vector space span(A) and is referred to as a basis of span(A). Formally,

Let a1, a2,..., am be a set of m linearly independent n-dimensional vec- • tor in a vector space that spans . Then a1, a2,..., am together forms V V a basis of and the dimension of a vector space is deﬁned by the number V of vectors in its basis. That is, dim( ) = m. V Chapter 2 37 BIOS 2083 Linear Models Abdus S. Wahed

2.2 Matrix

Deﬁnition 2.2.1. A matrix is a rectangular or square arrangement of numbers. A matrix with m rows and n columns is referred to as an m n (read × as ‘m by n’) matrix. An m n matrix A with (i, j)th element aij is written × as

a11 a12 ... a1n

 a21 a22 ... a2n  A = (aij)m n = . ×    ...   ··· ··· ···     am1 am2 ... amn      If m = n then the matrix is a square matrix.

Deﬁnition 2.2.2. A diagonal matrix is a square matrix with non-zero elements in the diagonal cells and zeros elsewhere.

A diagonal matrix with diagonal elements a1, a2,...,an is written as

a1 0 ... 0

 0 a2 ... 0  diag(a1, a2,...,a ) = . n    ...   ··· ··· ···     0 0 ... an      Deﬁnition 2.2.3. An n n diagonal matrix with all diagonal elements equal × to 1 is known as identity matrix of order n and is denoted by In.

Chapter 2 38 BIOS 2083 Linear Models Abdus S. Wahed

A similar notation Jmn is sometimes used for an m n matrix with all × elements equal to 1, i.e.,

1 1 ... 1  1 1 ... 1  J = = [1 1 ... 1 ] . mn   m m m  ...   ··· ··· ···     1 1 ... 1      Like vectors, matrices with the same dimensions can be added together and results in another matrix. Any matrix is conformable to multiplication by a scalar. If A = (aij)m n and B = (bij)m n, then × ×

1. A B = (aij bij)m n, and ± ± ×

2. cA = (caij)m n. ×

Deﬁnition 2.2.4. The transpose of a matrix A = (aij)m n is deﬁned by ×

T A = (aji)n m. ×

If A = AT , then A is symmetric. • (A + B)T = (AT + BT ). •

Chapter 2 39 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.2.5. Matrix product. If A = (aij)m n and B = (aij)n p, × × then T AB = (cij)m p, cij = aikbkj = ai bj, × k X where ai is the ith row (imagine as a vector) of A and bj is the jth column (vector) of B.

(AB)T = BT AT , • (AB)C = A(BC),whenever deﬁned, • A(B + C) = AB + AC, whenever deﬁned, •

JmnJnp = nJmp. •

Chapter 2 40 BIOS 2083 Linear Models Abdus S. Wahed

2.3 Rank, Column Space and Null Space

Deﬁnition 2.3.1. The rank of a matrix A is the number of linearly independent rows or columns of A. We denote it by rank(A).

rank(AT ) = rank(A). • An m n matrix A with with rank m (n) is said to have full row • × (column) rank.

If A is a square matrix with n rows and rank(A)

rank(AB) min(rank(A),rank(B)). • ≤ rank(AT A) = rank(AAT ) = rank(A) = rank(AT ). •

Chapter 2 41 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.3.2. Inverse of a square matrix. If A is a square matrix with n rows and rank(A) = n, then A is called non-singular and there exists 1 1 1 1 a matrix A− such that AA− = A− A = In. The matrix A− is known as the inverse of A.

1 A− is unique. • If A and B are invertible and has the same dimension, then •

1 1 1 (AB)− = B− A− .

1 1 (cA)− = A− /c. • T 1 1 T (A )− = (A− ) . •

Chapter 2 42 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.3.3. Column space. The column space of a matrix A is the vector space generated by the columns of A. If A = (aij)m n = (a1 a2 ... an, × then the column space of A, denoted by (A) or (A) is given by C R n

(A) = a : a = cjaj , C ( j=1 ) X for scalars cj,j = 1, 2,...,n.

Alternatively, a (A) iﬀ there exists a vector c such that ∈ C a = Ac.

What is the dimension of the vectors in (A)? • C How many vectors will a basis of (A) have? • C dim( (A)) =? • C If A = BC, then (A) (B). • C ⊆ C If (A) (B), then there exist a matrix C such that A = BC. • C ⊆ C Example 2.3.1. Find a basis for the column space of the matrix

1 2 1 − − A =  1 1 4  .    0 2 2     

Chapter 2 43 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.3.4. Null Space. The null space of an m n matrix A is de- × ﬁned as the vector space consisting of the solution of the system of equations Ax = 0. Null space of A is denoted by (A) and can be written as N (A) = x : Ax = 0 . N { }

What is the dimension of the vectors in (A)? • N How many vectors are there in a basis of (A)? • N dim( (A)) = n rank(A) Nullity of A. • N − →

Chapter 2 44 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.3.5. Orthogonal complements. Two sub spaces 1 and 2 V V of a vector space forms orthogonal complements relative to if every vector V V in 1 is orthogonal to every vector in 2. We write 1 = 2⊥ or equivalently, V V V V 2 = 1⊥. V V

1 2 = 0 . • V ∩V { }

If dim( 1) = r, then dim( 2) = n r, where n is the dimension of the • V V − vectors in the vector space . V Every vector a in can be uniquely decomposed into two components • V a1 and a2 such that

a = a1 + a2, a1 1, a2 2. (2.3.1) ∈V ∈V

If (2.3.1) holds, then •

2 2 2 a = a1 + a2 . (2.3.2) k k k k k k

How?

Chapter 2 45 BIOS 2083 Linear Models Abdus S. Wahed

Proof of (2.3.1).

Existence. Suppose it is not possible. Then a is independent of the • basis vectors of 1 and 2. But that would make the total number of V V independent vectors in n + 1. Is that possible? V Uniqueness. Let two such decompositions are possible, namely, •

a = a1 + a2, a1 1, a2 2, ∈V ∈V and

a = b1 + b2, b1 1, b2 2. ∈V ∈V Then,

a1 b1 = b2 a2. − − This implies

a1 = b1 & b2 = a2.(Why?)

Chapter 2 46 BIOS 2083 Linear Models Abdus S. Wahed

Proof of (2.3.2).

From (2.3.1), • a 2 = aT a k k T = (a1 + a2) (a1 + a2)

T T T T = a1 a1 + a1 a2 + a2 a1 + a2 a2

2 2 = a1 + a2 . (2.3.3) k k k k

This result is known as Pythagorean theorem.

Chapter 2 47 BIOS 2083 Linear Models Abdus S. Wahed

Figure 2.2: Orthogonal decomposition (direct sum)

V 2 1 = {(x, y): x = y İ R }

(3/2, 3/2) (2, 1) = + ( 1/2, -1/2 ) V = {(x, y): x, y İ R2}

V 2 2 = {(x, y): x, y İ R ,x+ y = 0}

Chapter 2 48 BIOS 2083 Linear Models Abdus S. Wahed

Theorem 2.3.2. If A is an m n matrix, and (A) and (AT ) respectively × C N denote the column and null space of A and AT , then

T (A) = (A )⊥. C N Proof. dim( (A)) = rank(A) = rank(AT ) = r (say), dim( (AT )) = • C N m r. − T Suppose a1 (A) and a2 (A ). Then, there exist a c such that • ∈ C ∈N

Ac = a1,

and T A a2 = 0.

Now,

T T T a1 a2 = c A a2

= 0. (2.3.4)

Chapter 2 49 BIOS 2083 Linear Models Abdus S. Wahed

(More on Orthogonality.) If 1 2, and 1⊥ and 2⊥ respectively denote • V ⊆V V V their orthogonal complements, then

2⊥ 1⊥. V ⊆V

Chapter 2 50 BIOS 2083 Linear Models Abdus S. Wahed

Proof. Proof of the result on previous page. Suppose a1 1. Then we ∈V can write

a1 = A1c1, for some vector c1 and the columns of matrix A1 consisting of the basis vectors of 1. And similarly, V

a2 = A2c2, a2 2. ∀ ∈V In other words,

1 = (A1) V C and

2 = (A2). V C

Since 1 2, there exists a matrix B such that A1 = A2B. (See PAGE 39) V ⊆V T Now let, a 2⊥ = a (A2 ) implying ∈V ⇒ ∈N

T A2 a = 0.

But T T T A1 a = B A2 a = 0,

T providing that a (A1 ) = 2⊥. ∈N V

Chapter 2 51 BIOS 2083 Linear Models Abdus S. Wahed

2.4 Trace

The trace of a matrix will become handy when we will talk about the distri- bution of quadratic forms.

Deﬁnition 2.4.1. Trace of a square matrix is the sum of its diagonal elements. Thus, if A = (aij)n n, then × n

trace(A) = aii i=1 X .

trace(In) = • trace(A) = trace(AT ) • trace(A + B) = trace(A) + trace(B) • trace(AB) = trace(BA) • trace(AT A) = trace(A2) = n n a2 . • i=1 j=1 ij P P

Chapter 2 52 BIOS 2083 Linear Models Abdus S. Wahed

2.5 Determinants

Deﬁnition 2.5.1. Determinant. The determinant of a scalar is the scalar itself. The determinants of an n n matrix A = (aij)m n is given by a scalar, × × written as A , where, | | n i+j A = aij( 1) Mij , | | − | | j=1 X for any ﬁxed i, where, the determinant Mij of the matrix Mij is known as | | the minor of aij and the matrix Mij is obtained by deleting the ith row and jth column of matrix A.

diagonal elements di, i = 1, 2,...,n.

Chapter 2 53 BIOS 2083 Linear Models Abdus S. Wahed

AB = A B • | | | || | cA = cn A • | | | | If A is singular (rank(A)

A C = A B . | || | 0 B

In general A B 1 = A D CA− B . | || − | C D

Chapter 2 54 BIOS 2083 Linear Models Abdus S. Wahed

2.6 Eigenvalues and Eigenvectors

Deﬁnition 2.6.1. Eigenvalues and eigen vectors. The eigenvalues (λ) of a square matrix An n and the corresponding eigenvectors (a) are deﬁned × by the set of equations Aa = λa. (2.6.1)

Equation (2.6.1) leads to the polynomial equation

A λIn = 0. (2.6.2) | − | For a given eigenvalue, the corresponding eigenvector is obtained as the solution to the equation (2.6.1). The solutions to equation (2.6.1) constitutes the eigenspace of the matrix A.

Example 2.6.1. Find the eigenvalues and eigenvectors for the matrix

1 2 0 − A =  1 2 1  .    0 2 1   −   

Chapter 2 55 BIOS 2083 Linear Models Abdus S. Wahed

Since in this course our focus will be on the eigenvalues of symmetric matrices, hereto forth we state the results on eigenvalues and eigenvectors applied to a symmetric matrix A. Some of the results will, however, hold for general A. If you are interested, please consult a linear algebra book such as Harville’s Matrix algebra from statistics perspective.

Deﬁnition 2.6.2. Spectrum. The spectrum of a matrix A is deﬁned as the set of distinct (real) eigenvalues λ1, λ2,...,λk of A. { }

The eigenspace of a matrix A corresponding to an igenvalue λ can be • L written as

= (A λIn). L N − n trace(A) = λi. • i=1 n P A = λi. • | | i=1 Q n In A = (1 λi). • | ± | i=1 ± Q Eigenvectors associated with diﬀerent eigenvalues are mutually orthog- • onal or can be chosen to be mutually orthogonal and hence linearly independent.

rank(A) is the number of non-zero λi’s. •

Chapter 2 56 BIOS 2083 Linear Models Abdus S. Wahed

The proof of some of these results can be easily obtained through the application of a special theorem called spectral decomposition theorem.

Deﬁnition 2.6.3. Orthogonal Matrix. A matrix An n is said to be or- × thogonal if T T A A = In = AA .

1 T This immediately implies that A− = A .

Theorem 2.6.2. Spectral decomposition. Any symmetric matrix Acan be decomposed as A = BΛBT , where Λ = diag(λ1,...,λn), is the diagonal matrix of eigenvalues and B is an orthogonal matrix having its columns as the eigenvectors of A, namely,

A = [a1a2 ... an], where aj’s are orthonormal eigenvectors corresponding to the eigenvalues λj,j = 1, 2,...,n.

Proof.

Chapter 2 57 BIOS 2083 Linear Models Abdus S. Wahed

Outline of the proof of spectral decomposition theorem:

By deﬁnition, B satisﬁes • AB = BΛ, (2.6.3)

and T B B = In.

Then from (2.6.3), 1 T A = BΛB− = BΛB .

Spectral decomposition of a symmetric matrix allows one to form ’square root’ of that matrix. If we deﬁne

√A = B√ΛBT , it is easy to verify that √A√A = A.

In general, one can deﬁne

Aα = BΛαBT ,α . ∈R

Chapter 2 58 BIOS 2083 Linear Models Abdus S. Wahed

Example 2.6.3. Find a matrix B and the matrix Λ (the diagonal matrix of eigenvalues) such that

6 2 A = − = BT ΛB.  2 9  −  

Chapter 2 59 BIOS 2083 Linear Models Abdus S. Wahed

2.7 Solutions to linear systems of equations

A linear system of m equations in n unknowns is written as

Ax = b, (2.7.1) where Am n is a matrix and b is a vector of known constants and x is an × unknown vector. The goal usually is to ﬁnd a value (solution) of x such that (2.7.1) is satisﬁed. When b = 0, the system is said to be homogeneous. It is easy to see that homogeneous systems are always consistent, that is, has at least one solution.

The solution set of a homogeneous system of equation Ax = 0 forms a • vector space and is given by (A). N A non-homogeneous system of equations Ax = b is consistent iﬀ • rank(A, b) = rank(A).

– The system of linear equations Ax = b is consistent iﬀ b (A). ∈ C – If A is square and rank(A) = n, then Ax = b has a unique solution 1 given by x = A− b.

Chapter 2 60 BIOS 2083 Linear Models Abdus S. Wahed

2.7.1 G-inverse

One way to obtain the solutions to a system of equations (2.7.1) is just to transform the augmented matrix (A, b) into a row-reduced-echelon form. However, such forms are not algebraically suitable for further algebraical treatment. Equivalent to the inverse of a non-singular matrix, one can deﬁne an inverse, referred to as generalized inverse or in short g-inverse of any matrix, square or rectangular, singular or non-singular. This generalized inverse helps ﬁnding the solutions of linear equations easier. Theoretical develop- ments based on g-inverse are very powerful for solving problems arising in linear models.

Deﬁnition 2.7.1. G-inverse. The g-inverse of a matrix Am n is a matrix × Gn m that satisﬁes the relationship ×

AGA = A.

Chapter 2 61 BIOS 2083 Linear Models Abdus S. Wahed

The following two lemmas are useful for ﬁnding the g-inverse of a matrix A.

Lemma 2.7.1. Suppose rank(Am n) = r, and Am n can be factorized as × ×

A11 A12 Am n = ×  A21 A22    such that A11 is of dimension r r with rank(A11) = r. Then, a g-inverse × of A is given by 1 A11− 0 Gn m = . ×  0 0    Example 2.7.2. Find the g-inverse of the matrix

111 1 A =  0 1 0 1  . −    101 2     

Chapter 2 62 BIOS 2083 Linear Models Abdus S. Wahed

Suppose you do not have an r r minor to begin with. What do you do × then?

Lemma 2.7.3. Suppose rank(Am n) = r, and there exists non-singular ma- × trices B and C such that

D 0 BAC = .  0 0    where D is a diagonal matrix with rank(D) = r. Then, a g-inverse of A is given by 1 1 D− 0 1 Gn m = C− B− . ×  0 0   

Chapter 2 63 BIOS 2083 Linear Models Abdus S. Wahed

rank(G) rank(A). • ≥ G-inverse of a matrix is not necessarily unique. For instance, • – If G is a g-inverse of a symmetric matrix A, then GAG is also a g-inverse of A.

T – If G is a g-inverse of a symmetric matrix A, then G1 = (G+G )/2 is also a g-inverse of A.

– The g-inverse of a diagonal matrix D = diag(d1, . . . , dn) is another g g g diagonal matrix D = diag(d1, . . . , dn), where

1/di, di = 0, dg = 6 i   0, di = 0. Again, as you can see, we concentrate on symmetric matrices as this matrix properties will be applied to mostly symmetric matrices in this course.

Chapter 2 64 BIOS 2083 Linear Models Abdus S. Wahed

Another way of ﬁnding a g-inverse of a symmetric matrix.

Lemma 2.7.4. Let A be an n-dimensional symmetric matrix. Then a g- inverse of A, G is given by

G = QT ΛQ, where Q and Λ bears the same meaning as in spectral decomposition theorem.

2.7.2 Back to the system of equations

Theorem 2.7.5. If Ax = b is a consistent system of linear equations and G be a g-inverse of A, then Gb is a solution to Ax = b.

Proof.

Chapter 2 65 BIOS 2083 Linear Models Abdus S. Wahed

Theorem 2.7.6. x∗ is a solution to the consistent system of linear equation Ax = b iﬀ there exists a vector c such that

x∗ = Gb + (I GA)c, − for some g-inverse G of A.

Proof. If part. For any compatible vector c and for any g-inverse G of A, deﬁne

x∗ = Gb + (I GA)c. − Then,

Ax∗ = A[Gb + (I GA)c] = AGb + (A AGA)c = b + 0 = b. − − Only If part.

Suppose x∗ is a solution to the consistent system of linear equation Ax = b. Then

x∗ = Gb + (x∗ Gb) = Gb + (x∗ GAx∗) = Gb + (I GA)c, − − − where c = x∗.

Remark 2.7.1. 1. Any solution to the system of equations Ax = b can be written as a sum of two components: one being a solution by itself and the other being in the null space of A.

Chapter 2 66 BIOS 2083 Linear Models Abdus S. Wahed

2. If one computes one g-inverse of A, then he/she has identiﬁed all possible solutions of Ax = b.

Chapter 2 67 BIOS 2083 Linear Models Abdus S. Wahed

Example 2.7.7. Give a general form of the solutions to the system of equations 1 2 1 0 x1 5

 1 1 1 1   x2   3  = .        0 1 0 1   x3   2   −             1 1 1 3   x4   1   −     −       

Chapter 2 68 BIOS 2083 Linear Models Abdus S. Wahed

Idempotent matrix and projections

Deﬁnition 2.7.2. Idempotent matrix. A square matrix B is idempotent if B2 = BB = B.

If B is idempotent, then rank(B) = trace(B). •

If Bn n is idempotent, then In B is also idempotent with rank(In B) = • × − − n trace(B). −

If Bn n is idempotent with rank(B) = n, then B = In. • ×

Lemma 2.7.8. If the m n matrix A has rank r, then the matrix In GA × − is idempotent with rank n r, where G is a g-inverse of A. −

Chapter 2 69 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.7.3. Projection. A square matrix Pn n is a projection onto a × vector space n iﬀ all three of the following holds: (a) P is idempotent, V⊆R (b) x n, Px , and (c) x , Px = x. An idempotent matrix is a ∀ ∈R ∈V ∀ ∈V projection onto its own column space.

Example 2.7.9. Let the vector space be deﬁned as

2 = (v1,v2),v2 = kv1 , V { }⊆R t (1 t)/k for some non-zero real constant k. Consider the matrix P = −  kt (1 t)   −  for any real number t . Notice that ∈R   (a) PP = P,

T 2 T (b) For any x = (x1,x2) , Px = (tx1+(1 t)x2/k,ktx1+(1 t)x2) ∈R − − ∈ . V T (c) For any x = (x1,x2) = (x1,kx1) , Px = x. ∈V Thus, P is a projection onto the vector space . Notice that the projection V P is not unique as it depends on the coice of t. Consider k = 1. Then is the V linear space representing the line with unit slope passing through the origin. 2 1 When multiplied by the projection matrix (for t = 2) P1 = − , any  2 1   −  point in the two-dimensional real space produces a point in . For instance, V  the point (1,.5) when multiplied by P1 produces (1.5, 1.5) which belongs to

Chapter 2 70 BIOS 2083 Linear Models Abdus S. Wahed Figure 2.3: Projections.

V = {(x,y), x = y}

(.75, .75) (1.5,1.5) P1

P2 (1,1/2)

.5 .5 . But the projection P2 = projects the point (1,.5) onto at V   V  .5 .5  (0.75, 0.75). See ﬁgure.  

Chapter 2 71 BIOS 2083 Linear Models Abdus S. Wahed

Back to g-inverse and solution of system of equations

Lemma 2.7.10. If G is a g-inverse of A, then I GA is a projection onto − (A). N Proof. Left as an exercise.

Lemma 2.7.11. If G is a g-inverse of A, then AG is a projection onto (A). C Proof. Left as an exercise (Done in class).

Lemma 2.7.12. If P and Q are symmetric and both project onto the same space n, then P = Q. V⊆R Proof.

Chapter 2 72 BIOS 2083 Linear Models Abdus S. Wahed

By deﬁnition, for any x n, Px & Qx . Let ∈R ∈V ∈V

Px = x1 & Qx = x2 . ∈V ∈V Then, n (P Q)x = (x1 x2), x . (2.7.2) − − ∀ ∈R Multiplying both sides by PT = P,

T T n P (P Q)x = P (x1 x2) = (x1 x2), x . − − − ∀ ∈R We get, n P(P Q)x = P(x1 x2) = (x1 x2), x . (2.7.3) − − − ∀ ∈R Subtracting (2.7.2) from (2.7.3) we obtain,

= [P(P Q) (P Q)]x = 0, x n, ⇒ − − − ∀ ∈R = Q = PQ. ⇒ Multiplying both sides of (2.7.2) by QT = Q and following similar procedure we can show that P = PQ = Q.

Chapter 2 73 BIOS 2083 Linear Models Abdus S. Wahed

n Lemma 2.7.13. Suppose 1, 2( 1 2) are vector spaces in and P1, V V V ⊆ V R P2, and P1⊥ are symmetric projections onto 1, 2, and 1⊥ respectively. V V V Then,

1. P1P2 = P2P1 = P1. (The smaller projection survives.)

2. P1⊥P1 = P1P1⊥ = 0. (If the spaces are orthogonal complements, then the projection matrices are orthogonal.)

3. P2 P1 is a projection matrix. (What does it project onto?) − Proof. See Ravishanker and Dey, Page 62, Result 2.6.7.

Chapter 2 74 BIOS 2083 Linear Models Abdus S. Wahed

2.8 Deﬁniteness

Deﬁnition 2.8.1. Quadratic form. If x is a vector in n and A is a matrix R n n T in × , then the scalar x Ax is known as a quadratic form in x. R The matrix A does not need to be symmetric but any quadratic form xT Ax can be expressed in terms of symmetric matrices, for,

xT Ax = (xT Ax + xT AT x)/2 = xT [(A + AT )/2]x.

Thus, without loss of generality, the matrix associated with a quadratic form will be assumed symmetric.

Definition 2.8.2. Non-negative definite. A quadratic form xT Ax and the corresponding matrix A is non-negative definite if xT Ax 0 for all ≥ x n. ∈R Definition 2.8.3. Positive definite. A quadratic form xT Ax and the corresponding matrix A is positive definite if xT Ax > 0 for all x n, x = 0, ∈ R 6 and xT Ax = 0 only when x = 0.

Definition 2.8.4. Positive semi-definite. A quadratic form xT Ax and the corresponding matrix A is positive semi-definite if xT Ax 0 for all x n ≥ ∈R and xT Ax = 0 for some x n, x = 0. ∈R 6

Chapter 2 75 BIOS 2083 Linear Models Abdus S. Wahed

Properties related to deﬁniteness

1. Positive definite matrices are non-singular. The inverse of a positive definite matrix is also positive definite.

2. A symmetric matrix is positive (non-negative) deﬁnite iﬀ all of its eigenvalues are positive (non-negative).

3. All diagonal elements and hence the trace of a positive deﬁnite matrix are positive.

4. If A is symmetric positive deﬁnite then there exists a nonsingular matrix Q such that A = QQT .

5. A projection matrix is always positive semi-deﬁnite.

6. If A and B are non-negative deﬁnite, then so is A + B. If one of A or B is positive deﬁnite, then so is A + B.

Chapter 2 76 BIOS 2083 Linear Models Abdus S. Wahed

2.9 Derivatives with respect to (and of) vectors

Definition 2.9.1. Derivative with respect to a vector. Let f(a) be any scalar function of the vector an 1. Then the derivative of f with respect to × a is defined as the vector δf δa1  δf  δf a = δ 2 , δa  .   .     δf   a   δ n    and the derivative with respect to the aT is defined as

δf δf T = . δaT δa The second derivative of f with respect to a is written as the derivative of each of the elements in δf with respect to aT and stacked as rows of n n δa × matrix,. i.e.,

δ2f δ2f δ2f 2 ... δa1 δa1δa2 δa1δan 2 2 2 2  δ f δ f δ f  δ f δ δf a a a2 ... a a = = δ 2δ 1 δ 2 δ 2δ n . δaδaT δaT δa  . . . .   . . . .     δ2f δ2f δ2f   a a a a ... 2a   δ nδ 1 δ nδ 2 δ n   

Chapter 2 77 BIOS 2083 Linear Models Abdus S. Wahed

Example 2.9.1. Derivative of linear and quadratic functions of a vector.

δaT b 1. δb = a.

δbT Ab T 2. δb = Ab + A b.

Derivatives with respect to matrices can be deﬁned in a similar fashion. We will only remind ourselves about one result on matrix derivatives which will become handy when we talk about likelihood inference.

Lemma 2.9.2. If An n is a symmetric non-singular matrix, then, ×

δ ln A 1 | | = A− . δA

2.10 Problems

1. Are the following sets of vectors linearly independent? If not, in each case ﬁnd at least one vectors that are dependent on the others in the set.

T T T (a) v1 = (0, 1, 0), v2 = (0, 0, 1), v3 = ( 1, 0, 0) − − T T (b) v1 = (2, 2, 6), v2 = (1, 1, 1) − T T T (c) v1 = (2, 2, 0, 2), v2 = (2, 0, 1, 1),v3 = (0, 2, 1, 1) − − − Chapter 2 78 BIOS 2083 Linear Models Abdus S. Wahed

2. Show that a set of non-zero mutually orthogonal vectors v1,v2,...,vn are linearly independent.

3. Find the determinant and inverse of the matrices

n n 1 ρ . . . ρ × 1 ρ ρ 1 ρ  ρ 1 . . . ρ  (a) ,  ρ 1 ρ  ,    . . .  ρ 1  ......         ρ ρ 1       ρ ρ ... 1        1 ρ ρ2 . . . ρn 1 ρ ρ2 n 1 1 ρ  ρ 1 ρ . . . ρ −  (b) ,  ρ 1 ρ  ,    . . .  ρ 1  ......   ρ2 ρ 1         n n 1 n 2     ρ ρ − ρ − ... 1        n n × 1 ρ 0 ... 0 0   ρ 1 ρ ... 0 0 1 ρ 0   1 ρ  0 ρ 1 ... 0 0      (c) , ρ 1 ρ ,      . . . . .  ρ 1    ......   0 ρ 1             0 0 0 ... 1 ρ       0 0 0 . . . ρ 1      Chapter 2 79 BIOS 2083 Linear Models Abdus S. Wahed

4. Find the the rank and a basis for the null space of the matrix

1 2 2 1 −   1 3 1 2  −   11 3 0         0 1 1 1   − −     1 2 2 1   −   

Chapter 2 80