Chapter 2 a Short Review of Matrix Algebra

Home , Idempotent matrix, Spectrum of a matrix

Chapter 2

A short review of matrix algebra

2.1 Vectors and vector spaces

Deﬁnition 2.1.1. A vector a of dimension n is a collection of n elements typically written as ⎛ ⎞ ⎜ a1 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ a2 ⎟ a = ⎜ ⎟ =(ai)n. ⎜ . ⎟ ⎝ . ⎠

an Vectors of length 2 (two-dimensional vectors) can be thought of points in

33 BIOS 2083 Linear Models Abdus S. Wahed the plane (See ﬁgures).

Chapter 2 34 BIOS 2083 Linear Models Abdus S. Wahed Figure 2.1: Vectors in two and three dimensional spaces

(-1.5,2)

(1, 1)

(1, -2)

(2.5, 1.5, 0.95)

(0, 1.5, 0.95)

Chapter 2 35 BIOS 2083 Linear Models Abdus S. Wahed

• A vector with all elements equal to zero is known as a zero vector and is denoted by 0.

• A vector whose elements are stacked vertically is known as column vector whereas a vector whose elements are stacked horizontally will be referred to as row vector. (Unless otherwise mentioned, all vectors will be referred to as column vectors).

• A row vector representation of a column vector is known as its trans-

T pose. We will use⎛ the⎞ notation ‘ ’or‘ ’ to indicate a transpose. For ⎜ a1 ⎟ ⎜ ⎟ ⎜ a2 ⎟ ⎜ ⎟ T instance, if a = ⎜ ⎟ and b =(a1 a2 ... an), then we write b = a ⎜ . ⎟ ⎝ . ⎠

an or a = bT .

• Vectors of same dimension are conformable to algebraic operations such as additions and subtractions. Sum of two or more vectors of dimension n results in another n-dimensional vector with elements as the sum of the corresponding elements of summand vectors. That is,

(ai)n ± (bi)n =(ai ± bi)n.

Chapter 2 36 BIOS 2083 Linear Models Abdus S. Wahed

• Vectors can be multiplied by a scalar.

c(ai)n =(cai)n.

• Product of two vectors of same dimension can be formed when one of them is a row vector and the other is a column⎛ vector.⎞ The result⎛ ⎞ is called ⎜ a1 ⎟ ⎜ b1 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ a2 ⎟ ⎜ b2 ⎟ inner, dot or scalar product.ifa = ⎜ ⎟ and b = ⎜ ⎟,then ⎜ . ⎟ ⎜ . ⎟ ⎝ . ⎠ ⎝ . ⎠

an bn

T a b = a1b1 + a2b2 + ...+ anbn.

Deﬁnition 2.1.2. The length, magnitude, or Euclidean norm of a vector is deﬁned as the square root of the sum of squares of its elements and is denoted by ||.||. For example,

n √ || || || || 2 T a = (ai)n = ai = a a. i=1

• The length of the sum of two or more vectors is less than or equal to the sum of the lengths of each vector. (Cauchy-Schwarz Inequality).

||a + b|| ≤ ||a|| + ||b||

Chapter 2 37 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.1.3. A set of vectors {a1, a2,...,am} is linearly dependent if at least one of them can be written as a linear combination of the others.

In other words, {a1, a2,...,am} are linearly dependent if there exists at least one non-zero cj such that m cjaj =0. (2.1.1) j=1 In other words, for some k,

ak = −(1/ck) cjaj. j=k Deﬁnition 2.1.4. A set of vectors are linearly independent if they are not linearly dependent. That is, in order for (2.1.1) to hold, all cj’s must be equal to zero.

Chapter 2 38 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.1.5. Two vectors a and b are orthogonal if their scalar product is zero. That is, aT b = 0, and we write a ⊥ b.

Deﬁnition 2.1.6. A set of vectors is said to be mutually orthogonal if members of any pair of vectors belonging to the set are orthogonal.

• If vectors are mutually orthogonal then they are linearly independent.

Chapter 2 39 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.1.7. Vector space. A set of vectors which are closed under addition and scalar multiplication is known as a vector space.

Thus if V is a vector space, for any two vectors a and b from V, (i) caa + cbb ∈V, and (ii) caa ∈Vfor any two constants ca and cb.

Deﬁnition 2.1.8. Span. All possible linear combinations of a set of linearly independent vectors form a Span of that set.

Thus if A = {a1, a2,...,am} is a set of m linearly independent vectors, then the span of A is given by

m span(A)= a : a = cjaj , j=1 for some numbers cj,j =1, 2,...,m. Viewed diﬀerently, the set of vectors A generates the vector space span(A) and is referred to as a basis of span(A). Formally,

• Let a1, a2,...,am be a set of m linearly independent n-dimensional vec-

tor in a vector space V that spans V.Thena1, a2,...,am together forms a basis of V and the dimension of a vector space is deﬁned by the number of vectors in its basis. That is, dim(V)=m.

Chapter 2 40 BIOS 2083 Linear Models Abdus S. Wahed

2.2 Matrix

Deﬁnition 2.2.1. A matrix is a rectangular or square arrangement of numbers. A matrix with m rows and n columns is referred to as an m × n (read as ‘m by n’) matrix. An m × n matrix A with (i, j)th element aij is written as ⎡ ⎤ ⎢ a11 a12 ... a1n ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ a21 a22 ... a2n ⎥ A =(aij)m×n = ⎢ ⎥ . ⎢ ··· ··· ··· ⎥ ⎣ ... ⎦

am1 am2 ... amn

If m = n then the matrix is a square matrix.

Deﬁnition 2.2.2. A diagonal matrix is a square matrix with non-zero elements in the diagonal cells and zeros elsewhere.

A diagonal matrix with diagonal elements a1,a2,...,an is written as ⎡ ⎤ ⎢ a1 0 ... 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ 0 a2 ... 0 ⎥ diag(a1,a2,...,an)=⎢ ⎥ . ⎢ ··· ··· ··· ⎥ ⎣ ... ⎦

00... an

Deﬁnition 2.2.3. An n×n diagonal matrix with all diagonal elements equal to 1 is known as identity matrix of order n and is denoted by In.

Chapter 2 41 BIOS 2083 Linear Models Abdus S. Wahed

A similar notation Jmn is sometimes used for an m × n matrix with all elements equal to 1, i.e., ⎡ ⎤ ⎢ 11... 1 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ 11... 1 ⎥ Jmn = ⎢ ⎥ =[1m 1m ... 1m] . ⎢ ··· ··· ··· ⎥ ⎣ ... ⎦ 11... 1

Like vectors, matrices with the same dimensions can be added together and results in another matrix. Any matrix is conformable to multiplication by a scalar. If A =(aij)m×n and B =(bij)m×n,then

1. A ± B =(aij ± bij)m×n, and

2. cA =(caij)m×n.

Deﬁnition 2.2.4. The transpose of a matrix A =(aij)m×n is deﬁned by

T A =(aji)n×m.

• If A = AT ,thenA is symmetric.

• (A + B)T =(AT + BT ).

Chapter 2 42 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.2.5. Matrix product. If A =(aij)m×n and B =(aij)n×p, then

T AB =(cij)m×p,cij = aikbkj = ai bj, k where ai is the ith row (imagine as a vector) of A and bj is the jth column (vector) of B.

• (AB)T = BT AT ,

• (AB)C = A(BC),whenever deﬁned,

• A(B + C)=AB + AC, whenever deﬁned,

• JmnJnp = nJmp.

Chapter 2 43 BIOS 2083 Linear Models Abdus S. Wahed

2.3 Rank, Column Space and Null Space

Deﬁnition 2.3.1. The rank of a matrix A is the number of linearly independent rows or columns of A.Wedenoteitbyrank(A).

• rank(AT )=rank(A).

• An m × n matrix A with with rank m (n)issaidtohavefull row (column) rank.

• If A is a square matrix with n rows and rank(A)

• rank(AB) ≤ min(rank(A),rank(B)).

• rank(AT A)=rank(AAT )=rank(A)=rank(AT ).

Chapter 2 44 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.3.2. Inverse of a square matrix.IfA is a square matrix with n rows and rank(A)=n,thenA is called non-singular and there exists −1 −1 −1 −1 a matrix A such that AA = A A = In. The matrix A is known as the inverse of A.

• A−1 is unique.

• If A and B are invertible and has the same dimension, then

(AB)−1 = B−1A−1.

• (cA)−1 = A−1/c.

• (AT )−1 =(A−1)T .

Chapter 2 45 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.3.3. Column space. The column space of a matrix A is the vector space generated by the columns of A.IfA =(aij)m×n =(a1 a2 ... an, then the column space of A, denoted by C(A)orR(A)isgivenby

n C(A)= a : a = cjaj , j=1 for scalars cj,j =1, 2,...,n.

Alternatively, a ∈C(A) iﬀ there exists a vector c such that

a = Ac.

• What is the dimension of the vectors in C(A)?

• How many vectors will a basis of C(A)have?

• dim(C(A)) =?

• If A = BC,thenC(A) ⊆C(B).

• If C(A) ⊆C(B), then there exist a matrix C such that A = BC.

Example 2.3.1. Find a basis for the column space of the matrix ⎡ ⎤ − − ⎢ 12 1 ⎥ ⎢ ⎥ ⎢ ⎥ A = ⎣ 114⎦ . 022

Chapter 2 46 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.3.4. Null Space.Thenullspaceofanm × n matrix A is de- ﬁned as the vector space consisting of the solution of the system of equations Ax = 0. Null space of A is denoted by N (A) and can be written as

N (A)={x : Ax =0} .

• What is the dimension of the vectors in N (A)?

• How many vectors are there in a basis of N (A)?

• dim(N (A)) = n − rank(A) → Nullity of A.

Chapter 2 47 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.3.5. Orthogonal complements. Two sub spaces V1 and V2 of a vector space V forms orthogonal complements relative to V if every vector ⊥ in V1 is orthogonal to every vector in V2. We write V1 = V2 or equivalently, ⊥ V2 = V1 .

•V1 ∩V2 = {0}.

• If dim(V1)=r,thendim(V2)=n − r,wheren is the dimension of the vectors in the vector space V.

• Every vector a in V can be uniquely decomposed into two components

a1 and a2 such that

a = a1 + a2, a1 ∈V1, a2 ∈V2. (2.3.1)

• If (2.3.1) holds, then

2 2 2 a = a1 + a2 . (2.3.2)

How?

Chapter 2 48 BIOS 2083 Linear Models Abdus S. Wahed

Proof of (2.3.1).

• Existence. Suppose it is not possible. Then a is independent of the

basis vectors of V1 and V2. But that would make the total number of independent vectors in V n + 1. Is that possible?

• Uniqueness. Let two such decompositions are possible, namely,

a = a1 + a2, a1 ∈V1, a2 ∈V2,

and

a = b1 + b2, b1 ∈V1, b2 ∈V2.

Then,

a1 − b1 = b2 − a2.

This implies

a1 = b1 & b2 = a2.(Why?)

Chapter 2 49 BIOS 2083 Linear Models Abdus S. Wahed

Proof of (2.3.2).

• From (2.3.1),

a 2 = aT a

T =(a1 + a2) (a1 + a2)

T T T T = a1 a1 + a1 a2 + a2 a1 + a2 a2

2 2 = a1 + a2 . (2.3.3)

This result is known as Pythagorean theorem.

Chapter 2 50 BIOS 2083 Linear Models Abdus S. Wahed

Figure 2.2: Orthogonal decomposition (direct sum)

2 V1 = {(x, y): x = y İ R }

(3/2, 3/2) (2, 1) = + ( 1/2, -1/2 ) V = {(x, y): x, y İ R2}

2 V2 = {(x, y): x, y İ R ,x+ y = 0}

Chapter 2 51 BIOS 2083 Linear Models Abdus S. Wahed

Theorem 2.3.2. If A is an m×n matrix, and C(A) and N (AT ) respectively denote the column and null space of A and AT ,then

C(A)=N (AT )⊥.

Proof. • dim(C(A)) = rank(A)=rank(AT )=r (say), dim(N (AT )) = m − r.

T • Suppose a1 ∈C(A)anda2 ∈N(A ). Then, there exist a c such that

Ac = a1,

and

T A a2 =0.

Now,

T T T a1 a2 = c A a2

=0. (2.3.4)

Chapter 2 52 BIOS 2083 Linear Models Abdus S. Wahed

⊥ ⊥ • (More on Orthogonality.) If V1 ⊆V2,andV1 and V2 respectively denote their orthogonal complements, then

⊥ ⊥ V2 ⊆V1 .

Chapter 2 53 BIOS 2083 Linear Models Abdus S. Wahed

Proof. Proof of the result on previous page. Suppose a1 ∈V1.Thenwe can write

a1 = A1c1, for some vector c1 and the columns of matrix A1 consisting of the basis vectors of V1. And similarly,

a2 = A2c2, ∀ a2 ∈V2.

In other words,

V1 = C(A1) and

V2 = C(A2).

Since V1 ⊆V2, there exists a matrix B such that A1 = A2B. (See PAGE 39)

⊥ T Now let, a ∈V2 =⇒ a ∈N(A2 ) implying

T A2 a =0.

But

T T T A1 a = B A2 a =0,

T ⊥ providing that a ∈N(A1 )=V2 .

Chapter 2 54 BIOS 2083 Linear Models Abdus S. Wahed

2.4 Trace

The trace of a matrix will become handy when we will talk about the distri- bution of quadratic forms.

Deﬁnition 2.4.1. Trace of a square matrix is the sum of its diagonal elements. Thus, if A =(aij)n×n,then n trace(A)= aii i=1 .

• trace(In)=

• trace(A)=trace(AT )

• trace(A + B)=trace(A)+trace(B)

• trace(AB)=trace(BA) • T 2 n n 2 trace(A A)=trace(A )= i=1 j=1 aij.

Chapter 2 55 BIOS 2083 Linear Models Abdus S. Wahed

2.5 Determinants

Deﬁnition 2.5.1. Determinant. The determinant of a scalar is the scalar itself. The determinants of an n×n matrix A =(aij)m×n is given by a scalar, written as |A|,where, n i+j |A| = aij(−1) |Mij|, j=1 for any ﬁxed i, where, the determinant |Mij| of the matrix Mij is known as the minor of aij and the matrix Mij is obtained by deleting the ith row and jth column of matrix A.

•|A| = |AT | •| | n diag(di,i=1, 2,...,n) = i=1 di. This also holds if the matrix is an upper or lower triangular matrix with

diagonal elements di,i=1, 2,...,n.

Chapter 2 56 BIOS 2083 Linear Models Abdus S. Wahed

•|AB| = |A||B|

•|cA| = cn|A|

• If A is singular (rank(A)

•|A−1| =1/|A|.

• The determinants of block-diagonal (block-triangular) matrices works the way as you would expect. For instance, AC = |A||B|. 0 B

In general AB −1 = |A||D − CA B|. CD

Chapter 2 57 BIOS 2083 Linear Models Abdus S. Wahed

2.6 Eigenvalues and Eigenvectors

Deﬁnition 2.6.1. Eigenvalues and eigen vectors. The eigenvalues (λ) of a square matrix An×n and the corresponding eigenvectors (a) are deﬁned by the set of equations Aa = λa. (2.6.1)

Equation (2.6.1) leads to the polynomial equation

|A − λIn| =0. (2.6.2)

For a given eigenvalue, the corresponding eigenvector is obtained as the solution to the equation (2.6.1). The solutions to equation (2.6.1) constitutes the eigenspace of the matrix A.

Example 2.6.1. Find the eigenvalues and eigenvectors for the matrix ⎡ ⎤ − ⎢ 12 0⎥ ⎢ ⎥ ⎢ ⎥ A = ⎣ 121⎦ . 02−1

Chapter 2 58 BIOS 2083 Linear Models Abdus S. Wahed

Since in this course our focus will be on the eigenvalues of symmetric matrices, hereto forth we state the results on eigenvalues and eigenvectors applied to a symmetric matrix A. Some of the results will, however, hold for general A. If you are interested, please consult a linear algebra book such as Harville’s Matrix algebra from statistics perspective.

Deﬁnition 2.6.2. Spectrum. The spectrum of a matrix A is deﬁned as the set of distinct (real) eigenvalues {λ1,λ2,...,λk} of A.

• The eigenspace L of a matrix A corresponding to an igenvalue λ can be written as

L = N (A − λIn). • n trace(A)= i=1 λi. •| | n A = i=1 λi. •| ± | n ± In A = i=1(1 λi).

• Eigenvectors associated with diﬀerent eigenvalues are mutually orthogonal or can be chosen to be mutually orthogonal and hence linearly independent.

• rank(A) is the number of non-zero λi’s.

Chapter 2 59 BIOS 2083 Linear Models Abdus S. Wahed

The proof of some of these results can be easily obtained through the application of a special theorem called spectral decomposition theorem.

Deﬁnition 2.6.3. Orthogonal Matrix. A matrix An×n is said to be orthogonal if

T T A A = In = AA .

This immediately implies that A−1 = AT .

Theorem 2.6.2. Spectral decomposition. Any symmetric matrix Acan be decomposed as A = BΛBT , where Λ=diag(λ1,...,λn), is the diagonal matrix of eigenvalues and B is an orthogonal matrix having its columns as the eigenvectors of A,namely,

A =[a1a2 ...an],whereaj’s are orthonormal eigenvectors corresponding to the eigenvalues λj,j =1, 2,...,n.

Proof.

Chapter 2 60 BIOS 2083 Linear Models Abdus S. Wahed

Outlineoftheproofofspectraldecompositiontheorem:

• By deﬁnition, B satisﬁes AB = BΛ, (2.6.3)

and

T B B = In.

Then from (2.6.3), A = BΛB−1 = BΛBT .

Spectral decomposition of a symmetric matrix allows one to form ’square root’ of that matrix. If we deﬁne √ √ A = B ΛBT , it is easy to verify that √ √ A A = A.

In general, one can deﬁne

Aα = BΛαBT ,α∈R.

Chapter 2 61 BIOS 2083 Linear Models Abdus S. Wahed

Example 2.6.3. Find a matrix B and the matrix Λ (the diagonal matrix of eigenvalues) such that ⎡ ⎤ 6 −2 A = ⎣ ⎦ = BT ΛB. −29

Chapter 2 62 BIOS 2083 Linear Models Abdus S. Wahed

2.7 Solutions to linear systems of equations

A linear system of m equations in n unknowns is written as

Ax = b, (2.7.1)

where Am×n is a matrix and b is a vector of known constants and x is an unknown vector. The goal usually is to ﬁnd a value (solution) of x such that (2.7.1) is satisﬁed. When b = 0, the system is said to be homogeneous.It is easy to see that homogeneous systems are always consistent, that is, has at least one solution.

• The solution set of a homogeneous system of equation Ax =0formsa vector space and is given by N (A).

• A non-homogeneous system of equations Ax = b is consistent iﬀ

rank(A, b)=rank(A).

– The system of linear equations Ax = b is consistent iﬀ b ∈C(A).

– If A is square and rank(A)=n,thenAx = b has a unique solution given by x = A−1b.

Chapter 2 63 BIOS 2083 Linear Models Abdus S. Wahed

2.7.1 G-inverse

One way to obtain the solutions to a system of equations (2.7.1) is just to transform the augmented matrix (A, b) into a row-reduced-echelon form. However, such forms are not algebraically suitable for further algebraical treatment. Equivalent to the inverse of a non-singular matrix, one can deﬁne an inverse, referred to as generalized inverse or in short g-inverse of any matrix, square or rectangular, singular or non-singular. This generalized inverse helps ﬁnding the solutions of linear equations easier. Theoretical develop- ments based on g-inverse are very powerful for solving problems arising in linear models.

Deﬁnition 2.7.1. G-inverse. The g-inverse of a matrix Am×n is a matrix

Gn×m that satisﬁes the relationship

AGA = A.

Chapter 2 64 BIOS 2083 Linear Models Abdus S. Wahed

The following two lemmas are useful for ﬁnding the g-inverse of a matrix A.

Lemma 2.7.1. Suppose rank(Am×n)=r,andAm×n can be factorized as ⎡ ⎤ ⎣ A11 A12 ⎦ Am×n = A21 A22 such that A11 is of dimension r × r with rank(A11)=r. Then, a g-inverse of A is given by ⎡ ⎤ −1 ⎣ A11 0 ⎦ Gn×m = . 00

Example 2.7.2. Find the g-inverse of the matrix ⎡ ⎤ ⎢ 111 1⎥ ⎢ ⎥ ⎢ − ⎥ A = ⎣ 010 1 ⎦ . 101 2

Chapter 2 65 BIOS 2083 Linear Models Abdus S. Wahed

Suppose you do not have an r × r minor to begin with. What do you do then?

Lemma 2.7.3. Suppose rank(Am×n)=r, and there exists non-singular matrices B and C such that ⎡ ⎤ D 0 BAC = ⎣ ⎦ . 00 where D is a diagonal matrix with rank(D)=r. Then, a g-inverse of A is given by ⎡ ⎤ −1 −1 ⎣ D 0 ⎦ −1 Gn×m = C B . 00

Chapter 2 66 BIOS 2083 Linear Models Abdus S. Wahed

• rank(G) ≥ rank(A).

• G-inverse of a matrix is not necessarily unique. For instance,

– If G is a g-inverse of a symmetric matrix A,thenGAG is also a g-inverse of A.

T – If G is a g-inverse of a symmetric matrix A,thenG1 =(G+G )/2 is also a g-inverse of A.

– The g-inverse of a diagonal matrix D = diag(d1,...,dn)isanother g g g diagonal matrix D = diag(d1,...,dn), where ⎧ ⎨ g 1/di,di =0, di = ⎩ 0,di =0.

Again, as you can see, we concentrate on symmetric matrices as this matrix properties will be applied to mostly symmetric matrices in this course.

Chapter 2 67 BIOS 2083 Linear Models Abdus S. Wahed

Another way of ﬁnding a g-inverse of a symmetric matrix.

Lemma 2.7.4. Let A be an n-dimensional symmetric matrix. Then a g- inverse of A, G is given by

G = QT ΛQ, where Q and Λ bears the same meaning as in spectral decomposition theorem.

2.7.2 Back to the system of equations

Theorem 2.7.5. If Ax = b is a consistent system of linear equations and G be a g-inverse of A,thenGb is a solution to Ax = b.

Proof.

Chapter 2 68 BIOS 2083 Linear Models Abdus S. Wahed

Theorem 2.7.6. x∗ is a solution to the consistent system of linear equation Ax = b iﬀ there exists a vector c such that

x∗ = Gb +(I − GA)c, for some g-inverse G of A.

Proof.

Chapter 2 69 BIOS 2083 Linear Models Abdus S. Wahed

Proof. ProofofTheorem??. If part. For any compatible vector c and for any g-inverse G of A, deﬁne

x∗ = Gb +(I − GA)c.

Then,

Ax∗ = A[Gb +(I − GA)c]=AGb +(A − AGA)c = b +0=b.

Only If part. Suppose x∗ is a solution to the consistent system of linear equation Ax = b. Then

x∗ = Gb +(x∗ − Gb)=Gb +(x∗ − GAx∗)=Gb +(I − GA)c, where c = x∗.

Remark 2.7.1. 1. Any solution to the system of equations Ax = b can be written as a sum of two components: one being a solution by itself and the other being in the null space of A.

2. If one computes one g-inverse of A, then he/she has identiﬁed all possible solutions of Ax = b.

Chapter 2 70 BIOS 2083 Linear Models Abdus S. Wahed

Example 2.7.7. Give a general form of the solutions to the system of equations ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎢ 1210⎥ ⎢ x1 ⎥ ⎢ 5 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ 1111⎥ ⎢ x2 ⎥ ⎢ 3 ⎥ ⎢ ⎥ ⎢ ⎥ = ⎢ ⎥ . ⎢ − ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ 010 1 ⎦ ⎣ x3 ⎦ ⎣ 2 ⎦

1 −11 3 x4 −1

Chapter 2 71 BIOS 2083 Linear Models Abdus S. Wahed

Idempotent matrix and projections

Deﬁnition 2.7.2. Idempotent matrix. A square matrix B is idempotent if B2 = BB = B.

• If B is idempotent, then rank(B)=trace(B).

• If Bn×n is idempotent, then In−B is also idempotent with rank(In−B)= n − trace(B).

• If Bn×n is idempotent with rank(B)=n,thenB = In.

Lemma 2.7.8. If the m × n matrix A has rank r, then the matrix In − GA is idempotent with rank n − r,whereG is a g-inverse of A.

Chapter 2 72 BIOS 2083 Linear Models Abdus S. Wahed

Deﬁnition 2.7.3. Projection. A square matrix Pn×n is a projection onto a vector space V⊆Rn iﬀ all three of the following holds: (a) P is idempotent, (b) ∀x ∈Rn, Px ∈V,and(c)∀x ∈V, Px = x. An idempotent matrix is a projection onto its own column space.

Example 2.7.9. Let the vector space be deﬁned as

2 V = {(v1,v2),v2 = kv1}⊆R, ⎧ ⎫ ⎨ ⎬ t (1 − t)/k for some non-zero real constant k. Consider the matrix P = ⎩ ⎭ kt (1 − t) for any real number t ∈R. Notice that (a) PP = P,

T 2 T (b) For any x =(x1,x2) ∈R, Px =(tx1+(1−t)x2/k, ktx1+(1−t)x2) ∈ V.

T (c) For any x =(x1,x2) =(x1,kx1) ∈V, Px = x. Thus, P is a projection onto the vector space V. Notice that the projection P is not unique as it depends on the coice of t.Considerk =1.ThenV is the linear space representing the line with unit slope passing through⎧ the⎫ origin. ⎨ ⎬ 2 −1 When multiplied by the projection matrix (for t =2)P1 = ⎩ ⎭,any 2 −1 point in the two-dimensional real space produces a point in V. For instance, the point (1,.5) when multiplied by P1 produces (1.5, 1.5) which belongs to

Chapter 2 73 BIOS 2083 Linear Models Abdus S. Wahed Figure 2.3: Projections.

V = {(x,y), x = y}

(.75, .75) (1.5,1.5) P1

P2 (1,1/2)

⎧ ⎫ ⎨ ⎬ .5 .5 V V . But the projection P2 = ⎩ ⎭ projects the point (1,.5) onto at .5 .5 (0.75, 0.75). See ﬁgure.

Chapter 2 74 BIOS 2083 Linear Models Abdus S. Wahed

Back to g-inverse and solution of system of equations

Lemma 2.7.10. If G is a g-inverse of A,thenI − GA is a projection onto N (A).

Proof. Left as an exercise.

Lemma 2.7.11. If G is a g-inverse of A,thenAG is a projection onto C(A).

Proof. Left as an exercise (Done in class).

Lemma 2.7.12. If P and Q are symmetric and both project onto the same space V⊆Rn,thenP = Q.

Proof.

Chapter 2 75 BIOS 2083 Linear Models Abdus S. Wahed

By deﬁnition, for any x ∈Rn, Px ∈V& Qx ∈V. Let

Px = x1 ∈V& Qx = x2 ∈V.

Then,

n (P − Q)x =(x1 − x2), ∀x ∈R . (2.7.2)

Multiplying both sides by PT = P,

T T n P (P − Q)x = P (x1 − x2)=(x1 − x2), ∀x ∈R .

We get,

n P(P − Q)x = P(x1 − x2)=(x1 − x2), ∀x ∈R . (2.7.3)

Subtracting (2.7.2) from (2.7.3) we obtain,

=⇒ [P(P − Q) − (P − Q)]x =0, ∀x ∈Rn,

=⇒ Q = PQ.

Multiplying both sides of (2.7.2) by QT = Q and following similar procedure we can show that P = PQ = Q.

Chapter 2 76 BIOS 2083 Linear Models Abdus S. Wahed

n Lemma 2.7.13. Suppose V1, V2(V1 ⊆V2) are vector spaces in R and P1, ⊥ ⊥ P2,andP1 are symmetric projections onto V1, V2,andV1 respectively. Then,

1. P1P2 = P2P1 = P1. (The smaller projection survives.)

⊥ ⊥ 2. P1 P1 = P1P1 =0.

3. P2 − P1 is a projection matrix. (What does it project onto?)

Proof. See Ravishanker and Dey, Page 62, Result 2.6.7.

Chapter 2 77 BIOS 2083 Linear Models Abdus S. Wahed

2.8 Deﬁniteness

Deﬁnition 2.8.1. Quadratic form. If x is a vector in Rn and A is a matrix in Rn×n, then the scalar xT Ax is known as a quadratic form in x.

The matrix A does not need to be symmetric but any quadratic form xT Ax can be expressed in terms of symmetric matrices, for,

xT Ax =(xT Ax + xT AT x)/2=xT [(A + AT )/2]x.

Thus, without loss of generality, the matrix associated with a quadratic form will be assumed symmetric.

Definition 2.8.2. Non-negative definite/Positive semi-definite. A quadratic form xT Ax and the corresponding matrix A is non-negative definite if xT Ax ≥ 0 for all x ∈Rn.

Definition 2.8.3. Positive definite. A quadratic form xT Ax and the corresponding matrix A is positive definite if xT Ax > 0 for all x ∈Rn, x =0, and xT Ax =0onlywhenx =0.

Chapter 2 78 BIOS 2083 Linear Models Abdus S. Wahed

Properties related to deﬁniteness

1. Positive definite matrices are non-singular. The inverse of a positive definite matrix is also positive definite.

2. A symmetric matrix is positive (non-negative) deﬁnite iﬀ all of its eigenvalues are positive (non-negative).

3. All diagonal elements and hence the trace of a positive deﬁnite matrix are positive.

4. If A is symmetric positive deﬁnite then there exists a nonsingular matrix Q such that A = QQT .

5. A projection matrix is always positive semi-deﬁnite.

6. If A and B are non-negative deﬁnite, then so is A + B.IfoneofA or B is positive deﬁnite, then so is A + B.

Chapter 2 79 BIOS 2083 Linear Models Abdus S. Wahed

2.9 Derivatives with respect to (and of) vectors

Definition 2.9.1. Derivative with respect to a vector. Let f(a)beany scalar function of the vector an×1. Then the derivative of f with respect to a is defined as the vector ⎡ ⎤ δf ⎢ δa1 ⎥ ⎢ ⎥ ⎢ δf ⎥ δf ⎢ δa2 ⎥ = ⎢ ⎥ , δa ⎢ . ⎥ ⎣ . ⎦ δf δan and the derivative with respect to the aT is defined as δf δf T = . δaT δa

The second derivative of f with respect to a is written as the derivative of δf T × each of the elements in δa with respect to a and stacked as rows of n n matrix,. i.e., ⎡ ⎤ δ2f δ2f δ2f 2 a a ... a a ⎢ δa1 δ 1δ 2 δ 1δ n ⎥ ⎢ 2 2 2 ⎥ 2 δ f δ f δ f ⎢ 2 ... ⎥ δ f δ δf ⎢ δa2δa1 δa2 δa2δan ⎥ = = ⎢ ⎥ . δaδaT δaT δa ⎢ . . . . ⎥ ⎣ . . . . ⎦ δ2f δ2f δ2f ... 2 δanδa1 δanδa2 δ an

Chapter 2 80 BIOS 2083 Linear Models Abdus S. Wahed

Example 2.9.1. Derivative of linear and quadratic functions of a vector.

δaT b 1. δb = a.

δbT Ab T 2. δb = Ab + A b.

Derivatives with respect to matrices can be deﬁned in a similar fashion. We will only remind ourselves about one result on matrix derivatives which will become handy when we talk about likelihood inference.

Lemma 2.9.2. If An n is a symmetric non-singular matrix, then, ×

δ ln A 1 | | = A− . δA

2.10 Problems

1. Are the following sets of vectors linearly independent? If not, in each case ﬁnd at least one vectors that are dependent on the others in the set.

T T T (a) v1 = (0, 1, 0), v2 = (0, 0, 1), v3 = ( 1, 0, 0) − − T T (b) v1 = (2, 2, 6), v2 = (1, 1, 1) − T T T (c) v1 = (2, 2, 0, 2), v2 = (2, 0, 1, 1),v3 = (0, 2, 1, 1) − − − Chapter 2 78 BIOS 2083 Linear Models Abdus S. Wahed

2. Show that a set of non-zero mutually orthogonal vectors v1, v2, . . . , vn are linearly independent.

3. Find the determinant and inverse of the matrices

 n n   ×    1 ρ . . . ρ  1 ρ ρ        1 ρ     ρ 1 . . . ρ  (a) ,  ρ 1 ρ  ,      . . .  ρ 1  ......  ρ ρ 1 ρ ρ . . . 1     1 ρ ρ2 . . . ρn   2   1 ρ ρ      n 1   1 ρ     ρ 1 ρ . . . ρ −  (b) ,  ρ 1 ρ  ,      . . .  ρ 1  ......  ρ2 ρ 1 n n 1 n 2 ρ ρ − ρ − ... 1   n n ×  1 ρ 0 ... 0 0         ρ 1 ρ . . . 0 0       1 ρ 0    1 ρ    0 ρ 1 ... 0 0  (c)   ,   ,    ρ 1 ρ   . . . . .  ρ 1  ......    0 ρ 1    0 0 0 ... 1 ρ    0 0 0 . . . ρ 1

Chapter 2 79 BIOS 2083 Linear Models Abdus S. Wahed

4. Find the the rank and a basis for the null space of the matrix    1 2 2 1   −     1 3 1 2   −     1 1 3 0       0 1 1 1   − −  1 2 2 1 −

Chapter 2 80