Chapter 2 A short review of matrix algebra 2.1 Vectors and vector spaces Definition 2.1.1. A vector a of dimension n is a collection of n elements typically written as ⎛ ⎞ ⎜ a1 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ a2 ⎟ a = ⎜ ⎟ =(ai)n. ⎜ . ⎟ ⎝ . ⎠ an Vectors of length 2 (two-dimensional vectors) can be thought of points in 33 BIOS 2083 Linear Models Abdus S. Wahed the plane (See figures). Chapter 2 34 BIOS 2083 Linear Models Abdus S. Wahed Figure 2.1: Vectors in two and three dimensional spaces (-1.5,2) (1, 1) (1, -2) x1 (2.5, 1.5, 0.95) x2 (0, 1.5, 0.95) x3 Chapter 2 35 BIOS 2083 Linear Models Abdus S. Wahed • A vector with all elements equal to zero is known as a zero vector and is denoted by 0. • A vector whose elements are stacked vertically is known as column vector whereas a vector whose elements are stacked horizontally will be referred to as row vector. (Unless otherwise mentioned, all vectors will be referred to as column vectors). • A row vector representation of a column vector is known as its trans- T pose. We will use⎛ the⎞ notation ‘ ’or‘ ’ to indicate a transpose. For ⎜ a1 ⎟ ⎜ ⎟ ⎜ a2 ⎟ ⎜ ⎟ T instance, if a = ⎜ ⎟ and b =(a1 a2 ... an), then we write b = a ⎜ . ⎟ ⎝ . ⎠ an or a = bT . • Vectors of same dimension are conformable to algebraic operations such as additions and subtractions. Sum of two or more vectors of dimension n results in another n-dimensional vector with elements as the sum of the corresponding elements of summand vectors. That is, (ai)n ± (bi)n =(ai ± bi)n. Chapter 2 36 BIOS 2083 Linear Models Abdus S. Wahed • Vectors can be multiplied by a scalar. c(ai)n =(cai)n. • Product of two vectors of same dimension can be formed when one of them is a row vector and the other is a column⎛ vector.⎞ The result⎛ ⎞ is called ⎜ a1 ⎟ ⎜ b1 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ a2 ⎟ ⎜ b2 ⎟ inner, dot or scalar product.ifa = ⎜ ⎟ and b = ⎜ ⎟,then ⎜ . ⎟ ⎜ . ⎟ ⎝ . ⎠ ⎝ . ⎠ an bn T a b = a1b1 + a2b2 + ...+ anbn. Definition 2.1.2. The length, magnitude, or Euclidean norm of a vec- tor is defined as the square root of the sum of squares of its elements and is denoted by ||.||. For example, n √ || || || || 2 T a = (ai)n = ai = a a. i=1 • The length of the sum of two or more vectors is less than or equal to the sum of the lengths of each vector. (Cauchy-Schwarz Inequality). ||a + b|| ≤ ||a|| + ||b|| Chapter 2 37 BIOS 2083 Linear Models Abdus S. Wahed Definition 2.1.3. A set of vectors {a1, a2,...,am} is linearly dependent if at least one of them can be written as a linear combination of the others. In other words, {a1, a2,...,am} are linearly dependent if there exists at least one non-zero cj such that m cjaj =0. (2.1.1) j=1 In other words, for some k, ak = −(1/ck) cjaj. j=k Definition 2.1.4. A set of vectors are linearly independent if they are not linearly dependent. That is, in order for (2.1.1) to hold, all cj’s must be equal to zero. Chapter 2 38 BIOS 2083 Linear Models Abdus S. Wahed Definition 2.1.5. Two vectors a and b are orthogonal if their scalar prod- uct is zero. That is, aT b = 0, and we write a ⊥ b. Definition 2.1.6. A set of vectors is said to be mutually orthogonal if members of any pair of vectors belonging to the set are orthogonal. • If vectors are mutually orthogonal then they are linearly independent. Chapter 2 39 BIOS 2083 Linear Models Abdus S. Wahed Definition 2.1.7. Vector space. A set of vectors which are closed under addition and scalar multiplication is known as a vector space. Thus if V is a vector space, for any two vectors a and b from V, (i) caa + cbb ∈V, and (ii) caa ∈Vfor any two constants ca and cb. Definition 2.1.8. Span. All possible linear combinations of a set of linearly independent vectors form a Span of that set. Thus if A = {a1, a2,...,am} is a set of m linearly independent vectors, then the span of A is given by m span(A)= a : a = cjaj , j=1 for some numbers cj,j =1, 2,...,m. Viewed differently, the set of vectors A generates the vector space span(A) and is referred to as a basis of span(A). Formally, • Let a1, a2,...,am be a set of m linearly independent n-dimensional vec- tor in a vector space V that spans V.Thena1, a2,...,am together forms a basis of V and the dimension of a vector space is defined by the number of vectors in its basis. That is, dim(V)=m. Chapter 2 40 BIOS 2083 Linear Models Abdus S. Wahed 2.2 Matrix Definition 2.2.1. A matrix is a rectangular or square arrangement of num- bers. A matrix with m rows and n columns is referred to as an m × n (read as ‘m by n’) matrix. An m × n matrix A with (i, j)th element aij is written as ⎡ ⎤ ⎢ a11 a12 ... a1n ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ a21 a22 ... a2n ⎥ A =(aij)m×n = ⎢ ⎥ . ⎢ ··· ··· ··· ⎥ ⎣ ... ⎦ am1 am2 ... amn If m = n then the matrix is a square matrix. Definition 2.2.2. A diagonal matrix is a square matrix with non-zero elements in the diagonal cells and zeros elsewhere. A diagonal matrix with diagonal elements a1,a2,...,an is written as ⎡ ⎤ ⎢ a1 0 ... 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ 0 a2 ... 0 ⎥ diag(a1,a2,...,an)=⎢ ⎥ . ⎢ ··· ··· ··· ⎥ ⎣ ... ⎦ 00... an Definition 2.2.3. An n×n diagonal matrix with all diagonal elements equal to 1 is known as identity matrix of order n and is denoted by In. Chapter 2 41 BIOS 2083 Linear Models Abdus S. Wahed A similar notation Jmn is sometimes used for an m × n matrix with all elements equal to 1, i.e., ⎡ ⎤ ⎢ 11... 1 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ 11... 1 ⎥ Jmn = ⎢ ⎥ =[1m 1m ... 1m] . ⎢ ··· ··· ··· ⎥ ⎣ ... ⎦ 11... 1 Like vectors, matrices with the same dimensions can be added together and results in another matrix. Any matrix is conformable to multiplication by a scalar. If A =(aij)m×n and B =(bij)m×n,then 1. A ± B =(aij ± bij)m×n, and 2. cA =(caij)m×n. Definition 2.2.4. The transpose of a matrix A =(aij)m×n is defined by T A =(aji)n×m. • If A = AT ,thenA is symmetric. • (A + B)T =(AT + BT ). Chapter 2 42 BIOS 2083 Linear Models Abdus S. Wahed Definition 2.2.5. Matrix product. If A =(aij)m×n and B =(aij)n×p, then T AB =(cij)m×p,cij = aikbkj = ai bj, k where ai is the ith row (imagine as a vector) of A and bj is the jth column (vector) of B. • (AB)T = BT AT , • (AB)C = A(BC),whenever defined, • A(B + C)=AB + AC, whenever defined, • JmnJnp = nJmp. Chapter 2 43 BIOS 2083 Linear Models Abdus S. Wahed 2.3 Rank, Column Space and Null Space Definition 2.3.1. The rank of a matrix A is the number of linearly inde- pendent rows or columns of A.Wedenoteitbyrank(A). • rank(AT )=rank(A). • An m × n matrix A with with rank m (n)issaidtohavefull row (column) rank. • If A is a square matrix with n rows and rank(A) <n,thenA is singular and the inverse does not exist. • rank(AB) ≤ min(rank(A),rank(B)). • rank(AT A)=rank(AAT )=rank(A)=rank(AT ). Chapter 2 44 BIOS 2083 Linear Models Abdus S. Wahed Definition 2.3.2. Inverse of a square matrix.IfA is a square matrix with n rows and rank(A)=n,thenA is called non-singular and there exists −1 −1 −1 −1 a matrix A such that AA = A A = In. The matrix A is known as the inverse of A. • A−1 is unique. • If A and B are invertible and has the same dimension, then (AB)−1 = B−1A−1. • (cA)−1 = A−1/c. • (AT )−1 =(A−1)T . Chapter 2 45 BIOS 2083 Linear Models Abdus S. Wahed Definition 2.3.3. Column space. The column space of a matrix A is the vector space generated by the columns of A.IfA =(aij)m×n =(a1 a2 ... an, then the column space of A, denoted by C(A)orR(A)isgivenby n C(A)= a : a = cjaj , j=1 for scalars cj,j =1, 2,...,n. Alternatively, a ∈C(A) iff there exists a vector c such that a = Ac. • What is the dimension of the vectors in C(A)? • How many vectors will a basis of C(A)have? • dim(C(A)) =? • If A = BC,thenC(A) ⊆C(B). • If C(A) ⊆C(B), then there exist a matrix C such that A = BC. Example 2.3.1. Find a basis for the column space of the matrix ⎡ ⎤ − − ⎢ 12 1 ⎥ ⎢ ⎥ ⎢ ⎥ A = ⎣ 114⎦ . 022 Chapter 2 46 BIOS 2083 Linear Models Abdus S. Wahed Definition 2.3.4. Null Space.Thenullspaceofanm × n matrix A is de- fined as the vector space consisting of the solution of the system of equations Ax = 0. Null space of A is denoted by N (A) and can be written as N (A)={x : Ax =0} . • What is the dimension of the vectors in N (A)? • How many vectors are there in a basis of N (A)? • dim(N (A)) = n − rank(A) → Nullity of A. Chapter 2 47 BIOS 2083 Linear Models Abdus S.