EE531 - System Identification Jitkomut Songsiri 3. Reviews on Linear algebra matrices and vectors • linear equations • range and nullspace of matrices • norm and inner product spaces • matrix factorizations • function of vectors, gradient and Hessian • function of matrices • 3-1 Vector notation n-vector x: x1 x x = 2 . x n also written as x =(x , x ,...,x ) • 1 2 n set of n-vectors is denoted Rn (Euclidean space) • x : ith element or component or entry of x • i x is also called a column vector • y = y y y is called a row vector • 1 2 ··· n unless stated otherwise, a vector typically means a column vector Reviews on Linear algebra 3-2 Special vectors zero vectors: x = (0, 0,..., 0) all-ones vectors: x = (1, 1, , 1) (we will denote it by 1) ··· standard unit vectors: ek has only 1 at the kth entry and zero otherwise 1 0 0 e1 = 0 , e2 = 1 , e3 = 0 0 0 1 (standard unit vectors in R3) unit vectors: any vector u whose norm (magnitude) is 1, i.e., u , u2 + u2 + + u2 = 1 k k 1 2 ··· n q example: u = (1/√2, 2/√6, 1/√2) − Reviews on Linear algebra 3-3 Inner products definition: the inner product of two n-vectors x, y is x y + x y + + x y 1 1 2 2 ··· n n also known as the dot product of vectors x, y notation: xT y properties ✎ (αx)T y = α(xT y) for scalar α • (x + y)T z = xT z + yT z • xT y = yT x • Reviews on Linear algebra 3-4 Euclidean norm x = x2 + x2 + + x2 = √xT x k k 1 2 ··· n properties q also written x to distinguish from other norms • k k2 αx = α x for scalar α • k k | |k k x + y x + y (triangle inequality) • k k ≤ k k k k x 0 and x = 0 only if x = 0 • k k≥ k k interpretation x measures the magnitude or length of x • k k x y measures the distance between x and y • k − k Reviews on Linear algebra 3-5 Matrix notation an m n matrix A is defined as × a11 a12 ... a1n a21 a22 ... a2n A = . , or A =[aij]m n . ... × a a ... a m1 m2 mn a are the elements, or coefficients, or entries of A • ij m n set of m n-matrices is denoted R × • × A has m rows and n columns (m, n are the dimensions) • the (i, j) entry of A is also commonly denoted by A • ij A is called a square matrix if m = n • Reviews on Linear algebra 3-6 Special matrices zero matrix: A = 0 0 0 0 0 0 ··· 0 A = . ···... 0 0 0 0 ··· aij = 0, for i = 1,...,m,j = 1,...,n identity matrix: A = I 1 0 0 0 1 ··· 0 A = . ···... 0 0 0 1 ··· a square matrix with a = 1,a = 0 for i = j ii ij 6 Reviews on Linear algebra 3-7 diagonal matrix: a square matrix with a = 0 for i = j ij 6 a1 0 0 0 a ··· 0 A = 2 . ···... 0 0 a ··· n triangular matrix: a square matrix with zero entries in a triangular part uppertriangular lowertriangular a11 a12 a1n a11 0 0 0 a ··· a a a ··· 0 A = 22 2n A = 21 22 . ···... . ···... 0 0 a a a a ··· nn n1 n2 ··· nn a = 0 for i j a = 0 for i j ij ≥ ij ≤ Reviews on Linear algebra 3-8 Block matrix notation example: 2 2-block matrix A × B C A = D E for example, if B,C,D,E are defined as 2 1 0 1 7 B = , C = , D = 0 1 , E = 4 1 1 3 8 1 9 1 − − then A is the matrix 21 0 1 7 A = 38 1 9 1 0 1 4 1 1 − − note: dimensions of the blocks must be compatible Reviews on Linear algebra 3-9 Column and Row partitions write an m n-matrix A in terms of its columns or its rows × T b1 T b2 A = a1 a2 an = ··· . bT m a for j = 1, 2,...,n are the columns of A • j bT for i = 1, 2,...,m are the rows of A • i 1 2 1 example: A = 4 9 0 1 2 1 a = , a = , a = , bT = 1 2 1 , bT = 4 9 0 1 4 2 9 3 0 1 2 Reviews on Linear algebra 3-10 Matrix-vector product product of m n-matrix A with n-vector x × a11x1 + a12x2 + ... + a1nxn a x + a x + ... + a x Ax = 21 1 22 2 2n n . a x + a x + ... + a x m1 1 m2 2 mn n dimensions must be compatible: # columns in A = # elements in x • if A is partitioned as A = a a a , then 1 2 ··· n Ax = a x + a x + + a x 1 1 2 2 ··· n n Ax is a linear combination of the column vectors of A • the coefficients are the entries of x • Reviews on Linear algebra 3-11 Product with standard unit vectors post-multiply with a column vector 0 a11 a12 ... a1n 0 a1k . a21 a22 ... a2n . a2k Aek = = = the kth column of A . ... 1 . a a ... a . a m1 m2 mn mk 0 pre-multiply with a row vector a11 a12 ... a1n a a ... a eT A = 0 0 1 0 21 22 2n k ··· ··· . ... am1 am2 ... amn = a a a = the kth row of A k1 k2 ··· kn Reviews on Linear algebra 3-12 Trace Definition: trace of a square matrix A is the sum of the diagonal entries in A tr(A)= a + a + + a 11 22 ··· nn example: 2 1 4 A = 0 1 5 3− 4 6 trace of A is 2 1+6=7 − properties ✎ tr(AT )= tr(A) • tr(αA + B)= α tr(A)+ tr(B) • tr(AB)= tr(BA) • Reviews on Linear algebra 3-13 Eigenvalues n n λ C is called an eigenvalue of A C × if ∈ ∈ det(λI A) = 0 − equivalent to: there exists nonzero x Cn s.t. (λI A)x = 0, i.e., • ∈ − Ax = λx any such x is called an eigenvector of A (associated with eigenvalue λ) there exists nonzero w Cn such that • ∈ wT A = λwT any such w is called a left eigenvector of A Reviews on Linear algebra 3-14 Computing eigenvalues (λ) = det(λI A) is called the characteristic polynomial of A • X − (λ) = 0 is called the characteristic equation of A • X eigenvalues of A are the root of characteristic polynomial • Reviews on Linear algebra 3-15 Properties if A is n n then (λ) is a polynomial of order n • × X if A is n n then there are n eigenvalues of A • × even when A is real, eigenvalues and eigenvectors can be complex, e.g., • 2 0 1 2 1 − A = − , A = 6 2 0 1 2 − − 19 5 4 − if A and λ are real, we can choose the associated eigenvector to be real • if A is real then eigenvalues must occur in complex conjugate pairs • if x is an eigenvector of A, so is αx for any α C, α = 0 • ∈ 6 an eigenvector of A associated with λ lies in (λI A) • N − Reviews on Linear algebra 3-16 Important facts denote λ(A) an eigenvalue of A λ(αA)= αλ(A) for any α C • ∈ tr(A) is the sum of eigenvalues of A • det(A) is the product of eigenvalues of A • A and AT share the same eigenvalues ✎ • λ(AT )= λ(A) ✎ • λ(AT A) 0 • ≥ λ(Am)=(λ(A))m for any integer m • A is invertible if and only if λ = 0 is not an eigenvalue of A ✎ • Reviews on Linear algebra 3-17 Eigenvalue decomposition if A is diagonalizable then A admits the decomposition 1 A = TDT − D is diagonal containing the eigenvalues of A • columns of T are the corresponding eigenvectors of A • note that such decomposition is not unique (up to scaling in T ) • recall: A is diagonalizable iff all eigenvectors of A are independent Reviews on Linear algebra 3-18 Inverse of matrices Definition: a square matrix A is called invertible or nonsingular if there exists B s.t. AB = BA = I B is called an inverse of A • it is also true that B is invertible and A is an inverse of B • if no such B can be found A is said to be singular • assume A is invertible an inverse of A is unique • 1 the inverse of A is denoted by A− • Reviews on Linear algebra 3-19 assume A, B are invertible Facts ✎ 1 1 1 (αA)− = α− A− for nonzero α • T T 1 1 T A is also invertible and (A )− =(A− ) • 1 1 1 AB is invertible and (AB)− = B− A− • 1 1 1 (A + B)− = A− + B− • 6 Reviews on Linear algebra 3-20 Inverse of 2 2 matrices × the matrix a b A = c d is invertible if and only if ad bc = 0 − 6 and its inverse is given by 1 1 d b A− = − ad bc c a − − example: 2 1 1 1 3 1 A = , A− = − 1 3 7 1 2 − Reviews on Linear algebra 3-21 Invertible matrices ✌ Theorem: for a square matrix A, the following statements are equivalent 1. A is invertible 2. Ax = 0 has only the trivial solution (x = 0) 3. the reduced echelon form of A is I 4. A is invertible if and only if det(A) = 0 6 Reviews on Linear algebra 3-22 Inverse of special matrices diagonal matrix a1 0 0 0 a ··· 0 A = 2 .