sign in sign up
<<
Home , Circulant matrix, DFT matrix, Linear independence

Appendix: Concepts of

In this appendix, some essential topics in linear algebra are reviewed. For each topic, we present some definitions, basic properties, and numerical examples. Notations An m ! n A consists of m rows and n columns and mn elements (real or complex numbers) and is denoted by

a11 a12 ! a1n

a21 a22 ! a2n mn A " " $aij %i,j 1 " $aij %mn " $aij %. " "#" "

am1 am2 ! amn

The element aii is called the ith diagonal element of A and aij for i ! j is called the &i,j'th element of A. We say the size of A is m ! n or the order of A is m when m " n.Anm ! 1 matrix is said to be an m-vector or a column m-vector; and an 1 ! n matrix is said to be an n-vector or a row n "vector. To avoid any confusion, an n-vector means a column vector in this appendix and a row vector is represented by the (it will be defined shortly) of a column vector. Commonly, Rn and Cn are notations for the sets of real and complex column n-vectors, respectively; and Rm!n and Cm!n are notations for the sets that contain all m ! n real and complex matrices, respectively. If we do not specify the type of a matrix A, then A can be either real or complex. The following are examples of a 2 ! 3 matrix, a column 2-vector and a row 3-vector: 102" i 1 A " , v " , w " abc . "2.5 3i "4 "2 A is said to be a if m " n, otherwise a rectangular matrix. Z is said to be a , denoted by Z " $0%mn " 0, if all elements of Z are zero. Matrix D is said to be an n ! n if all elements of D are zero except its diagonal elements and is commonly written as D "diag d1, #, dn .Ann ! n diagonal matrix with all diagonal elements equal to 1 is called the n ! n , denoted by In or I. A matrix T is said to be an upper (lower) if all its elements below (above) its diagonal are zero. A matrix S is said to be a submatrix of A if the rows and columns of S are consecutive rows and columns of A. If the rows and and columns start from the first ones, S is also called a leading submatrix of A. For example, 3 "14 1 "2 S " 12 is a submatrix of A " and S " is a leading submatrix 512 "45 1 "23 of A " "45"6 . 7 "89 Basic Operations Transpose and Hermitian m!n T Given A " $aij % in R , the transpose of A, denoted by A ,isann ! m matrix whose rows are columns of A and columns are rows of A. When A is in Cm!n, the Hermitian of A, denoted by A!, n!m # is in C and its &i,j'th element is aji. For example, 14 123 1 $ i "2i 1 " i 3 A " , AT " 25 and B " , B! " . 456 34" i 2i 4 $ i 36 Trace of a Square Matrix The trace of an n ! n real square matrix A " $aij % is defined by the sum of the diagonal elements of A, that is, tr A n a . & ' " $k"1 kk 12 Example Let A " . Then, tr&A' " 1 $ 4 " 5. 34 It is not difficult to show that tr&AB' "tr&BA' provided that AB and BA both exist. (or Inner Product) and

u1 v1 Given two vectors u " " and v " " in Cn, the dot product or inner product of u

un vn and v is a ! and is defined as

v1 n ! # # # ! " u v " u1 ! un " " $ uk vk. k"1 vn

1 "4 $ i "3 Example Let u " , v " and w " . Then 2 " 3i 5 " 6i 2

"4 $ i "3 u!v " 12$ 3i " "12 $ 26i, and wTw " "32 " 13. 5 " 6i 2

! Vectors u and v are said to be orthogonal if u v " 0. A set of vectors (v1,...,vm ) is said to be ! ! orthogonal if vi vj " 0 for all i ! j; and is said to be orthonormal if in addition vi vi " 1 for all i " 1,...,m. Consider vectors

1 2 1 1 1 2 u1 " , u2 " , v1 " and v2 " . "1 2 2 "1 22 2

The set u1, u2 is orthogonal and the set v1, v2 is orthonormal. The dot product satisfies the Cauchy-Schwarz Inequality: &x!y'2 % &x!x'&y!y' for any vectors x and y in Cn. Matrix Addition and Two matrices with the same size can be added or subtracted element-wise, and a matrix can be

multiplied by a scalar (real or complex) element-wise. Let A " $aij %mn, B " $bij %mn and !," be scalars. Then

A $ B " $aij $ bij %mn, !A " $!aij %mn and !A $ "B " $!aij $ "bij %mn. 123 789 Example Let A " , B " and ! " "2j. Then 456 10 11 12

81012 "2j "4j "6j "11 "10 "9 A $ B " , !A " , and 3A " 2B " . 14 16 18 "8j "10j "12j "8 "7 "6

Matrix addition and scalar multiplication have the following properties: 1. A $ B " B $ A; 2. A $&B $ C'"&A $ B'$C; 3. &!"'A " !&"A' " "&!A'; 4. &A $ B'T " AT $ BT. Given two matrices A " $aij % and B " $bkl % with sizes m ! r and r ! n, the product C " AB " $cij % is an m ! n matrix and its &i,j'thelement is defined as

b1j r b2j cij " $ aktbkj " ai1 ai2 ! air k"1 "

brj

the dot product of the ith row of A " and the jth column of B.

12 1 "1 Example Let A " 34 , and B " . Then "12 56

"13 "1 "1 "1 2 "3 AB " "15 , BAT " , BB " , and 357 "35 "17

51117 AAT " 11 25 39 . 17 39 61

For a square matrix A, the notation An for a positive n stands for the product AA#A (n times) and A0 & I. Matrix multiplication has the following properties: 1. ABC " A&BC'"&AB'C; 2. &A $ B'C " AC $ BC; 3. A&B $ C' " AB $ AC; 4. &AB'T " BTAT if A and B are real, and &AB'! " B!A! if A and B are complex. In general, matrix multiplication is not commutative, i.e. AB ! BA even if both AB and BA are well-defined and have the same size. When A is a matrix and B is a vector, we can write AB in terms of the columns of A and elements of B, or the rows of A and vector B. Let A be an m ! n matrix and

R1

A " C1 # Cn " "

Rm

b1 % % where Cis and Ris are columns and rows of A, respectively. Let B " " . Then

bn

R1B

AB " b1C1 $ ! $ bnCm " " .

RmB Partitioned Matrices In many applications it is convenient to partition a matrix into blocks (submatrices). For 123 A11 A12 example, the matrix A " 456 can be partitioned as A " where A21 A22 789 12 3 A11 " , A " , A21 " 78 , and A22 " $9%;or 45 12 6 4 56 A11 " 12 , A12 " $3%, A21 " , and A22 " . Operations on partitioned 7 89 matrices work as if the blocks were scalars. For example,

A11 A12 A13 B11 B12 B13 A11$B11 A12$B12 A13$B13 $ " , A21 A22 A23 B21 B22 B23 A21$B21 A22$B22 A23$B23

A11 A12 A11B11$A12B21 A11B12$A12B22 B11 B12 A21 A22 " A21B11$A22B21 A21B12$A22B22 B21 B22 A31 A32 A31B11$A32B21 A31B12$A32B22 provided that all the block products are well-defined. of a Square Matrix Determinant The determinant of a square matrix A, denoted by det&A', is a scalar which provides some useful information about A. The of 2 ! 2 and 3 ! 3 matrices are defined respectively as: a11 a12 det " a11a22 " a12a21, a21 a22

a11 a12 a13 a11a22a33 $ a21a13a32 $ a31a12a23 det a21 a22 a23 " . "a11a23a32 " a21a12a33 " a31a13a22 a31 a32 a33

For a general n ! n matrix A " $aij %, the determinant is defined as: n n i$k k$j det&A' " $&"1' aik det&Aik ' " $&"1' akj det&Akj ' k"1 k"1 for any 1 % i,j % n where Apq is the &n " 1' ! &n " 1' matrix resulting from the deletion of the row p and the column q of A. For example, 1 "23 let i " 1 5 "6 det "45"6 &"1'1$1&1'det " "89 7 "89

"4 "6 "45 $ &"1'1$2&"2'det $ &"1'1$3&3'det 79 7 "8 " &"3' " &"2'&6' $ &3'&"3' " 0

1 "23 let j " 2 "4 "6 det "45"6 &"1'1$2&"2'det " 79 7 "89

13 13 $ &"1'2$2&5'det $ &"1'3$2&"8'det 79 "4 "6 " "&"2'&6' $ &5'&"12' " &"8'&6' " 0 p$q The determinant of Apq, det&Apq ', is called the &p,q'th of A and &"1' det&Apq ' is called the cofactor of apq. Directly from the definition, the determinant of a diagonal matrix is the product of its diagonal elements and the determinant of an upper or lower triangular matrix is also the product of its diagonal elements. Determinants have the following properties: 1. det&AB' " det&A'det&B'; 2. det&!A' " !n det&A' for any scalar ! and n ! n matrix A; 3. det&AT ' " det&A'; 4. det Ak " &det&A''k; 5. det&A' " 0 if any row (or column) of A is a scalar multiple of another row (or column); 6. det&A' " 0 if any row (or any column) of A is zero; 7. If B is obtained from A by interchanging two rows (or two columns), then det&B' " "det&A'. Singular and Nonsingular Matrices A square matrix A is said to be nonsingular if det&A' ! 0 and is singular if det&A' " 0. Since det&AB' " det&A'det&B', the matrix AB is singular if and only if either A is singular or B is singular; and is nonsingular if and only if both A and B are nonsingular. of Vectors, Vectors and of a Matrix Linear Independence of Vectors n n n n Let v1, #,vm be a set of vectors in R or C . A vector u in R or C is said to be a of v ,...,v if there are scalars , ..., such that u m v . The vector 1 m !1 !m " $i"1 !i i T T T u " "12 is a linear combination of v1 " 23 and v2 " "35 since 1 7 T u " v1 $ v2. The vector ... is a solution of the following linear system: 19 19 !1 !n

!1

v1 ... vm " " u.

!m

If the solution does not exist then u is not a linear combination of v1,...,vm. When m " n, the T solution !1 ... !n is unique. The set S containing all linear combinations of v1,...,vm is

called the spanning set of v1,...,vm and is denoted by S "span&v1,...,vm '. A set of vectors v1,...,vm is said to be linearly independent if !1v1 $...$!mvm " 0 implies !1 " 0,...,!m " 0. A set of vectors is linearly dependent if it is not linearly independent, i.e., there are some !i ! 0 such that !1v1 $...$!mvm " 0. Amongst the vectors 2 "3 1 "3 v1 " , v2 " , u1 " , u2 " , 3 5 2 "6

the set (v1,v2 ) is linearly independent and the set (u1,u2 ) is linearly dependent since &"3'u1$u2 " 0. A set of two vectors is linearly dependent if one vector is a scalar multiple of the other, i.e., v2 " !v1 for some nonzero scalar !. A set of m vectors is linearly dependent if one vector is a linear combination of others. Independent vectors have the following properties. 1. A set of orthogonal vectors is linear independent. n 2. Let v1,...,vn be in R and A " v1 ... vn . The set of v1,...,vn is linearly independent

if and only if det&A' ! 0. So, if det&A' " 0, the set of v1,...,vn is linearly dependent. Consider the above vectors v1, v2, u1 and u2. Since 2 "3 det v1 v2 " det " 19 ! 0 and 35

1 "3 det u1 u " det " 0, 2 2 "6 we can conclude that v1, v2 is linearly independent and u1, u2 is linearly dependent. Basis Vectors n Let S "span&v1,...,vm ' where vi’s are in R .If(v1,...,vm ) is linearly independent, then (v1,...,vm ) is called a set of basis vectors of S and the of S is m. In this case, S is also n n called an m-dimensional subspace of the R . Let ei be in R with all elements are zero except T the ith element is 1, i.e., ei " 0 # 010# 0 for i " 1,...,n. A vector T n u " u1 ... un in R can be written as

u " u1e1 $ u2e2 $...$unen, n that is, a linear combination of e1,...,en.So,(e1,...,en ) is a set of basis vectors of R and the dimension of Rn is n. Hence, any n linearly independent vectors in Rn form a set of basis vectors of Rn, and m linearly independent vectors in Rn for m & n cannot form a set of basis vectors for Rn. n If v1,...,vm form a set of basis vectors for an m-dimensional subspace S of R , then each element u in S can be written as

!1

u " !1v1 $...$!mvm " v1 ! vm " .

!m

Values of !1,...,!m can be obtained by solving the linear system:

!1

v1 ! vm " " u.

!m Rank of a Matrix The rank of a m ! n matrix A, denoted by rank&A', is the largest number of columns (or rows) of A that form a set of linearly independent vectors. 1 "23 Example Let A " "45"6 . The rank of A is 2 since the third column of A is a 7 "89 linear combination of the first two columns and the first two columns are linearly independent: 3 1 "2 "2 1 "6 " &"1' "4 $ &"2' 5 and 5 ! ! "4 . 9 7 "8 "8 7

Rank of a matrix has the following properties: 1. An n ! n matrix A is nonsingular if and only if rank&A' " n; 2. rank&AB' % min rank&A', rank&B' ; 3. rank&A $ B' % rank&A' $ rank&B'; 4. The rank of a zero matrix is 0; 5. If rank&A' " k, then there is a k ! k submatrix of A with nonzero determinant but all &k $ 1' ! &k $ 1' submatrices of A have determinant 0. 1 "23 1 "2 Example Let A " "45"6 . det&A' " 0, but det ! 0. So, "45 7 "89 rank&A' " 2 by Property 5. Inverse of a Square Matrix The inverse matrix of an n ! n matrix A, denoted by A"1,isann ! n matrix such that "1 "1 AA " A A " In. A matrix is said to be invertible if its inverse exists. The inverse matrix of a square matrix is unique ab d "b if it exists. The inverse of a 2 ! 2 matrix A " is A"1 " 1 which cd ad " bc "ca exists if and only if ad " bc ! 0. For n ' 3, the inverse matrix A"1 can be obtained by solving n "1 linear systems: AA " In, i.e., "1 A B1 ... Bn " e1 ... en or ABi " ei for i " 1,...n and A " B1 ... Bn . A linear system Ax " b has a unique solution if and only if A is nonsingular. Hence, the inverse of A exists if and only if A is nonsingular or det&A' ! 0, or rank&A' " n. 1 "23 Example Let A " "45"6 . Find A"1 if it exists. 7 "810

Since det&A' " "3 ! 0, A"1 exists. A"1 can be computed as follows. Solve 2 1 "23 1 " 3 v , v 2 , "45"6 1 " 0 1 " 3 7 "810 0 1

4 1 "23 0 3 v , v 11 "45"6 2 " 1 2 " 3 7 "810 0 2

1 "23 0 1

"45"6 v3 " 0 , v3 " 2 7 "810 1 1

2 4 " 3 3 1 A"1 v v v 2 11 . " 1 2 3 " 3 3 2 121

There is a closed form for the inverse of a nonsingular square matrix A " $aij %, given by "1 1 j$i A " Adj&A' where Adj&A' " cofactor of aji " &"1' det&Aji ' . det&A' Inverse matrices have the following properties: 1. det A"1 " 1 ; det&A' 2. The matrix AB is invertible if and only if both A and B are invertible and &AB'"1 " B"1A"1; 3. It A is invertible, then the linear system Ax " b has a unique solution x " A"1b; 4. Let A be an n ! n , and u and v be in Cn. Then "1 &A $ uvH ' " A"1 " 1 A"1uvHA"1. 1 $ vHA"1u This is known as the matrix inversion lemma. The RLS algorithm is based on this lemma. 5. Let A and R be n ! n and m ! m invertible matrices, respectively. Let U be in Cn!m and V be in Cm!n. Then "1 "1 &A $ URVH ' " A"1 " A"1U R"1 $ VA"1U VA"1. This is known as the Sherman-Morrison formula. The block RLS algorithm is based on this formula. Eigenvalues and Eigenvectors of a Square Matrix and Spectral Radius Eigenvalues and Eigenvectors of a Square Matrix Let A be an n ! n matrix, # a scalar and x a nonzero vector in Cn. The pair &#,x' is said to be an eigenpair of A if the equation Ax " #x holds. In this case, # is called an eigenvalue of A and x is called an eigenvector of A associated with #. Example Let 2 "1 1 1 A " , #1 " 1, #2 " 3, x1 " and x2 " . "12 1 "1 With a quick check, 2 "1 1 1 2 "1 1 1 " &1' and " &3' "12 1 1 "12 "1 "1

#1, x1 and #2, x2 are eigenpairs of A. An eigenpair #, x of A satisfies the equation Ax " #x " 0 which implies det&A " #In ' " 0. Define P&#' " det&A " #In '. P&#' is an nth degree in # and is called the characteristic polynomial of A. Eigenvalues of A are zeros of the polynomial P&#' or equivalently, roots of the polynomial equation P&#' " 0. A has an eigenvalue #i with multiplicity k k if and only if P&#' has a factor &# " #i ' . The following are the steps needed to compute all eigenpairs of A: Step I. Solve the polynomial equation P&#' " 0 for #; Step II. For each solution # obtained in Step I, solve &A " #In 'x " 0 for all x which are linearly independent. 1 "2 Example Let A " . Then "34

1 " # "2 2 P&#' " det&A " #I2 ' " det " # " 5# " 2. "34" #

Step I. Solving P&#' " 0 gives 5 1 5 1 #1 " $ 33 , and #2 " " 33 . 2 2 2 2 Step II. Solve for x. 5 1 5 1 For #1 " $ 33 , solve A " $ 33 I2 x1 " 0 for x1. 2 2 2 2 1 " 5 $ 1 33 "2 2 2 x11 0 " "34" 5 $ 1 33 x21 0 2 2

x11 t for any real scalar t. " 3 1 x21 " 4 t " 4 t 33 1 Let t 1. Then x . " 1 " 3 1 " 4 " 4 33 5 1 5 1 For #2 " " 33 , solve A " " 33 I2 x2 " 0 for x2. In a similar 2 2 2 2 1 1 2 $ 6 33 way, we have x2 " . Hence, 1

1 1 $ 1 33 5 $ 1 33 , and 5 " 1 33 , 2 6 2 2 3 1 2 2 " 4 " 4 33 1 are eigenpairs of A. When n ' 3, it may not be possible to find the solutions of the equation P&#' " 0 analytically. In general, eigenpairs #, x are solved numerically. Eigenvalues and eigenvectors have the following properties. 1. If #, x is an eigenpair of A, then #, !x is also an eigenpair of A for any nonzero scalar !.

2. Let #1, x1 ,..., #k, xk be eigenpairs of A.If#1,...,#k are distinct, then x1,...,xk are linearly independent. 3. A has a zero eigenvalue if and only if A is singular. 4. Eigenvalues of A and AT are the same. 5. Eigenvalues of a diagonal matrix, an upper triangular matrix or a lower triangular matrix are the diagonal elements of the matrix. 6. Let A be invertible. If #, x is an eigenpair of A, then 1 , x is an eigenpair of A"1. # 7. Let A be an n ! n matrix with eigenvalues #1,...,#n. Then the determinant and trace of A can be expressed as n n det&A' " ( #i " #1!#n , and tr&A' " $ #i " #1 $ ! $ #n. i"1 i"1 8. If #, x is an eigenpair of A, then # $ $, x is an eigenpair of the matrix A $ $I. Spectral Radius of a Square Matrix For an n ! n matrix A with eigenvalues #1,...,#n, the spectral radius of A, denoted by %&A',is defined as

%&A' " max |#i | 1 % i % n 2 2 k where |a $ ib| " a $ b . A matrix A is said to be convergent if limk') A " 0. Let A be an n ! n complex matrix. We have the following two results. Theorem A is convergent if and only if %&A' & 1. Theorem If %&A' & 1, then the matrix I " A is invertible and &I " A'"1 " I $ A $ A2 $...$An $.... and Kronecker Sum Kronecker Product

Let A " $aij %mn and B " $bkl %pq. The Kronecker product of A and B denoted by A * B is an mp ! nq matrix defined by A * B " $aijB%. 123 1 "1 Example Let A " and B " . Then 456 12

1 "12"23"3 122"43 6 A * B " . 4 "45"56"6 48510612

The Kronecker product is not commutative in general. It has the following properties: 1. &!A' * B " A * &!B'; 2. &A * B'*C " A *&B * C'; 3. &A $ B' * C " A * C $ B * C; 4. A * &B $ C' " A * B $ A * C; 5. &A * B'T " AT * BT and &A * B'! " A! * B!; 6. tr&A * B' "tr&A'tr&B'; 7. Let A be m ! m and B be n ! n. det&A * B' " $det&A'%m$det&B'%n; 8. &A * B'&C * D' " &AC' * &BD'; 9. Let A and B be nonsingular. Then A * B is nonsingular and &A * B'"1 " A"1 * B"1; 10. A * B " 0 if and only if A " 0 and B " 0;

11. If #i, xi is an eigenpair of A and $j, yj is an eigenpair of B, then #i$j, xi * yj is an eigenpair of A * B. Kronecker Sum Let A and B be m ! m and n ! n matrices, respectively. The Kronecker sum of A and B denoted by A + B is an mn matrix defined by A + B " &In * A' $ &Im * B'. 123 1 "1 Example Let A " 456 and B " . Then 12 789 B 00 A 0 A + B " &I2 * A' $ &I3 * B' " $ 0 B 0 0 A 00B

123000 1 "10 0 0 0 21 3 0 0 0 456000 120000 57 6 0 0 0 789000 001"10 0 7810"10 0 " $ " . 000123 001200 00 1 3 2 3 000456 00001"1 00 0 4 6 5 000789 000012 00 0 7 911

If #i, xi is an eigenpair of A and $j, yj is an eigenpair of B, then #i $ $j, yi * xj is an eigenpair of A + B. Vector and Matrix Norms, and Condition Numbers Vector Norms Let S be Rn or Cn. A real valued ||*|| defined on S is said to be a norm or vector norm if ||*|| satisfies the following properties. P1 ||x|| ' 0 for any x in S, and ||x|| " 0 if and only if x " 0, a zero vector. P2 ||!x|| " |!|||x|| for any x in S where ! is an arbitrary scalar. P3 ||x $ y|| % ||x|| $ ||y|| for any x and y in S. (triangular inequality) T Commonly used vector norms: let x " x1 ! xn . E1 The l -norm: x n x . 1 || ||1 & $k"1| i | E2 The l -norm: x n x p 1/p for p an integer and 1 p . In particular, p || ||p & $k"1| i | & & ) 2 2 ||x||2 " x1 $ ! $ xn .

E3 The l)-norm: ||x||) & max1 % k % n |xi | . A vector u is a if ||u|| " 1. A given vector x can be normalized as a unit vector: u " x . Vector norms satisfy the following inequalities. Let x and y be in Rn or Cn. ||x|| ! 2 1. The Cauchy-Schwarz Inequality. &x y' % ||x||2||y||2 with equality if and only if y " !x for some scalar !. 2. ||x|| " ||y|| % ||x " y|| for any vector norm. &k' &k' 3. limk') x " x if any only if limk')||x " x|| " 0 for any vector norm. Matrix Norms Let S be Rm!n or Cm!n. A real valued function |||*||| defined on S is said to be a norm or matrix norm if |||*||| satisfies the following properties. P1 |||A||| ' 0 for any A in S, and |||A||| " 0 if and only if A " 0, a zero matrix. P2 |||!A||| " |!||||A||| for any A in S where ! is an arbitrary scalar. P3 |||A $ B||| % |||A||| $ |||B||| for any A and B in S. (triangular inequality) P4 ||AB|| % |||A||| |||B||| for any A and B in S. (submultiplicative) mn Commonly used matrix norms: let A " $aij %i,j"1. E1 The l matrix norm: A m n a . 1 || ||1 & $i"1 $j"1| ij | 1/p E2 The l matrix norm: A m n a p for p an integer and 1 p . In particular, p || ||p & $i"1 $j"1| ij | & & ) the l matrix norm or the Euclidean norm: A m n a 2 . 2 || ||2 & $i"1 $j"1| ij | E3 The maximum column sum matrix norm: A max m a . ||| |||1 & 1 % j % n $k"1,| kj | E4 The maximum row sum matrix norm: A max n a . ||| |||) & 1 % i % m $k"1,| ik | T Note that when A is real |||A|||) " |||A |||1. ! E5 The spectral norm: |||A|||2 & max # : # is an eigenvalue of A A . ! Note that |||A|||2 " %&A A' . F6. The Frobenius norm: A m n a2 . ||| |||F & $i"1 $j"1 ij ! Note that |||A|||F " tr&A A' " ||A||2. "12"3 Example Let A " . Then 4 "56

||A||1 " 21, ||A||2 " 91 , |||A|||1 " 9, |||A|||) " 15, |||A||| " % ATA " 91 $ 1 8065 , |||A||| " 91 . 2 & ' 2 2 F

Notice that ||A||) & max1 % i,j % n|aij | is not a matrix norm since it does not satisfy the submultiplicative property. 11 22 Example A " , A2 " . Then ||A|| " 1, and 11 22 ) ||A2 || 2 ||A|| ||A|| . ) " ' ) ) Notice also that the spectral radius %&*' is not a matrix norm. Example Let 01 00 A1 " , and A2 " . 00 10 It is easy to verify that properties P1,P3 and P4 do not hold for these two matrices.

P1: %&A1 ' " %&A2 ' " 0 but A1 ! 0 and A2 ! 0.

P3: %&A1 $ A2 ' " 1 ( 0 " %&A1 ' $ %&A2 '.

P4: %&A1A2 ' " 1 ( 0 " %&A1 '%&A2 '. The following inequalities hold for an m ! n matrix A :

1. |||A|||2 % |||A|||F % n |||A|||2;

2. max1 % i % m,1% j % n |aij | % |||A|||2 % mn max1 % i % m,1% j % n |aij |; 3. 1 |||A||| % |||A||| % m |||A||| ; n ) 2 ) 4. 1 |||A||| % |||A||| % n |||A||| ; m 1 2 1

5. |||A|||2 % |||A|||1|||A|||) when A is a square matrix; 6. %&A' % |||A||| for any square matrix and any matrix norm. Condition Numbers For a square matrix A, the quantity |||A||| |||A"1 ||| if A is nonsingular &&A' & ) if A is singular

is called the condition number of A with respect to the matrix norm |||*|||. Commonly, &1, &2 and &), respectively denote the condition numbers of A with respect to the matrix norms |||A|||1, |||A|||2 and |||A|||). 1 "2 "2 "1 Example Let A . Then A"1 and " " 3 1 "34 " 2 " 2

|||A|||1 " 21, |||A|||2 " 14.93303 and |||A|||) " 21. Condition numbers have the following properties. 1. &&I' " 1 for any matrix norm. 2. If A is invertible, then & A"1 " &&A'. #max&A' 3. If A is invertible with eigenvalues #1,...,#n, then &&A' ' for any matrix norm #min&A' where

#max&A' " max |#i | , and #min&A' " min |#i | . 1 % i % n 1 % i % n 4. &&AB' % &&A'&&B'. Similarity Transformation An n ! n matrix A is said to be similar to an n ! n matrix B, denoted by A , B, if there exists a nonsingular n ! n matrix S such that A " S"1BS. The transformation from B to S"1BS is called a similarity transformation. Similarity is an equivalence relation on square matrices, i.e., similarity is &a' reflective: A , A; &b' symmetric: B , A implies A , B; and &c' transitive: C , B and B , A imply C , A. Similar matrices have the following properties. 1. If A , B, then A and B have the same eigenvalues, counting multiplicity. 2. If A , B, then rank&A' "rank&B'. 3. If A , C and B , D, then A $ B , C $ D. Special Matrices and Properties Symmetric and Hermitian Matrices A real (complex) square matrix A is said to be symmetric (Hermitian) if AT " A &A!" A', and skew-symmetric (skew-Hermitian)ifAT" "A &A!" "A'. Any real (complex) square matrix A can be written as A " Ah$As where 1 T 1 ! Ah " A $ A Ah " A $ A 2 & ' 2 & ' a symmetric (Hermitian) matrix and 1 T 1 ! As " A " A As " A " A 2 & ' 2 & ' a skew-symmetric (skew-Hermitian) matrix. Symmetric and Hermitian matrices have the following basic properties. 1. A $ AT, AAT and ATA are symmetric matrices and A $ A!, AA! and A!A are Hermitian matrices. 2. If A is symmetric (Hermitian), then Ak is symmetric (Hermitian) for all k " 1,2,.... IfA is also nonsingular, then A"1is symmetric (Hermitian). 3. If A and B are symmetric (Hermitian), then !A $ "B is symmetric (Hermitian) for all real scalars ! and ". 4. If A is an n ! n symmetric (Hermitian) matrix, then xTAx (x!Ax' is real for all x in Cn and therefore, all eigenvalues of A are real. 5. If A is an n ! n symmetric (Hermitian) matrix then STAS &S!AS' is symmetric (Hermitian) for all S in Rn!n &Cn!n '. #max&A' 6. If A is symmetric (Hermitian) and invertible, then %&A' " |||A|||2 and &2&A' " #min&A' where

#max&A' " max |#i | , and #min&A' " min |#i | , 1 % i % n 1 % i % n % #is are eigenvalues of A. Normal Matrices An n ! n matrix A is said to be normal if A!A " AA!. If a matrix is symmetric or Hermitian, then it is also normal. If A * B is normal, then B * A is also normal. Orthogonal and Unitary Matrices T ! An n ! n real (complex) matrix U is said to be orthogonal (unitary) if U U " In &U U " In '. The determinant of an is either 1 or "1 since det&U2 ' " &det&U''2 " 1. Example Let 1 " 1 " 1 " 1 2 2 2 2 U1 " , and U2 " . 1 1 " 1 1 2 2 2 2

T T Both U1 and U2 are orthogonal matrices since U1 U1" I2 and U2 U2" I2. And det&U1 ' " 1 and det&U2 ' " "1. The following are equivalent: 1. U is unitary; 2. U is nonsingular and U!" U"1; 3. UU!" I;

! 0ifi ! j 4. The columns ui for i " 1,...,n of U are orthonormal, i.e., ui uj " ; 1ifi " j 5. The rows of U are orthogonal; 6. For all x in Cn, the 2-norm of y " Ux is the same as the one of x, i.e., y!y " x!x. ! ! Since UU " I " U U, U is also normal. If U1 and U2 are unitary, then U1U2 and U2U1 are also unitary. If A * B is unitary, then B * A is also unitary. Positive Definite and Semidefinite Matrices An n ! n matrix A is said to be positive definite (semidefinite)ifA is a and x!Ax ( 0 &x!Ax ' 0' for all nonzero x in Cn. Note that x!Ax is always a nonnegative real number. For a positive definite (semidefinite) matrix A, commonly it is indicated as A ( 0 &A ' 0'. Positive definite (semidefinite) matrices have the following properties. 1. All eigenvalues of a positive definite (semidefinite) matrix are positive (nonnegative). 2. All diagonal elements of a positive definite (semidefinite) matrix are positive (nonnegative). 3. If A and B are positive definite (semidefinite), then A $ B is positive definite (semidefinite). n 4. Let A " $aij %i,j"1 be positive definite. Then n det&A' % ( akk k"1 with equality if and only if A is diagonal. 5. If A and B are positive definite (semidefinite), then A * B is positive (semidefinite). Vandermonde matrices An n ! n V is a matrix of the form n"2 n"1 1 x1 ! x1 x1 n"2 n"1 1 x2 ! x x V " 2 2 . ""# " " n"2 n"1 1 xn ! xn xn

The matrix V depends on n elements x1,...,xn. The transpose of a Vandermonde matrix is also called a Vandermonde matrix. Note that the DFT matrix is Vandermonde. It is a fact that n det&V' " ( &xi " xj '. i,j"1, i(j

So, a Vandermonde matrix is nonsingular if and only if x1, x2, ..., xn are distinct. 12 4 % Example Let V " 1 "39 . Here x1 " 2, x2 " "3 and x4 " 4. Since xis are 1416 distinct, det&V' ! 0. Actually, det&V' " &"3 " 2'&4 " 2'&4 " &"3'' " "70. Circulant Matrices An n ! n matrix C of the form

c1 c2 ! cn"1 cn

cn c1 ! cn"2 cn"1

C " cn"1 cn ! cn"3 cn"2 " circ c1, c2, #, cn ""#""

c2 c3 ! cn c1 is called a . Each row is obtained by cyclically shifting to the right the previous row. 1234 4123 Example C "circ&1,2,3,4' " . 3412 2341 A circulant matrix C "circ&c1,...,cn ' can also be written in the form 010! 0

n"1 001! 0 k C " $ ck$1P , where P " ""#"" . k"0 00001 10000 Matrix P is called the basic circulant and satisfies the equation: n P " In. Circulant matrices have the following properties. 1. C is circulant if and only if C! is circulant.

2. If C1 and C2 are circulant, then C1C2 is circulant. 3. Circulant matrices are normal matrices, i.e., CTC " CCT or C!C " CC!. 4. Let #l, xl for l " 1,...,n be eigenpairs of the circulant matrix C "circ&c1,...,cn '. Then 1 "j&2'l/n' n e "j&2'&&k"1'l'/n' 1 "j 4'l/n #l " $ cke , and xl " e & ' for l " 1,...,n. k 1 n " " e"j&2&n"1''l/n'

% Note that the eigenvectors of a circulant matrix do not depend on the cis. So, all n ! n circulant matrices for a given n have the same set of eigenvectors. Toeplitz Matrices An n ! n matrix T of the form

t0 t1 ! tn"2 tn"1

t"1 t0 ! tn"3 tn"2

T " t"2 t"1 ! tn"4 tn"3 ""#""

t"&n"1' t"&n"2' ! t"1 t0 is called a . A Toeplitz T depends on 2n " 1 elements t"&n"1', t"&n"2', ..., t"1, t0, t1, ..., tn"2, tn"1. 1234 5123 Example T " is a 4 ! 4 Toeplitz matrix. 6512 7651 Circulant matrices are also Toeplitz. Hankel Matrices An n ! n matrix H of the form h0 h1 ! hn"2 hn"1

h1 h2 ! hn"1 hn

H " h2 h3 ! hn hn$1 ""#" "

hn"1 hn ! h2n"2 h2n"1

is called a . A Hankel matrix H depends on 2n " 1 elements h0, h1, ..., h2n"1, h2n"1 and its elements are constant along the diagonals perpendicular to the main diagonal. 1234 2345 Example H " . 3456 4567 A real square Hankel matrix is symmetric. Hadamard Matrices T An n ! n matrix H whose elements are )1 such that HH " nIn is called a . An n ! n Hadamard matrix with n ( 2 exists only if n is divisible by 4. The 2 ! 2 and 4 ! 4 Hadamard matrices H2 and H4 are 1111 11 1 "11"1 H2 " , and H4 " . 1 "1 11"1 "1 1 "1 "11 A2n ! 2n Hadamard matrix can be obtained by

H2n" H2*Hn. Note that a Hadamard matrix after normalization is unitary. Diagonalization, Unitary Triangularization and Jordan Form Diagonalization A square matrix A is said to be diagonalizable if it is similar to a diagonal matrix D, i.e. there exists a nonsingular matrix S such that A " SDS"1. 2 "1 11 Example Let A " and B " . Then "12 01

1 "1 10 1 1 A " 2 2 " SDS"1 1 1 11 03 " 2 2 is diagonalizable and B is not diagonalizable. Since A and D are similar, they have the same eigenvalues. Eigenvalues of D are diagonal elements of D. So eigenvalues of A are known if we know D. The process of finding matrices D and S is called the diagonalization. The following results identify diagonalizable matrices: let A be an n ! n matrix. 1. If A has a set of n linearly independent eigenvectors v1,...,vn, then A is diagonalizable and "1 A " S DS where S " v1 ! vn and D "diag(#1,...,#n ), #i is the eigenvalue

corresponding to the eigenvector vi. Recall that eigenvectors corresponding to distinct eigenvalues are linearly independent. So, if eigenvalues of A are distinct then A is diagonalizable. 2. Every symmetric (Hermitian) matrix can be diagonalized by an orthogonal (unitary) matrix U: A " UDUT &A " UDU! '. Recall that if A is positive definite (semidefinite) then all % eigenvalues #is of A are positive (nonnegative). Then D "diag(#1,...,#n ) can be written as T T D " D D where D & diag #1 ,..., #n , and A " UD D U " VV &A " VV! ' where V " UD. 3. An n ! n matrix A is normal if and only if there exists U such that A " UDU! % where D "diag(#1,...,#n ) and #is are eigenvalues of A. 1 1 ! 4. A n ! n circulant matrix C "circ&c1,...,cn ' can be diagonalized by F, i.e., C " F!F , n n where 11 1 ! 1 1 ((2 ! (m"1 F " 1 ((4 ! (2&m"1' , ( " e"j2'/m and j " "1 , """#" 2 1 (m"1 (2&m"1' ! (&m"1' n "j&2'&&k"1'l'/n' ! " diag(#1,...,#n ) where #l " $ cke . k"1 5. If A and B are diagonalizable simultaneously, i.e., there exists a nonsingular matrix S such that "1 "1 A " SDAS and B " SDBS where DA and DB are diagonal matrices, then A and B commute, i.e., AB " BA. Unitary Triangularization For any square matrix A, there is a unitary matrix U such that A " UTU! where T is upper triangular. So, every square matrix is similar to an upper triangular matrix. Note that eigenvalues of A are diagonal elements of T. Jordan Canonical Form For any square matrix A, there is a nonsingular matrix X such that "1 A " X diag Jk1  ', Jk2  ', #, Jkl &#l ' X k!k where Jk&#', called a Jordan block,inC is of the form # 10! 00 0 # 1 ! 00 00# ! 00 Jk&#' " """#"" 000! # 1 000! 0 #

and #1,...,#l are eigenvalues of A with multiplicity k1,...,kl, respectively. The block diagonal matrix Jk1  ' 0 ! 0

0 Jk2  ' ! 0 diag Jk1  ', Jk2  ', #, Jkl &#l ' " ""#"

00! Jkl  ' is called a Jordan matrix. The Jordan canonical form is a set of Jordan matrices. Observe that a % Jordan matrix is “almost diagonal” and is diagonal when all kis are 1 (then A is diagonalizable). Singular Values, Singular Value Decomposition and Pseudo-inverses Singular Values Singular values of an m ! n matrix A, denoted by )i&A', are the square root of eigenvalues of the matrix AA!. 1 "2 5 "11 Example Let A " . Then AAT " , and its eigenvalues are "34 "11 25

#1 " 15 $ 221 and #2 " 15 " 221 . So, singular values of A are: )1 " 15 $ 221 " 5.464986 and )2 " 15 " 221 .

Notice that the spectral norm of a square matrix A is max()i ), that is |||A|||2 " max()i ). Singular Value Decomposition Let A be an m ! n matrix. Then A can be written as A " U"V! where U is an m ! m unitary matrix, V is an n ! n unitary matrix and

"p " " or " " "p 0 0 ! is an m ! n matrix where "A "diag()1,...,)p ), p " min(m,n) and )1 ' ... ' )p. U"V is called the singular value decomposition, SVD in short, of A. Columns of U are eigenvectors of AA! and columns of V are eigenvectors of A!A.IfA is a real matrix, then U and V are orthogonal matrices. 1 "2 1 "23 Example Let A " and B " "34 . The numerical SVD of A and B "45"6 5 "6 are:

"0.42867 0.56631 "0.70395 "0.38632 0.92237 9. 508 0 0 A " "0.80596 0.11238 0.58120 0.92237 0.38632 0 0.77287 0 0.40825 0.81650 0.40825 and "0.22985 "0.88346 0.40825 9. 5255 0 "0.61963 0.78489 B " 0.52474 0.24078 0.81650 0 0.5143 0.78489 0.61963 "0.81964 0.40190 0.40825 00

If the rank of A is r, then )1 ' ... ' )r ( 0 and )r$1 " 0, ..., )p " 0. Pseudo-inverses Let A be an m ! n matrix with rank r and A " U"V!. Then

"r 0 " " where "r " diag )1, ..., )r . 00

"1 &"r ' 0 Define ""1 " . Then A"" V""1U! is called the pesudo-inverse of A, or the 00 Moore-Penrose generalized inverse of A. It can be shown that A" satisfies the following properties: 1. AA" and A"A are Hermitian; 2. AA"A " A; and 3. A"AA" " A. One of the applications is to solve the linear system Ax " b where A " U"V! is an m ! n matrix with rank r. A least-squares solution xls to the linear system Ax " b is " "1 ! xls " A b " &V" U 'b. References 1. G. Golub and C. Van Loan, “Matrix Computation”, Johns Hopkins University Press, 1983. 2. R. Horn and C. Johnson, “Matrix Analysis”, Cambridge University Press, 1985. 3. G. Strang, “ Introduction to Linear Algebra”, Wellesley-Cambridge Press, 1998