sign in sign up
<<
Home , Matrix norm, Spectral radius

ECE5580: Multivariable Control in the Frequency Domain 3–1

Matrix Theory and Norms

In this chapter, we will review some important fundamentals of ! algebra necessary to the development of multivariable frequency response techniques Importantly, we’ll focus on the geometric construct of the: ! – eigenvalue-eigenvector decomposition – decomposition

Additionally, we’ll discuss the notion of norms !

Basic Concepts

Complex numbers

Define the complex scalar quantity ! c ˛ jˇ D C where ˛ Re c and ˇ Im c D f g D f g We may express c in polar form, c c c, where c denotes the ! D j j † j j magnitude (or modulus) of c and c its phase angle: † c pcc ˛2 ˇ2 j j D N D C pˇ c arctan † D ˛ Â Ã Here, the c notation indicates complex conjugation. N

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–2 Complex vectors and matrices

A complex column vector a with m components may be written, ! a1 a a 2 2 3 D : 6 7 6 7 6 am 7 6 7 4 5 where the ai are complex scalars We define the following transpose operations: ! T a a1 a2 am D ### h i a$ a1 a2 am D N N ### N h i – The latter of these is often called the complex conjugate or Hermitian transpose

Similarly, we may define a complex ` m matrix A, ! %

a11 a12 a1m ### 2 a21 a22 a2m 3 A ### D : : ::: : 6 7 6 7 6 a`1 a`2 a`m 7 6 ### 7 with elements 4 5 aij Re aij j Im aij D C Here, ` denotes the number of˚ rows« (outputs),˚ « and m the number of ! columns (inputs) Similar to the vector, we define the following operations: ! AT transpose of A D Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–3 A conjugate of A N D A$ of A D MATRIX INVERSE ! 1 adj A A& D det A where adj A is the classical adjoint (“adjugate”) of A which is

computed as the transpose of the matrix of cofactors cij of A, i j ij cij Œadj Aji , . 1/ C det A D & Here Aij is the submatrix formed by deleting row i and column j of A.

Example 3.1

a a A 11 12 D " a21 a22 #

det A a11a22 a12a21 D &

1 1 a22 a12 A& & D det A a21 a11 " & # Some Determinant Identities n

det A aij cij .expansion along column j / ! D i 1 XD n

det A aij cij .expansion along row i/ ! D j 1 XD Let c be a complex scalar and A be an n n matrix, ! % det .cA/ cn det A D

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–4 Let A be a nonsingular , ! 1 1 det A& D det A Let A and B be matrices of compatible such that AB and ! BA are square, det .I AB/ det .I BA/ C D C For block triangular matrices, ! AB A0 det det " 0D# D " CD# det .A/ det .D/ D # Schur’s Formula for the determinant of a partitioned matrix: !

AB 1 det det .A/ det D CA& B " CD# D # & $ 1 % det .D/ det A BD& C D # & where it is assumed that A and/or D are non-singular.$ %

Vector Spaces

We’ll start with the simplest example of a : the Euclidian ! n n-dimensional space, R , e.g.,

u Œu1;u2; :::; un with ui R for i 1; 2; : : : ; n D 2 D n R exhibits some special properties that lead to a general definition of ! a vector space:

n – R is an additive abeliean group n – Any n-tuple ku is also in R

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–5 n – Given any k R and u R we can obtain by scalar multiplication 2 2 the unique vector ku

In abstract algebra, an abelian group, also called a commutative group, is a group in which the result of ap- plying the group operation to two group elements does not depend on the order in which they are written (the axiom of commutativity). Abelian groups generalize the arithmetic of addition of integers. [Jacobson, Nathan, Basic Algebra I, 2nd ed., 2009]

Some further results: ! – k .u v/ ku kv (i.e., scalar multiplication is distributive under C D C addition) – .k l/u ku lu C D C – .kl/ u k .lu/ D – 1 u u # D To define a vector space in general we need the following four ! conditions:

1. A set V which forms an additive abeliean group 2. A set F which forms a field 3. A scalar multiplication of k F with u V to generate a unique 2 2 ku V 2 4. Four scalar multiplication axioms from above.

Then we say that V is a vector space of F . !

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–6 Examples:

All ordered n-tuples, u Œu1;u2;:::;un with ui C forms a vector ! D 2 space Cn over C n m The set of all n m real matrices R % is a vector space over R ! %

Linear Spans, Spanning Sets and Bases

Let S be the set of vectors, S= u1; u2;:::;um V where V is a vector ! f g 2 space of some F

Define L.S/ v V v a1u1 a2u2 ::: amum with ai F ! Df 2 W D C C C 2 g Thus L.S) is the set of all linear combinations of the vectors ui ! Then, ! – L.S/ is a subspace of V (note, it is closed under vector addition and scalar multiplication) – L.S/ is the smallest subspace containing S

We say that L.S/ is the linear span of S, or that L.S/ is spanned by S ! Conversely, S is called a spanning set for L.S/ !

IMPORTANT: A spanning set v1; v2;: : : ; vm of V is a basis if the vi are linearly independent

NOTE: If dim.V / m, then any set v1; v2;: : : ; vn V with n>mmust be D 2 linearly dependent

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–7 Linear Transformations, Matrix Representation & Matrix Algebra

Linear Mappings

Let X; Y be vector spaces over F and let T be a mapping of X to Y , ! T X Y W ! Then T L.X; Y /; the set of linear mappings (or transformations) of ! 2 X into Y , if T.u v/ T.u/ T.v/ C D C and T .ku/ k T.u/; for all u; v X and k F D # 2 2 Linear Tranform ) Matrix Respresentation of Linear Mappings

Linear mappings can be made concrete and conveniently ! represented by matrix multiplications...

– once x X and y Y are both expressed in terms of their 2 2 coordinates relative to appropriate basis sets

We can now show that A is a matrix representation of the linear ! transform T :

– y T.x/ is equivalent to y Ax D D k Note that when dealing with actual vectors in Rk or C , an all-time ! favorite basis set is the standard basis set, which consists of vectors th ei which are zero everywhere except for a “1” appearing in the i position

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–8 Change of Basis

0 0 0 Let e ; e ;: : : ; e m be a new basis for X ! 1 2 Then each e0 is in X and can be written as a linear combination of the ! i ei : n

e0 pij ej i D j 1 XD E0 EP D

where P is a square and invertible matrix with elements pji Thus we can write, ! 1 E E0P & D 1 meaning that P & transforms the old coordinates into the new coordinates. We define a similarity transformation as ! 1 T 0 P & TP D

Image, Kernal, & Nullity Isomorphism. A linear transformation is an isomorphism if the mapping is “one-to-one” Image. The image of a linear transformation T is, Image.T / y Y T.x/ y; x X D f 2 W D 2 g Kernel. The kernel of a linear transformation T is, ker.T / x X T.x/ 0 D f 2 W D g Image.T / is a subspace of Y (the co-domain) and ker.T / is a ! subspace of X (the domain)

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–9 Example 3.2

2 101 A R3 R with A W ! D " 011# It is clear to see that matrix A transforms a 3 1 input vector into a ! % 2 1 output vector, which is the dimension of the image of A % The vector n, ! 1 & n ˛ 1 D 2 & 3 1 6 7 4 5 denotes the null space of A

Rank of a Matrix

The rank of a matrix is defined as the number of linearly independent ! columns (or equivalently the number of linearly independent rows) contained in the matrix Note that in the case of T L.X; Y / defined by x y Ax, the ! 2 ! D dimension of Image.T / is the dimension of the linear span of the columns of A, which equals the number of linearly independent column vectors An alternative but equivalent way to define the rank of a square matrix ! is by way of minors:

– If an n n matrix A has a non-zero r r minor, while every minor % % of order higher than r is zero, then A has rank r.

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–10 Example 3.3

110 2 0 113 A & D 110 6 7 6 7 6 3217 6 7 4 5 Since the dimension of A is 4 3, it’s rank can be no larger than 3 ! % But since column 2 plus column 3 equals column 1, the rank reduces ! to 2 All four 3 3 minors of A are equal to 0, but A has non-zero minors of ! % dimension 2 2, indicating A has rank 2 % Corrollary: A square n n matrix is full rank (non-singular) if its % determinant is non-zero.

Some additional properties of determinants:

Let A; B Rn 2 1. det.A/ 0 if and only if the columns (rows) of A are linearly D dependent

(a) If a column (row) of A is zero, then det.A/ 0 D (b) det.A/ det.AT / D (c) det.AB/ det.A/ det.B/ D # (d) Swapping two rows (columns) changes the sign of det.A/. i. Scaling a row (column) by k, scales the determinant by k. ii. Adding a multiple of one row (column) onto another does not affect the determinant.

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–11

(e) If A diagonal aii or lower triangular, or A upper triangular with D f g D diagonal elements aii; then det.A/ a11 a22 ann D # #####

Eigenvalues and Eigenvectors

What is an Eigenvalue?

Often in engineering we deal with matrix functions, e.g., ! 1 G.s/ C.sI A/& B D & Here, A is a square real matrix, and G.s/ is the transfer ! matrix of a dynamic system with many inputs and many outputs Furthermore, ! At y e x0 D is the form of solution of n first-order ordinary differential equations 1 For scalar systems, we know how to handle terms like , whose ! s a inverse Laplace transform is just eat & 1 But how do we handle the matrix .sI A/& ? And how do we ! & evaluate the eAt ?

– Remember that an n n matrix A can be thought of as a matrix % representation of a linear tranformation on Rn, relative to some basis (e.g., the standard basis) – Yet we know that a change of basis, described in matrix form by 1 E0 EP , will transform A to A0 P & AP D D Suppose that it were possible to choose our new basis such that ! 1 P & AP were a diagonal matrix D

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–12 – This transforms a general matrix problem into a diagonal (scalar) 1 one – it now consists of a collection of simple elements, =.s di / and & edi t , which we can easily solve

Such a basis exists for almost all A0s and is defined by the set of ! eigenvectors of A

The corresponding diagonal elements di of D turn out to be the ! eigenvalues of A

Eigenvalue-Eigenvector Relationship

A linear transformation T maps x y Ax where in general x and ! ! D y have different directions There exist special directions w however which are invariant under ! the mapping Aw "w; where " is a scalar D Such A-invariant directions are called eigenvectors of A and the ! corresponding "’s are called eigenvalues of A; eigen in German means “own” or “self” The above equation can be re-arranged as: ! ."I A/ w 0 & D and thus implies that w lies in the kernal of ."I A/ & A non-trivial solution for w can be obtained if and only if, ! ker."I A/ 0 , which is only possible if ."I A/ is singular, i.e., & ¤ f g & det."I A/ 0 & D

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–13 This equation is called the characteristic equation of A, and defines ! all possible values for the eigenvalues "

The " may assume one of the n roots, "i of the characteristic ! equation, and are thus called characteristic roots or simply the eigenvalues of A Interesting and important properties of the eigenvalues of a matrix A: ! n

– .A/ sum of the diagonal elements of A "i sum of Á D D i 1 the eigenvalues XD n

– det.A/ determinant of A "i product of the eigenvalues Á D D i i YD Once the values of " have been determined, the corresponding ! eigenvectors may be calculated by solving the set of homogeneous equations:

w1

0 w2 1 th ."I A/ 0; where wi denotes the i element of w & : D B C B C B wn C B C @ A NOTE: If Aw w , then A .kw/ kAw k"w ".kw/, so that if w D D D D is an eigenvector of A, then so also will be any scalar multiple of w – the important property of w is its direction, not its scaling or length

Example 3.4: 1 35 & A 012 D 2 3 0 14 6 & 7 4 5 Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–14 Solving det .sI A/ 0 gives, ! & D det."I A/ "3 6"2 11" 6 0 & D & C & D and thus, "1 1"2 2"3 3 D D D Solving for the eigenvectors we obtain ! 1 1 1

w1 ˛1 0 w2 ˛2 2 w3 ˛3 1 D 2 3 D 2 & 3 D 2 3 0 1 1 6 7 6 & 7 6 7 4 5 4 5 4 5 where the ˛’s are arbitrary scaling factors.

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–15 IMPORTANT EIGENVALUE/EIGENVECTOR PROPERTIES:

1. The eigenvalues of a diagonal or are equal to the diagonal elements of the matrix. 2. The eigenvectors of a diagonal matrix (with distinct diagonal elements) are the standard basis vectors. 3. The eigenvalues of the identity matrix are all "1" and the eigenvectors are arbitrary. 4. The eigenvalues of a scalar matrix kI are all k and the eigenvectors are arbitrary. 5. Multiplying a matrix by a scalar k has the effect of multiplying all its eigenvalues by k , but leaves the eigenvectors unaltered. 6. (Shift Theorem) Adding onto a matrix a scalar matrix, kI, has the effect of adding k to all its eigenvalues, but leaves the eigenvectors unaltered: "i .A kI/ "i .A/ k C D C 7. The eigenvalues of the inverse of a matrix are given by:

1 1 "i A& D "i .A/ 8. The eigenvectors of a matrix$ and% its transpose are the same.

9. The left eigenvectors of a matrix A are the right eigenvectors of A$ 10. The eigenvalues/eigenvectors of a real matrix are real or appear in complex conjugate pairs.

11. #.A/ , max "i .A/ is called the of A i j j 12. Eigenvalues are invariant under similarity transformations.

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–16 Gershgorin’s Theorem The eigenvalues of the n n matrix A lie in the union of n circles in ! % the complex plane, each with center aii and radius

ri aij D j i X¤ ˇ ˇ – Radius equals the sum of off-diagonalˇ ˇ elements in row i

They also lie in the union of n circles, each with center aii and radius !

ri aji D j i X¤ ˇ ˇ – Radius equals the sum of off-diagonalˇ ˇ elements in column i

Example 3.5

Consider the matrix, ! 51 2 & A 42 1 D 2 & 3 3 26 6 & & 7 4 5 The computed eigenvalues are ! 5:1866 & ! 5:6081 D 2 3 2:5785 6 7 4 5 Gershgorin bands for both row and column sums are depicted in the ! plots below (overlaid with actual eigenvalues)

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–17

Eigen-Decomposition By convention, we collect all the eigenvectors of an n n square ! % matrix A as the column vectors of the eigenvector matrix, W :

W Œw1; w2; ; wn D ### Upon inversion we can obtain the dual set of left eigenvectors as the ! 1 row vectors of the dual eigenvector matrix, V W & : D T v1 T 1 2 v2 3 V W & D D : 6 7 6 T 7 6 vn 7 6 7 The left eigenvectors vT satisfy 4 5 ! i T T T v .A "i I/ 0 v A "i v i & D , i D i Combining the eigenvector and dual eigenvector matrices together ! we get the eigenvalue/eigenvector decomposition (or characteristic decomposition): AW Wƒ D

where ƒ is a diagonal matrix containing the eigenvalues, "i , of A.

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–18 1 – Post-mulitplying the above by W & we obtain:

1 A WƒW& WƒV D D This is also known as the spectral decomposition of A. ) The eigen-decomposition can be ill-conditioned with respect to its ! computation as shown in the following example.

Example 3.6

1$ A D " 0 1:001 # where by inspection, "1 1 and "2 1:001 D D It is easy to see that, ! 1$ 1 1 1 " 0 1:001 # 0 D # " 0 # 1 Hence the first eigenvector is w1 ! D " 0 # From, ! 1$ a a $b a C .1:001/ " 0 1:001 #"b # D " 1:001b # D " b # we get,

$b :001a D 1:001b 1:001b D which gives, a 1000$b D Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–19 1000$ Letting b 1, then the second eigenvector is w2 ! D D " 1 #

The w2 eigenvector has a component which varies 1000 times faster ! than $ in the A matrix In general, eigenvectors can be extremely sensitive to small changes ! in matrix elements when the eignvalues are clustered closely together

Some Special Matrices T REAL : S S ! D – Here S has real eigenvalues, "i , and hence real eigenvectors, wi , where the eigenvectors are orthonormal (i.e., wT w 0, for all j ! i D i j ) . Therefore its eigenvalue/eigenvector decomposition ¤ becomes: S RƒRT ; D where RT R RRT I . D D T Corollary: Any quadratic form, aij xi xj can be written as x Sx and is

positive definite (i.e., is positive for x 0 ) if and only if "i .S/ > 0 for X ¤ all i.

NORMAL MATRIX: N $N NN$ ! D – If ."i ; wi / is an eigen-pair of N , then "N i ; wi is an eigen-pair of N $ – The eigenvectors of N are orthogonal.$ %

HERMITIAN MATRIX: H $ H ! D – H has real eigenvalues and orthogonal eigenvectors.

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–20

UNITARY MATRIX: U $U UU$ I ! D D – The eigenvalues of U have unit modulus and the eigenvectors are orthogonal.

Jordan Form If the n n matrix A cannot be diagonalized, then it can always be ! % brought into Jordan form via a similarity transformation,

J1 0 1 : T & AT J :: D D 2 3 0J 6 q 7 4 5 where

"i 10 :: 2 "i : 3 n n Ji R i % i D ::: 1 2 6 7 6 7 6 0 "i 7 6 7 4 5 is called a Jordan block of size ni with eigenvalue "i . Note that J is block-diagonal and upper bi-diagonal.

Singular Value Decomposition Despite its extreme usefulness, the eignenvalue/eigenvector ! decomposition suffers from two main drawbacks: 1. It can only handle square matrices 2. Its computation may be sensitive to even small errors in the elements of a matrix To overcome both of these we can instead use the Singular Value ! Decomposition (when appropriate).

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–21 Let M be any matrix of dimension p m. It can then be factorized as: ! % M U †V $ D where UU$ Ip and V V $ Im. Here, † is a rectangular matrix of D D dimension p m which has non-zero entries only on its leading diagonal: % %1 0 00 0 ### ### 2 0%2 00 0 3 † ### ### D : : ::: : : ::: : 6 7 6 7 6 00 %N 0 0 7 6 ### ### 7 and %1 %2 %N 40. 5 ( (###( ( This factorization is called the Singular Value Decomposition (SVD). ! The %i ’s are called the singular values of M . ! The number of positive (non-zero) singular values is equal to the rank ! of the matrix M . The SVD can be computed very reliably, even for very large matrices, ! although it is computationally heavy.

– In MATLAB it can be obtained by the function svd: [U,S,V] = svd(M).

Matrices U and V are unitary matrices, i.e., they satisfy ! 1 U $ U & D and "i .U/ 1 i k k D 8

The largest singular value of M , %1, (also denoted, %) is an induced ! N of the matrix M :

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–22

M x %1 sup k k D x x k k It is important to note that, ! 2 MM$ U† U $ D and 2 M $M V† V $ D These are eigen-decompositions since U $U I and V $V I with ! D D † diagonal.

SOME PROPERTIES th %i "i .M M/ "i .MM /= i singular value of M ! D $ D $ th vi pwi .M $M/ = i pinput principal direction of M ! D th ui wi .MM $/ = i output principal direction of M ! D

Example 3.7 (Golub & Van Loan):

0:96 1:72 A U †V T D " 2:28 0:96 # D

T 0:6 0:8 30 0:8 0:6 & D 0:8 0:6 01 0:6 0:8 " & #" #" # Any real matrix, viewed geometrically, maps a unit-radius ! (hyper)sphere into a (hyper)ellipsoid.

The singular values %i give the lengths of the major semi-axes of the ! ellipsoid.

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–23

The output principal directions ui give the mutually orthogonal ! directions of these major axes.

The input principal directions vi are mapped into the ui vectors with ! gain %i such that, Avi %i ui . D

3 × u1

v2 v1

u2

Domain of Matrix Input Image of Matrix Output

Numerically, the SVD can be computed much more accurately than ! the eigenvectors, since only orthogonal transformations are used in the computation.

Singular value inequalities

% .A/ "i .A/ % .A/ ! Ä j j ÄN T %.A$/ %.A/ and % A %.A/ ! N DN N DN %.AB/ %.A/ %.B/ $ % ! N ÄN N % .A/ %.B/ %.AB/ or %.A/ % .B/ %.AB/ ! N ÄN N ÄN % .A/ % .B/ % .AB/ ! Ä A max %.A/ ; %.B/ % p2 max %.A/ ; %.B/ ! N N ÄN" B # Ä N N n o n o Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–24 A % %.A/ %.B/ ! N " B # ÄN CN A0 % max % .A/ ; % .B/ ! N " 0B# D N N n o %i .A/ % .B/ %i .A B/ %i .A/ % .B/ ! &N Ä C Ä CN

The condition number of a matrix A is defined as the ratio of the ! maximum to the minimum singular values, %.A/ & .A/ N D % .A/

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–25 Vector & Matrix Norms

Vector Norms

A vector norm on C is a function f C R with the following ! W ! properties:

n – f.x/ 0 x C ( 8 2 – f.x/ 0 x 0 D , D n – f.x y/ f.x/ f.y/ x; y C C Ä C 8 2 n – f .˛x/ ˛ f.x/ ˛ C; x C Dj j 8 2 2 The p-norms are defined by ! n 1=p p x xi k kp D j j i 1 ! XD where p 1 ( Three norms of particular interest to control theory are: ! – VECTOR 1-NORM n

x xi k k1 D j j i 1 X& – VECTOR 2-NORM (EUCLIDIAN NORM)

n 2 x xi px x k k2 D v j j D $ u i 1 uX& t – VECTOR -NORM 1 x max xi k k1 D i j j The following relationships hold for 1, 2, and norms for vectors: ! 1

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–26 – x x pn x k k2 Ä k k1 Ä k k2

– x x 2 pn x k k1 Ä k k Ä k k1

– x x 1 n x k k1 Ä k k Ä k k1 The following plot shows contours for the vector p-norms defined ! above (n 2) D = ∞ p x2 1 p =2

p =1 x1 −1 1

−1

Matrix Norms Matrix norms satisfy the same properties outlined above for vector ! norms Matrix norms normally used in control theory are also the 1-norm, ! 2-norm and -norm. 1 Another useful norm is the Frobenius norm, defined as follows: ! 1=2 m n 2 m n A aij A C % k kF D 0 j j 1 8 2 i 1 j 1 XD XD @ A th th Here aij denotes the element of A in the i row, j column.

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–27 The p-norms for matrices are defined in terms of induced norms, that ! is they are induced by the p-norms on vectors. One way to think of the norm A is as the maximum gain of the ! k kp matrix A as measured by the p-norms of vectors acting as inputs to the matrix A:

Ax p m A p sup k k A C k k D x 0 x p 8 2 ¤ k k – Norms defined in this manner are known as induced norms – NOTE: the supremum is used here instead of the maximum since the maximum value may not be achieved

The matrix norms are computed by the following expressions: ! m

A 1 max aij k k D j j j i 1 XD A %.A/ k k2 DN n

A max aij k k1 D i j j j 1 XD

SOME USEFUL RELATIONSHIPS: AB A B .multiplicative property/ ! k kp Ä k kp k kp A A pn A ! k k2 Ä k kF Ä k k2

max aij A 2 pmnmax aij ! i;j j jÄk k Ä i;j j j

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–28

A 2 A 1 A ! k k Ä k k k k1 1 p A A 2 pm A ! pn k k1 Ä k k Ä k k1 1 A1 A pn A ! pm k k Ä k k2 Ä k k1

Spectral Radius

The spectral radius was defined earlier as ! #.A/ max "i .A/ D i j j Note: the spectral radius is not a norm! ! Example 3.8

Consider the matrices ! 10 1 10 A1 A2 D " 10 1 # D " 01# It is obvious that ! #.A1/ 1; #.A2/ 1 D D But, ! #.A1 A2/ 12 C D #.A1A2/ 101:99 D Hence the spectral radius does not satisfy the multiplicative property, ! i.e.,

# .A1A2/ # .A1/#.A2/ —

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–29 System Norms

Consider the input-output system below ! w z G

– Given information about allowable input signals w.t/, how large can the outputs z.t/ become? – To answer this question, we must examine relevant system norms.

We shall evaluate the output signals in terms of the signal 2-norm, defined by

1 2 z.t/ zi .'/ d' k k2 D j j s i X Z&1 and consider three inputs: 1. w .t/ is a series of unit impulses 2. w .t/ is any signal such that w .t/ 1 k k2 D 3. w .t/ is any signal such that w .t/ 1, but w .t/ 0 for t 0 k k2 D D ( and we only measure z .t/ for t 0 ( The relevant system norms are: 2, , and Hankel norms, ! H H1 respectively

2 norm H Consider a stricly proper system G .s/ (i.e., D 0 ) ! D For the 2 norm we use the Frobenius norm spatially (for the matrix) ! H and integrate over frequency, i.e.,

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–30

1 1 G .s/ 2 , tr G .j!/$ G .j!/ d! k k s2( Z&1 $ % – G .s/ must be strictly proper, or else the 2 norm is infinite H By Parseval’s theorem we have !

1 T G .s/ 2 g .t/ 2 , tr .g .'/ g .'// d' k k D k k sZ0 where g .t/ is a matrix of impulse responses. Note we can change the order of integration and summation to get !

1 2 G .s/ 2 g .t/ 2 gij .'/ d' k k D k k D v 0 u ij Z uX ˇ ˇ th t ˇ ˇ where gij .t/is the ij element of the impulse response matrix.

Thus, the 2 norm is the 2-norm output resulting from applying unit ! H impulses ıj .t/ to each input

Numerical computation of the 2 norm H 1 Consider G .s/ C .sI A/& B ! D & Then, ! G .s/ tr .BT QB /or G .s/ tr .CP C T / k k2 D k k2 D where Q is the observabilityp Gramian and P is thep controllability Gramian.

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–31 H norm 1 Consider a proper, stable system G .s/ (for this case, nonzero D is ! allowed) For the norm, we use the singular value (induced 2-norm) ! H1 spatially for the matrix and pick out the peak value as a function of frequency G .s/ , max % .G .j!// k k1 ! N – The norm is interpreted as the peak transfer function H1 “magnitude”

Time Domain Interpretation: ! – Worst-case steady-state gain for sinusoidal inputs at any frequency – Induced (worst-case) 2-norm in the time domain:

z .t/ 2 G .s/ max k k max z .t/ 2 k k1 D w.t/ 0 w .t/ D w.t/ 1 k k ¤ k k2 k k2D – NOTE: at a given frequency !, the gain depends on the direction of w .!/

Numerical computation of the norm H1 Computation of the H norm is non-trivial ! 1 – It can be approximated via the defnition G .s/ , max %.G .j!// k k1 ! N – But in practice, it’s normally computed via its state-space representation as shown below

Consider the system represented by ! G .s/ C .sI A/B D D & C Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–32 The norm is the smallest value of ) such that the Hamiltonian ! H1 matrix H has no eigenvalues on the imaginary axis

1 T 1 T A BR& D C BR& B H T C 1 T 1 T T D " C I DR & D C A BR& D C # & C & C where R ) 2I DT$D % $ % D & – The derivation of this expression will not be addressed here

Comparison of 2 and norms H H1 We first recognize that we can write the Frobenius norm in terms of ! singular values, 1 G .s/ 1 % 2 .G .j!// d! k k2 D 2( i s i Z&1 X Minimizing the norm essentially “pushes down the peak” of the ! H1 largest singular value

Minimizing the 2 norm “pushes down the entire function” of singular ! H values over all frequencies

Example 3.9

Consider the plant ! 1 G .s/ D s a C Computing the H2 norm ! 1 1 2 G .s/ 1 G .j!/ 2 d! k k2 D 2( j j  Z&1 Ã

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–33 1 2 1 1 ! 1 tan& D 2(a a  h & Ái&1à 1 D r2a Alternatively, consider the impulse response ! 1 1 at g .t/ & e& ;t 0 D L s a D (  C à – which gives 1 g .t/ 1 .e at/2 dt k k2 D & D 2a sZ0 r Computing the norm, ! H1 1 1 G .s/ max G .j!/ max 1 k k1 D ! j j D ! .!2 a2/2 D a C Note that there is no general relationship between the 2 and ! H H1 norms

Hankel norm

A special norm in systems theory is the Hankel norm, which relates ! the future output of a system to its past input:

2 1 z .'/ d' 0 k k2 G .s/ H , max k k w.t/ qR0 2 w .'/ 2 d' &1 k k It can be shown that the Hankel normqR is equal to ! G .s/ #.PQ/ k kH D p

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–34 where # is the spectral radius and P and Q are the controllability and observability Gramians, respectively The name “Hankel norm” derives from the fact that the matrix PQ is ! in the form of a Hankel matrix The Hankel singular values are given by ! %H;i "i .PQ/ D Hankel and norms can be relatedp by ! H1 n

G .s/ H %H G .s/ 2 %H;i k k DN Ä k k1 Ä i 1 XD

Example 3.10

Let’s compute the various norms for the system of Example 3.9 using ! state-space formulations.

For the plant G .s/ 1=.s a/ we have the following state-space ! D C realization: AB a1 & " CD# D " 10# The controllability Gramian P is the solution of the Lyapunov equation ! AP PAT BB T C D& aP aP 1 & & D& 1 P D 2a Similarly, for the observability Gramian, ! 1 Q D 2a

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–35

So for the 2 norm we have ! H T 1 G .s/ 2 tr .B QB / k k D D r2a Computing the H norm, we firstp find the eigenvalues of the ! 1 Hamiltonian matrix, 1 a 1 ".H / " 2 a2 & " 2 D 2 1a3 D˙s & ) & 4 5 1 – Clearly, H has no imaginary eigenvalues for )> , so a 1 G .s/ k k1 D a 1 Finally, the Hankel matrix is PQ , so the Hankel norm is ! D 4a2 1 G .s/ #.PQ/ k kH D D 2a p

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–36 Signal Norms

Example 3.11

Consider the following two time signals !

e(t)

1 e1(t)

e2(t)

t

For these two signals, we compute the following values for their ! respective norms:

e1.t/ 1 k k1 D e1.t/ k k2 D1

e2.t/ k k1 D1 e2.t/ 1 k k2 D Computation of signal norms 1. “Sum up” the channels at a given time or frequency using a vector norm (for a scalar signal, take the ) 2. “Sum up” in time or frequency using a temporal norm

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–37

!e!∞

e(t) Area = !e!1

t

We define the temporal p-norm of a time-varying vector as ! 1=p 1 p e.t/ ei .'/ d' k kp D j j i ! Z&1 X The following temporal norms are commonly used: ! – 1-norm in time 1 e.t/ ei .'/ d' k k1 D j j i Z&1 X – 2-norm in time

1 2 e.t/ ei .'/ d' k k2 D j j s i Z&1 X – -norm in time 1 e .t/ sup max ei .'/ k k1 D ' i j j  Ã

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–38 Additional Topics

Toeplitz & Hankel Matrix Forms

Systems theory makes extensive use of Toeplitz and Hankel matrices ! to simplify much of the algebra. Consider the polynomial n.z/: ! 1 m n.z/ n0 n1z& nmz& D C C###C

We define the Toeplitz matrices, €n, Cn for n.z/ from the following ! matrix form:

Cn €n D M Ä n ) where,

n0 00 0 ### n1 n0 0 0 2 ### 3 Cn n2 n1 n0 0 D 6 ### 7 6 : : : : : 7 6 : : : : : 7 6 : 7 6 nm nm 1 nm 2 : 7 6 & & 7 4 5 and : 0nm nm 1 : : : :& : : Mn : : : : : D 2 3 00 0 n0 6 ### 7 4 5

Define the Hankel matrix Hn as !

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–39

n1 n2 n3 nm 1 nm ### & 2 n2 n3 n4 nm 0 3 ### 6 n3 n4 n5 007 6 : : : ###: : : 7 6 : : : : : : 7 Hn 6 7 D 6 n n 0 :00: 7 6 m 1 m 7 6 & : 7 6 nm 00 :007 6 7 6 000:00: 7 6 7 6 : : : : : : 7 6 : : : : : : 7 6 7 4 5

NOTE: The dimension of matrices €n;Cn;Hn are not defined here as ! they are implicit in the context in which they are used.

Toeplitz matrices defined in this manner are often used for changing ! polynomial convolution into a matrix-vector multiplication

Analytic Functions of Matrices

Let A be an n n matrix with eigenvalues "1;"2;:::;"n, and let p.x/ ! % be an infinite series in a scalar variable x, 2 k p.x/ a0 a1x a2x ::: akx ::: D C C C C C – Such an infinite series may converge or diverge depending on the value of x.

EXAMPLES 1 x x2 x3 ::: xk ::: C C C C C C This geometric series converges for all x <1. ! j j

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–40

x2 x3 xk 1 x C C 2Š C 3Š C###C kŠ C### This series converges for all values of x and is in fact, ex. ! These results have analogies for matrices. !

– Let A be an n n matrix with eigenvalues "i . If the infinite series % p.x/ defined above is convergent for each of the n values of "i , then the corresponding matrix inifinite series

2 k 1 k p.A/ a0I a1A a2A akA akA D C C C###C C###D k 0 XD converges.

DEFINITION: A single-valued function f .z/, with z a complex scalar, is ! analytic at a point z0 if and only if its derivative exists at every point in

some neighborhood of z0: Points at which the function is not analytic are called singular points.

RESULT. If a function f .z/ is analytic at every point in some circle , ! in the complex plane, then f .z/ can be represented as a comnvergent power series (Taylor series) at every point z inside ,.

IMPORTANT RESULT. If f .z/ is any function which is analytic withing a ! circle in the complex plane which contains all the eigenvalues "i of A, then a corresponding matrix function f .A/ can be defined by a convergent power series.

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–41 EXAMPLE 3.12 The function e˛x is analytic for all values of x; therefore it has a ! convergent series representation:

˛2x2 ˛3x3 ˛kxk e˛x 1 ˛x D C C 2Š C 3Š C###C kŠ C###

– The corresponding matrix function is

˛2A2 ˛3A3 ˛kAk e˛A I ˛A D C C 2Š C 3Š C###C kŠ C###

EXAMPLE 3.13 At Bt .A B/t e e e C # D if and only if, AB BA. D

OTHER USEFUL REPRESENTATIONS: A3 A5 sin A A D & 3Š C 5Š &### A2 A4 cos A I D & 2Š C 4Š &### A3 A5 sinh A D C 3Š C 5Š C### A2 A4 cosh I D C 2Š C 4Š C### sin2 A cos2 A I C D jA jA jA jA e e& e e& sin A & and cos A C D 2j D 2

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " ECE5580, Matrix Theory and Norms 3–42 Cayley-Hamilton Theorem

If the characteristic polynomial of a matrix A is ! n n 1 n 2 "I A . "/ cn 1" & cn 2" & c1" c0 ."/ j & j D & C & C & C###C C D4 then the corresponding matrix polynomial is n n n 1 n 2 .A/ . 1/ A cn 1A & cn 2A & c1A c0I 4 D & C & C & C###C C CAYLEY-HAMILTON THEOREM. Every matrix satisfies its own characteristic equation. .A/ 0 4 D The proof of this follows by applying a diagonalizing similarity ! transformation to A and substituing into .A/. 4

EXAMPLE 3.14 31 A D " 12#

"I A .3 "/.2 "/ 1 "2 5" 5 j & j D & & & D & C

10 5 31 10 00 .A/ A2 5A 5I 5 5 4 D & C D " 55# & # " 12# C # " 01# D " 00#

Lecture notes prepared by M. Scott Trimboli. Copyright c 2016, M. Scott Trimboli " (mostly blank) (mostly blank)