MAE280A Linear Systems Theory http://numbat.ucsd.edu/~bob/linearsystems Regular Professor Bob Bitmead [email protected], 858 822 3477, Jacobs Hall 1609 email policy: If you send me an email I will read it. If it takes longer to deal with it than it took you to write it I will generally not deal with it TA Behrooz Amini [email protected], Jacobs Hall 2103A Reader Yunfeng (Joe) Jiang
Known/planned absence (Taipei) Thursday October 26 Behrooz will take class
Midterm Thursday November 2 in class Final Thursday December 7 in class
Grade determined by homework, midterm, final 1 MAE280A Linear Systems Theory João Hespanha, Linear Systems Theory, 2009 $50.54-$88.45
Alan Laub, Matrix Analysis for Scientists & Engineers, 2005, $33.00
Tom Bewley, Numerical Renaissance, various, $000.00 http://numerical-renaissance.com/NR.pdf
Horn & Johnson Matrix Analysis, 2012, $62.69
2 Bob Skelton quote re MAE280A Bob developed MAE280A
“I do not know whether this is a linear algebra course with strong control overtones or a control course with strong linear algebra content”
Still nobody knows, me included. I shall begin with a review of linear algebra Then move onto the material associated with systems The central thrust is the state-space description of linear dynamic systems This course is the first course in systems and control and is therefore meant to be rigorous I will do my best to lighten it up a little We shall follow Laub for a while to get started
3 My favorite subject: Notation n R denotes the set of n-tuples of real numbers represented as column vectors x1 x2 2 3 n x x R x = 3 2 , 6 . 7 6 . 7 6 7 6x 7 6 n7 4 5 x i R are the elements of x 2 n A row vector yT is the transpose of vector y in R n T 1 For x, y R , x y is a scalar (in R ) and xy T is an nxn matrix n 2 C denotes the set of n-tuples of complex numbers represented as column vectors m n m n R ⇥ denotes the set of real mxn matrices. Similarly for C ⇥ m n The element in the i th row and j th column of M R ⇥ is denoted mij 2
4 Specially shaped matrices n n n n A matrix in R ⇥ or in C ⇥ , i.e. square matrices with real or complex elements is diagonal if a = 0 for i = j ij 6 upper triangular if aij = 0 for i>j
lower triangular if aij = 0 for i tridiagonal if aij = 0 for i j > 1 | | upper Hessenberg if aij = 0 for i j>1 lower Hessenberg if a = 0 for j i>1 ij Toeplitz if aij = ai+1,j+1 = ai j Hankel if aij = ai 1,j+1 The same designations hold for block matrices. So the (n+m)x(n+m) block AB n n n m m m matrix with A R ⇥ , B R ⇥ , C R ⇥ is block upper 0 C 2 2 2 triangular 5 More revision T T The transpose of a matrix A is denoted A and has (A )ij = Aji n m T m n If A R ⇥ then A R ⇥ 2 2 m n The hermitian transpose or conjugate transpose of A C ⇥ is denoted H H 2 A or A ⇤ and satisfies (A )ij = aji A matrix A is symmetric if AT=A and is hermitian if AH=A Theorem 1.3 Laub m n m Let U = u 1 u 2 ... u n R ⇥ with u i R and 2 2 p n p V = ⇥ v 1 v 2 ... v n ⇤ R ⇥ with v i R then n 2 2 ⇥ T ⇤ T m p UV = uiv R ⇥ i 2 i=1 X This explores matrix multiplication as linear combinations of the columns of U and of the rows of VT 3 987 50 9 8 7 Ax = 2 = =3 +2 +1 6542 3 32 6 5 4 1 6 4 5 Inner products and orthogonality n n T For vectors x, y R the inner product is given by 7 Determinants n n For A R ⇥ we denote the determinant of A as det A 2 Properties of determinants A zero row or column of A causes det A=0 Interchanging a row or a column of A changes the sign of det A Multiplying a row or a column of A by a multiplies det A by a Multiplying a row by a scalar and adding it to another row does not change the determinant ... ditto for columns det AT = det A and det AH = det A det AB = det A det B whence det A-1=(det A)-1 If A is diagonal then det A=a11a22...ann ditto if A is upper or lower triangular AB 1 1 det =detA det(D CA B)=detD det(A BD C) CD This uses the Schur complement AB 1 The Schur complement of A in is D CA B CD AB 1 The Schur complement of D in is A BD C CD 8 Schur complement formulæ n n m m Suppose A R ⇥ and D R ⇥ with D invertible 2 2 Then 1 1 AB In BD A BD C 0 In 0 = 1 CD 0 I 0 D D CI m m So AB 1 det =det(A BD C)detD CD Further 1 1 1 1 AB In 0 (A BD C) 0 In BD = 1 1 CD D CI 0 D I m m If A is invertible then 1 AB In 0 A 0 In A B = 1 1 CD CA I 0 D CA B I m m 9 Fields A field F is a set together with two binary operations + ,. : F F F such that ⇥ ! (A1)↵ +( + )=(↵ + )+ for all ↵, , F 2 (A2) there exists an element 0 F such that ↵ +0=↵ for all ↵ F 2 2 (A3) for all ↵ F there exists an element ↵ F such that ↵ +( ↵)=0 2 2 (A4)↵ + = + ↵ for all ↵, F 2 (M1)↵.( . )=(↵. ). for all ↵, , F 2 (M2)there exists an element 1 F such that ↵.1=↵ for all ↵ F 2 2 1 (M3) for all ↵ F, ↵ = 0 there exists an element ↵ F 2 6 2 1 such that ↵.↵ = 1 for all ↵ F 2 (M4)↵. = .↵ for all ↵, F 2 (D)↵.( + )=↵. + ↵. for all ↵, , F 2 Examples R or C with ordinary addition and multiplication Ra[ x ] the field of rational functions (ratios of polynomials) in x m n Not R ⇥ since (M1) fails for m = n and (M3) and (M4) do not hold in general 6 10 Vector Spaces A vector space over a field F is a set together with two binary operations V +: and . : F such that V ⇥ V ! V ⇥ V ! V (V1)( , +) is an abelian group [(A1-A4) hold] V (V2)(↵. ).v = ↵.( .v) for all ↵, F and v 2 2 V (V3)(↵ + ).v = ↵.v + .v for all ↵, F and v 2 2 V (V4)↵.(v + w)=↵.v + ↵.w for all ↵ F and v, w 2 2 V Niels Henrik Abel (V5)1.v = v for all v (1 F) 2 V 2 Subspaces Let ( , F ) be a vector space and let . Then ( , F ) is a subspace V W of ( , F ) if and only if ( , F ) itself isW a✓ vectorV space V W equivalently (↵.w1 + .w2) for all ↵, F,w1,w2 2 W 2 2 W same operations but we require closure 11 Vector space examples Vector spaces n ( R , R ) with usual addition and scalar multiplication m n ( R ⇥ , R ) with usual addition and scalar multiplication ( , F ) arbitrary vector space and an arbitrary set. Then ( , ) the set V of all functions mapping to D is a vector space with additionD V D V (f + g)(d)=f(d)+g(d) for all d and for all f,g 2 D 2 and scalar multiplication (↵f)(d)=↵f(d) for all ↵ F, for all d for all f 2 2 D 2 n e.g. =[ t 0 , ) , ( , F )=( R , R ) and the functions are continuous D n n 1 V Let A R ⇥ then x ( t ): x ˙ ( t )= Ax ( t ) is a vector space (of dimension n) 2 { } Subspaces n n ( R ⇥ , R ) and is the set of symmetric nxn matrices and is a subspace n nW = A R ⇥ : A orthogonal is not a subspace W { 2 } 2 c ( , F )=( R , R ) and ↵ , = v : v = ; c R is a subspace V W ↵c + 2 only if = 0 or ↵ = ⇢ 1 12 Linear independence Let X be a collection of vectors v ,v ,... in some vector space { 1 2 } X is a linearly independent set of vectors if and only if for any collection of k distinct elements of and for any scalars ↵1, ↵2,...,↵k X ↵1v1 + ↵2v2 + + ↵kvk =0 = ↵1 = ↵1 = = ↵k =0 n ··· ) ··· n k Let v i R ,i =1 , 2 ,...,k and consider V = v1 v2 ... vk R ⇥ 2 2 then the linear dependence of V equates with the existence of a nonzero k ⇥ ⇤ vector a R such that Va=0 2 The span of X= v 1 ,v 2 ,...,v k is defined as { } Sp(X)= v : v = ↵1v1 + ↵2v2 + + ↵kvk; ↵i F,vi X, k N { ··· 2 2 2 } A set of vectors X is a basis for vector space if X is linearly independent and Sp(X)= V V Examples: 1 4 7 2 , 5 , 8 is linearly independent and is a basis of 3 82 3 2 3 2 39 R < 3 6 10 = 14 7 det 25 8 = 3 4 5 4 5 4 5 2 3 : 1 4 2 ; 3 6 10 2 , 5 , 1 is linearly dependent 4 5 82 3 2 3 2 39 142 3 6 0 det 251=0 < = 23603 4 5 4 5 4 5 4 5 13 : ; Bases and Dimension If v 1 ,v 2 ,...,v k is a basis for a vector space then for each vector v { } V 2 V there is a unique set of scalar values ↵ 1 , ↵ 2 ,..., ↵ k such that v = ↵ v + ↵ v + + ↵ v 1 1 2 2 ··· k k The scalars are called the components or coordinates of v with respect to this basis 1 1 Example: Consider the basis of 2 : , R 2 1 1 1 1 ⇢ The vector =3 +4 2 2 1 1 11 3 Note that this might be written as = 2 2 1 4 1 So to find the coordinates we solve 3 11 1 = 4 2 1 2 Theorem: The number of elements in a basis of a vector space is independent of the particular basis considered. This number is called the dimension of the vector space A vector space can be finite-dimensional or infinite-dimensional 14 Linear Transformations ( , F ) and ( , F ) are two vector spaces. Then : is a linear V transformationW if and only if L V ! W (↵v1 + v2)=↵ v1 + v2 for all ↵, F and for all v1,v2 L L L 2 2 V The vector space is called the domain of the transformation while the vector space V , into which it maps, is called the co-domainL W Examples F = R, = = PC[t0, ) V W 1 : PC[t0, ) PC[t0, ) given by L 1 ! 1 t (t ⌧) v(t) w(t)=( v)(t)= e v(⌧) d⌧ 7! L Zt0 m n m m m n m n F = R, = = R ⇥ . Fix M R ⇥ . : R ⇥ R ⇥ given by V W 2 L ! X Y = X = MX 7! L 15 Matrix representation of linear transformations Linear transformations between vector spaces with specific bases can be represented in matrix form Suppose :( , F ) ( , F ) is linear and suppose that v 1 ,v 2 ,...,v n L V ! W { } is a basis for and w 1 ,w 2 ,...,w m is a basis for V { } W ↵11 ... ↵1n . . Define the matrix A = Mat( ) as A = . . L 2 . . 3 ↵ ... ↵ 6 m1 mn7 where the basis vectors map into 4 -basis coordinates5 as V W v = ↵ w + ↵ w + + ↵ w = Wa with W = w w ... w L i 1i 1 2i 2 ··· mi m i 1 2 m th and ai is the i column of A ⇥ ⇤ Now consider a general vector v = ⇠ v + ⇠ v + + ⇠ v = Vx 1 1 2 2 ··· n n Vx= v = ⇠ v + + ⇠ v L L 1L 1 ··· nL n = ⇠ Wa + + ⇠ Wa 1 1 ··· n n = W Ax = A LV W n m If we choose = R , = R , v i ; i =1 ,..,n and w j ; j =1 ,..,m as the naturalV basis, thenW = { A becomes } = A{ } LV W L m n n m We think of A R ⇥ as the linear transformation R R 16 2 ! Linear transformations continued We can compose linear transformations B : and A : to yields a linear transformation C : Uwith! C=ABV givenV by! matrixW multiplication U ! W Let A : then the range of A, ( A ) , or the image of A,Im ( A ) , is the set V ! W R w : w = Av for some v { 2 W 2 V} The nullspace of A, ( A ) , or kernel of A, Ker ( A ) , is the set v : Av =0 N { 2 V } (A) , (A) n R ✓ W N ✓ V Let R then the orthogonal complement of is the set S ✓ n T S ? = v R : v s = 0 for all s S { 2 2 S} n n n Let , R then ? R ? = R ( ?)? = R S ✓ S ✓ S S S S if and only if ? ? ( + )? = ? ? R ✓ S S ✓ R R S R \ S ( )? = ? + ? R \ S R S Here + = r + s : r ,s = v : v and v R S { 2 R 2 S} R \ S { 2 R 2 S} = the direct sum of and if = 0 and = + T R S R S R \ S T R S 17 Still more linear transformations n m T T Let A : R R then ( A ) ? = ( A ) and (A)? = (A ) ! N R R N (this holds for finite-dimensional vector spaces only) Proof: Take x ( A ) So Ax=0 or yTAx=0 for all y. So (ATy)Tx=0 and x is 2 N T orthogonal to all vectors ATy, i.e. x (A )? 2 R n m n For A : R R the set v R : Av =0 is called the right nullspace of A ! m { T 2 } and the set w R : w A =0 is called the left nullspace of A { 2 } The right nullspace is ( A ) The left nullspace is (AT ) N N n m Let A : R R ! n Every vector in the domain v R can be written uniquely as 2 T v = x + y with x (A) and y (A)? = (A ) 2 N 2 N R n T That is R = (A) (A ) N R m Every vector in the co-domain w R can be written uniquely as 2 T w = x + y with x (A) and y (A)? = (A ) 2 R 2 R N m T That is R = (A) (A ) R N 18 Four fundamental subspaces m n n m Consider A R ⇥ or equivalently A : R R 2 ! Many properties of A can be associated with the four subspaces (A), (A)?, (A), (A)? R R N N Let A : then A is surjective/onto/epic if (A)= V ! W R W A is injective/one-to-one/monic if (A)=0 N The rank of A is the dimension of (A) R This is the maximal number of independent columns also called column rank dim ( A T ) is the maximal number of independent rows R also called row rank The nullity of A is the dimension of (A) N dim (A)=dimN(A)? R column rank = row rank = rank m n Matrix A R ⇥ is surjective if rank A =m. It is injective of rank A=n 2 Square matrix A is invertible if it is bijective, i.e. injective and surjective Number of columns n = rank + nullity 19 Eigenvalues and eigenvectors n n n A right eigenvector of A C ⇥ is a nonzero vector x C such that Ax = x 2 2 for scalar C called an eigenvalue 2 n n n H H A left eigenvector of A C ⇥ is a nonzero vector y C such that y A = µh 2 2 for scalar µ C 2 The scalar polynomial ⇡ ( )=det( I A ) is called the characteristic polynomial. The eigenvalues of A are the roots of ⇡( ) n n Cayley-Hamilton Theorem: For any A C ⇥ , ⇡(A)=0n n 2 ⇥ n n The minimum polynomial of A C ⇥ is the least degree (scalar) polynomial ↵ ( ) such that ↵(A)=0 2 Eigenvalues, but not eigenvectors, are invariant under a similarity 1 transformation, i.e. [T AT ] = [A] { i } { i } 20 Jordan Canonical Form n n For all A C ⇥ with eigenvalues 1 ,..., n C , not necessarily distinct, 2 n {n } 2 there exists invertible X C ⇥ such that 1 2 X AX = J = blockdiag(J1,...,Jq) where each of the Jordan block matrices J1, ..., Jq is of the form i 10... 0 0 i 100 2 3 ...... i . . 6 7 ki ki Ji = 6 . 7 C ⇥ 6 .. 107 2 6 7 6 . . 7 6 . .. 1 7 6 i 7 6 0 ...... 0 7 6 i7 4 5 q and ki = n i=1 If the eigenvaluesX of A are distinct then the Jordan blocks are 1x1 and the columns of matrix X are the right eigenvectors of A 21 Other Matrix Decompositions n n 1 We have just seen the Jordan decomposition for A C ⇥ : X AX = J 2 There are more: m n The QR Decomposition for A R ⇥ : A=QR with Q orthogonal (mxm) and R upper triangular (mx2n) m n For A C ⇥ : A=QR with Q unitary and R upper triangular 2 m n T The Singular Value Decomposition forA R ⇥ : A = U⌃V m m n n 2 with U R ⇥ and V R ⇥ orthogonal and where 2 2 S 0 m n ⌃ = R ⇥ 002 r r with S = diag( 1 ,..., r ) R ⇥ where r = rank(A) and the singular 2 values i satisfy > 0 { } 1 2 ··· r m n H For A C ⇥ we have A = U ⌃ V with U, V unitary and the same 2 properties for ⌃ Alternatively we can write S 0 V T A = U U 1 = U SV T =(m r)(r r)(r n) 1 2 00 V T 1 1 ⇥ ⇥ ⇥ 2 ⇥ ⇤ 22 SVD continued The SVD writes A = U ⌃ V T with U and V orthogonal The squared singular values are the ordered eigenvalues of AA T or of A T A 2 = (AAT ) = (AT A) { i } { i } { i } The columns of U and V are called the left and right singular vectors T They satisfy Av i = i u i and A ui = ivi Since U and V are orthogonal their rows are orthonormal and we can write r T A = iuivi j=1 We also have the followingX observations T T (U )= (A)= (A )? (U )= (A)? = (A ) R 1 R N R 2 R N T T (V1)= (A)? = (A ) (V )= (A)= (A )? R N R R 2 N R 23 Moore-Penrose Pseudoinverse S 1 0 From the SVD A = U ⌃ V T we can define a matrix A+ = V U T 00 This is called the Moore-Penrose pseudoinverse of A It satisfies: AA+A = A A+AA+ = A+ (AA+)T = AA+ (A+A)T = A+A (AT )+ =(A+)T A+ =(AT A)+AT = AT (AAT )+ (A+)+ = A (AT A)+ = A+(AT )+ (AAT )+ =(AT )+A+ (A+)= (AT )= (A+A)= (AT A) R R R R (A+)= (AA+)= ((AAT )+)= (AAT )= (AT ) N N N N N Note that in general ( AB ) + = B + A + and AkA+ = A+Ak 6 6 24 Definite Matrices n n Symmetric matrix A R ⇥ 2 T n is positive definite if x Ax > 0 for all nonzero x R 2 is nonnegative definite (positive semidefinite) if xT Ax 0, x =0 8 6 is negative definite if -A is positive definite is nonpositive definite or negative semidefinite if -A is nonnegative definite is indefinite otherwise A symmetric matrix A is positive (nonnegative) definite iff any of the following equivalent conditions holds The determinants of all the leading principal minors of A are positive (nonnegative) The eigenvalues of A are positive (nonnegative) A can be written in the form MTM where M is nonsingular (has rank equal to the rank of A) A positive definite matrix A has a positive definite square root A positive definite matrix A can be written as A=LLT where L is lower triangular. This is called the Cholesky Decomposition. 25