LECTURE NOTES on LINEAR ALGEBRA, LINEAR DIFFERENTIAL EQUATIONS, and LINEAR SYSTEMS A. S. Morse August 29, 2020 Contents I Linear Algebra and Linear Differential Equations 1 1 Fields and Polynomial Rings 3 2 Matrix Algebra 7 2.1 Matrices:Notation .................................. ....... 7 2.2 TypesofMatrices ................................... ....... 8 2.3 Determinants:Review ................................ ........ 9 2.4 MatrixRank ........................................ ..... 9 2.5 BasicMatrixOperations............................... ........ 10 2.5.1 Transposition..................................... .... 10 2.5.2 Scalar Multiplication . ... 11 2.5.3 Addition.......................................... .. 11 2.5.4 Multiplication . .. 12 2.5.5 Linear Algebraic Equations . ..... 14 2.5.6 MatrixInversion ................................... .... 15 2.6 LinearRecursionEquations .. .. .. .. .. .. .. .. .. .. .. .. ......... 17 2.7 Invariants under Premultiplication . .......... 19 2.7.1 ElementaryRowOperations. ...... 20 2.7.2 EchelonForm...................................... ... 21 2.7.3 Gauss Elimination . .. 22 2.7.4 Elementary Column Operations . ...... 27 2.7.5 Equivalent Matrices . .... 28 3 Linear Algebra 31 i ii CONTENTS 3.1 LinearVectorSpaces................................ ......... 31 3.1.1 Subspaces....................................... .... 32 3.1.2 SubspaceOperations............................... ...... 33 3.1.3 Distributative Rule . .... 34 3.1.4 IndependentSubspacesandDirectSum . ......... 36 3.1.5 Linear Combination and Span . ..... 36 3.1.6 LinearIndependence............................... ...... 36 3.1.7 Basis............................................ .. 37 3.1.8 Dimension......................................... .. 40 3.2 Functions ......................................... ...... 40 3.2.1 LinearTransformations .. .. .. .. .. .. .. .. .. .. .. .. ...... 41 3.2.2 Operations With Linear Transformations . ........ 42 3.2.3 Representations of Linear Transformations . ........... 42 3.2.4 CoordinateTransformations. ........ 43 3.2.5 Defining a Linear Transformation on a Basis . ....... 44 3.2.6 Linear Equations: Existence, Image, Epimorphism . .......... 45 3.2.7 Linear Equations: Uniqueness, Kernel, Monomorphism . ........... 47 3.2.8 IsomorphismsandIsomorphicVectorSpaces . .......... 48 3.2.9 Endomorphisms and Similar Matrices . ..... 48 4 Basic Concepts from Analysis 51 4.1 NormedVectorSpaces ............................... ......... 51 4.1.1 OpenSets........................................ ... 53 4.2 Continuous and Differentiable Functions . ........... 53 4.3 Convergence...................................... ........ 55 5 Ordinary Differential Equations - First concepts 57 5.1 TypesofEquation ................................... ....... 57 5.2 Modeling ........................................... .... 59 5.3 StateSpaceSystems ................................ ......... 62 5.3.1 ConversiontoState-SpaceForm . ........ 63 5.4 Initial Value Problem . ...... 65 CONTENTS iii 5.5 PicardIterations ................................... ........ 68 5.5.1 Three Relationships . .... 69 5.5.2 Convergence ..................................... .... 70 5.5.3 Uniqueness ....................................... ... 72 5.5.4 SummaryofFindings ................................. ... 74 5.6 TheConceptofState................................ ......... 77 5.6.1 Trajectories, Equilibrium Points and Periodic Orbits . ......... 78 5.6.2 Time-Invariant Dynamical Systems . ........ 80 5.7 Simulation.......................................... ..... 81 5.7.1 NumericalTechniques ............................... ..... 81 5.7.2 Analog Simulation . .. 83 5.7.3 ProgrammingDiagrams ............................... .... 84 6 Linear Differential Equations 87 6.1 Linearization ....................................... ...... 87 6.2 Linear Homogenous Differential Equations . ............ 90 6.2.1 State Transition Matrix . ..... 91 6.2.2 TheMatrixExponential.............................. ..... 92 6.2.3 State Transition Matrix Properties . ........ 93 6.3 Forced Linear Differential Equations . ........... 96 6.3.1 Variation of Constants Formula . ....... 96 6.4 Periodic Homegeneous Differential Equations . ............ 97 6.4.1 CoordinateTransformations. ........ 98 6.4.2 LyapunovTransformations . ....... 99 6.4.3 Floquet’sTheorem.................................. 100 7 Matrix Similarity 103 7.1 Motivation ......................................... ..... 103 7.1.1 The Matrix Exponential of a Diagonal Matrix . ....... 104 7.1.2 The State Transition Matrix of a Diagonalizable Matrix . ........ 104 7.2 Eigenvalues ........................................ ...... 105 7.2.1 Characteristic Polynomial . ...... 105 iv CONTENTS 7.2.2 ComputationofEigenvectors . ....... 106 7.2.3 Semisimple Matrices . 107 7.2.4 Eigenvectors and Differential Equations . ......... 108 7.3 A -InvariantSubspaces ................................. ...... 109 7.3.1 Direct Sum Decomposition of IKn ..............................111 7.4 Similarity Invariants . ....... 112 7.5 TheCayley-HamiltonTheorem . ........ 112 7.6 Minimal Polynomial . ..... 113 7.6.1 The Miminal Polynomial of a Vector . ..... 114 7.6.2 The Minimal Polynomial of a Finite Set of Vectors . ....... 114 7.6.3 The Minimal Polynomial of a Subspace . ..... 114 7.6.4 The Minimal Polynomial of A ................................115 7.6.5 A Similarity Invariant . 115 7.7 AUsefulformula .................................... ....... 116 7.7.1 Application . 117 7.8 CyclicSubspaces .................................... ....... 117 7.8.1 CompanionForms ................................... 118 7.8.2 RationalCanonicalForm ............................. ..... 120 7.9 CoprimeDecompositions............................... ........ 121 7.9.1 TheJordonNormalForm ............................. ..... 124 7.10 TheStructureoftheMatrixExponential. .............. 126 7.10.1 RealMatrices.................................... ..... 128 7.10.2 AsymptoticBehavior.. .. .. .. .. .. .. .. .. .. .. .. .. ...... 129 7.11 LinearRecursionEquations . .......... 129 8 Inner Product Spaces 131 8.1 Definition .......................................... ..... 131 8.1.1 Triangle Inequality . 132 8.2 Orthogonality...................................... ....... 133 8.2.1 OrthogonalVectors............................... ....... 133 8.2.2 OrthonormalSets ................................. ..... 133 CONTENTS v 8.3 Gram’sCriterion ..................................... ...... 134 8.4 Cauchy-SchwartzInequality . ........... 135 8.5 OrthogonalizationofaSetofVectors. ............ 136 8.6 OrthogonalProjections ............................. .......... 137 8.7 Bessel’s Inequality . ....... 138 8.8 LeastSquares..................................... ........ 139 9 Normal Matrices 143 9.1 AlgebraicGroups.................................... ....... 143 9.1.1 OrthogonalGroup ................................. ..... 144 9.1.2 UnitaryGroup ..................................... 145 9.2 AdjointMatrix ...................................... ...... 145 9.2.1 OrthogonalComplement. ...... 145 9.2.2 Properties of Adjoint Matrices . ....... 146 9.3 NormalMatrices .................................... ....... 146 9.3.1 Diagonalization of a Normal Matrix . ..... 147 9.4 Hermitian, Unitary, Orthogonal and Symmetric Matrices . ...............149 9.4.1 Hermitian Matrices . 149 9.4.2 UnitaryMatrices................................... 150 9.4.3 OrthogonalMatrices ............................... ...... 150 9.4.4 SymmetricMatrices ................................. 150 9.5 RealQuadraticForms ................................ ........ 152 9.5.1 SymmetricQuadraticForms . ..... 153 9.5.2 Change of Variables: Congruent Matrices . ......... 153 9.5.3 Reduction to Principal Axes . ...... 153 9.5.4 SumofSquares.................................... 154 9.6 Positive Definite Quadratic Forms . ......... 156 9.6.1 Conditions for Positive Definiteness . ....... 156 9.6.2 MatrixSquareRoots................................ ..... 157 9.7 Simultaneous Diagonalization . ........ 158 9.7.1 Constrained Optimization . ..... 159 vi CONTENTS 9.7.2 ASpecialCase ..................................... 159 II Linear Systems 163 10 Introductory Concepts 165 10.1 LinearSystems.................................... ........ 165 10.2 Continuous-TimeLinearSystems . ........... 166 10.2.1 Time-Invaraint, Continuous-Time, Linear Systems . ............ 166 10.3 Discrete-TimeLinearSystems. ........... 168 10.3.1 Time-Invariant Discrete Time Systems . ......... 168 10.3.2 SampledDataSystems. .. .. .. .. .. .. .. .. .. .. .. .. ...... 169 10.4 TheConceptofaRealization . ......... 170 10.4.1 Existence of Continuous-Time Linear Realizations . .......... 171 11 Controllability 173 11.1ReachableStates .................................. ......... 173 11.2 Controllability . ...... 175 11.2.1 Controllability Reduction . ..... 176 11.3 Time-Invariant Continuus-Time Linear Systems . .............. 178 11.3.1 Properties of the Controllable Space of (A, B).......................181 11.3.2 Control Reduction for Time-Invariant Systems . ............ 182 11.3.3 Controllable Decomposition . ...... 184 12 Observability 187 12.1UnobservableStates ............................... .......... 187 12.2Observability ...................................... ....... 189 12.2.1 Observability Reduction . ...... 190 12.3MinimalSystems .................................... ....... 192 12.4 Time-Invariant Continuous-Time Linear Systems . .............. 193 12.4.1 Properties of the Unobservable Space