GLOBAL INITIATIVE OF ACADEMIC NETWORKS Ivan Slapniˇcar MODERN APPLICATIONS OF NUMERICAL LINEAR ALGEBRA METHODS Module B - Eigenvalue and Singular Value Decompositions IIT INDORE, 2016 GLOBAL INITIATIVE OF ACADEMIC NETWORKS Ivan Slapniˇcar MODERN APPLICATIONS OF NUMERICAL LINEAR ALGEBRA METHODS Module B - Eigenvalue and Singular Value Decompositions https://github.com/ivanslapnicar/GIAN-Applied-NLA-Course Cover photo: TU Berlin, Institut f¨urMathematik IIT INDORE, 2016 Contents 1 Eigenvalue Decomposition - Definitions and Facts5 1.1 Prerequisites......................................5 1.2 Competences......................................5 1.3 Selected references...................................5 1.4 General matrices....................................5 1.4.1 Definitions...................................5 1.4.2 Facts.......................................6 1.4.3 Examples....................................7 1.4.4 Example.....................................9 1.4.5 Example..................................... 10 1.4.6 Example - Circulant matrix.......................... 12 1.5 Hermitian and real symmetric matrices........................ 14 1.5.1 Definitions................................... 14 1.5.2 Facts....................................... 14 1.5.3 Example - Hermitian matrix......................... 16 1.5.4 Example - real symmetric matrix....................... 21 1.6 Positive definite matrices............................... 22 1.6.1 Definitions................................... 22 1.6.2 Facts....................................... 22 1.6.3 Example - Positive definite matrix...................... 23 1.6.4 Example - Positive semidefinite matrix.................... 24 1.6.5 Example - Covariance and corellation matrices............... 25 2 Eigenvalue Decomposition - Perturbation Theory 28 2.1 Prerequisites...................................... 28 2.2 Competences...................................... 28 2.3 Norms.......................................... 28 2.3.1 Definitions................................... 28 2.3.2 Examples.................................... 29 2.3.3 Facts....................................... 29 2.4 Errors and condition numbers............................. 32 2.4.1 Definitions................................... 32 2.5 Peturbation bounds.................................. 32 2.5.1 Definitions................................... 32 2.5.2 Facts....................................... 33 2.5.3 Example - Nondiagonalizable matrix..................... 34 2.5.4 Example - Jordan form............................ 36 2.5.5 Example - Normal matrix........................... 40 2.5.6 Example - Hermitian matrix......................... 41 3 Symmetric Eigenvalue Decomposition - Algorithms and Error Analysis 47 3.1 Prerequisites...................................... 47 3.2 Competences...................................... 47 3.3 Backward error and stability............................. 47 1 3.3.1 Definitions................................... 47 3.4 Basic methods..................................... 47 3.4.1 Definitions................................... 47 3.4.2 Facts....................................... 48 3.4.3 Examples.................................... 49 3.5 Tridiagonalization................................... 52 3.5.1 Facts....................................... 52 3.6 Tridiagonal QR method................................ 57 3.6.1 Definition.................................... 57 3.6.2 Facts....................................... 57 3.6.3 Examples.................................... 58 3.6.4 Computing the eigenvectors.......................... 60 3.6.5 Symmetric QR method............................ 63 4 Symmetric Eigenvalue Decomposition - Algorithms for Tridiagonal Matrices 65 4.1 Prerequisites...................................... 65 4.2 Competences...................................... 65 4.3 Bisection and inverse iteration............................. 65 4.3.1 Facts....................................... 65 4.4 Divide-and-conquer................................... 67 4.4.1 Facts....................................... 67 4.5 MRRR.......................................... 70 5 Symmetric Eigenvalue Decomposition - Jacobi Method and High Relative Accuracy 72 5.1 Prerequisites...................................... 72 5.2 Competences...................................... 72 5.3 Jacobi method..................................... 72 5.3.1 Definitions................................... 72 5.3.2 Facts....................................... 73 5.3.3 Examples.................................... 73 5.4 Relative perturbation theory............................. 76 5.4.1 Definition.................................... 76 5.4.2 Facts....................................... 76 5.4.3 Example - Scaled matrix............................ 77 5.5 Indefinite matrices................................... 79 5.5.1 Definition.................................... 79 5.5.2 Facts....................................... 79 5.6 # Symmetric Eigenvalue Decomposition - Lanczos Method............ 79 5.7 Prerequisites...................................... 80 5.8 Competences...................................... 80 5.9 Lanczos method.................................... 80 5.9.1 Definitions................................... 80 5.9.2 Facts....................................... 80 5.9.3 Examples.................................... 81 2 5.9.4 Operator version................................ 90 5.9.5 Sparse matrices................................. 93 6 Singular Value Decomposition - Definitions and Facts 95 6.1 Prerequisites...................................... 95 6.2 Competences...................................... 95 6.3 Selected references................................... 95 6.4 Singular value problems................................ 95 6.4.1 Definitions................................... 95 6.4.2 Facts....................................... 95 6.4.3 Example - Symbolic computation....................... 97 6.4.4 Example - Random complex matrix..................... 98 6.4.5 Example - Random real matrix........................ 100 6.5 Matrix approximation................................. 102 7 Singular Value Decomposition - Perturbation Theory 104 7.1 Prerequisites...................................... 104 7.2 Competences...................................... 104 7.3 Peturbation bounds.................................. 104 7.3.1 Definitions................................... 104 7.3.2 Facts....................................... 104 7.3.3 Example..................................... 105 7.4 Relative perturbation theory............................. 108 7.4.1 Definitions................................... 108 7.4.2 Facts....................................... 108 7.4.3 Example - Bidiagonal matrix......................... 109 8 Singular Value Decomposition - Algorithms and Error Analysis 113 8.1 Prerequisites...................................... 113 8.2 Competences...................................... 113 8.3 Basics.......................................... 113 8.3.1 Definitions................................... 113 8.3.2 Facts....................................... 113 8.4 Bidiagonalization.................................... 114 8.4.1 Facts....................................... 114 8.5 Bidiagonal QR method................................. 118 8.5.1 Facts....................................... 118 8.5.2 Examples.................................... 119 8.6 QR method....................................... 122 9 Singular Value Decomposition - Jacobi and Lanczos Methods 125 9.1 Prerequisites...................................... 125 9.2 Competences...................................... 125 9.3 One-sided Jacobi method............................... 125 9.3.1 Definition.................................... 125 9.3.2 Facts....................................... 125 3 9.3.3 Example - Standard matrix.......................... 126 9.3.4 Example - Strongly scaled matrix....................... 128 9.4 Lanczos method.................................... 132 9.4.1 Example - Large matrix............................ 135 9.4.2 Example - Very large sparse matrix..................... 135 10 Algorithms for Structured Matrices 138 10.1 Prerequisites...................................... 138 10.2 Competences...................................... 138 10.3 Rank revealing decompositions............................ 138 10.3.1 Definitions................................... 138 10.3.2 Facts....................................... 138 10.3.3 Example - Positive definite matrix...................... 140 10.3.4 Example - Hilbert matrix........................... 142 10.4 Symmetric arrowhead and DPR1 matrices...................... 149 10.4.1 Definitions................................... 149 10.4.2 Facts on arrowhead matrices......................... 149 10.4.3 Example - Random arrowhead matrix.................... 150 10.4.4 Example - Numerically demanding matrix.................. 152 10.4.5 Facts on DPR1 matrices............................ 152 10.4.6 Example - Random DPR1 matrix....................... 153 10.4.7 Example - Numerically demanding matrix.................. 153 11 Updating the SVD 155 11.1 Prerequisites...................................... 155 11.2 Competences...................................... 155 11.3 Facts........................................... 155 11.3.1 Example - Adding row to a tall matrix.................... 156 11.3.2 Example - Adding row to a flat matrix.................... 158 11.3.3 Example - Adding columns.......................... 158 11.3.4 Example - Updating a