DOCSLIB.ORG

Fundamentals of Numerical Linear Algebra

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Fundamentals of Numerical Linear Algebra Seongjai Kim Department of Mathematics and Statistics Mississippi State University Mississippi State, MS 39762 USA Email: [email protected] Updated: August 12, 2021 Seongjai Kim, Department of Mathematics and Statistics, Mississippi State University, Mississippi State, MS 39762-5921 USA Email: [email protected]. The work of the author is supported in part by NSF grant DMS-1228337. Prologue In organizing this lecture note, I am indebted by Demmel [4], Golub and Van Loan [6], Varga [10], and Watkins [12], among others. Currently the lecture note is not fully grown up; other useful techniques would be soon incorporated. Any questions, suggestions, comments will be deeply appreciated. i ii Contents 1 Introduction to Matrix Analysis1 1.1. Basic Algorithms and Notation...................2 1.1.1. Matrix notation........................2 1.1.2. Matrix operations.......................2 1.1.3. Vector notation........................3 1.1.4. Vector operations.......................4 1.1.5. Matrix-vector multiplication & “gaxpy"..........4 1.1.6. The colon notation......................5 1.1.7. Flops..............................6 1.2. Vector and Matrix Norms......................7 1.2.1. Vector norms..........................7 1.2.2. Matrix norms.........................9 1.2.3. Condition numbers...................... 12 1.3. Numerical Stability.......................... 16 1.3.1. Stability in numerical linear algebra............ 17 1.3.2. Stability in numerical ODEs/PDEs............. 18 1.4. Homework............................... 19 2 Gauss Elimination and Its Variants 21 2.1. Systems of Linear Equations.................... 22 2.1.1. Nonsingular matrices..................... 24 2.1.2. Numerical solutions of differential equations....... 26 2.2. Triangular Systems.......................... 30 2.2.1. Lower-triangular systems.................. 31 2.2.2. Upper-triangular systems.................. 33 iii iv Contents 2.3. Gauss Elimination.......................... 34 2.3.1. Gauss elimination without row interchanges....... 36 2.3.2. Solving linear systems by LU factorization........ 39 2.3.3. Gauss elimination with pivoting.............. 49 2.3.4. Calculating A−1 ........................ 56 2.4. Special Linear Systems........................ 58 2.4.1. Symmetric positive definite (SPD) matrices........ 58 2.4.2. LDLT and Cholesky factorizations............. 63 2.4.3. M-matrices and Stieltjes matrices............. 67 2.5. Homework............................... 68 3 The Least-Squares Problem 71 3.1. The Discrete Least-Squares Problem................ 72 3.1.1. Normal equations....................... 74 3.2. LS Solution by QR Decomposition................. 78 3.2.1. Gram-Schmidt process.................... 78 3.2.2. QR decomposition, by Gram-Schmidt process....... 81 3.2.3. Application to LS solution.................. 91 3.3. Orthogonal Matrices......................... 94 3.3.1. Householder reflection.................... 95 3.3.2. QR decomposition by reflectors............... 106 3.3.3. Solving LS problems by Householder reflectors...... 111 3.3.4. Givens rotation........................ 113 3.3.5. QR factorization by Givens rotation............ 119 3.3.6. Error propagation....................... 122 3.4. Homework............................... 125 4 Singular Value Decomposition (SVD) 127 4.1. Introduction to the SVD....................... 128 4.1.1. Algebraic Interpretation of the SVD............ 131 4.1.2. Geometric Interpretation of the SVD............ 133 4.1.3. Properties of the SVD..................... 135 4.1.4. Computation of the SVD................... 141 Contents v 4.1.5. Numerical rank........................ 145 4.2. Applications of the SVD....................... 147 4.2.1. Image compression...................... 148 4.2.2. Rank-deficient least-squares problems........... 151 4.2.3. Principal component analysis (PCA)............ 158 4.2.4. Image denoising........................ 165 4.3. Homework............................... 168 5 Eigenvalue Problems 171 5.1. Definitions and Canonical Forms.................. 172 5.2. Perturbation Theory: Bauer-Fike Theorem............ 177 5.3. Gerschgorin’s Theorem........................ 182 5.4. Taussky’s Theorem and Irreducibility............... 185 5.5. Nonsymmetric Eigenvalue Problems................ 189 5.5.1. Power method......................... 190 5.5.2. Inverse power method.................... 195 5.5.3. Orthogonal/Subspace iteration............... 198 5.5.4. QR iteration.......................... 201 5.5.5. Making QR iteration practical................ 203 5.5.6. Hessenberg reduction..................... 205 5.5.7. Tridiagonal and bidiagonal reduction........... 214 5.5.8. QR iteration with implicit shifts – Implicit Q theorem.. 221 5.6. Symmetric Eigenvalue Problems.................. 232 5.6.1. Algorithms for the symmetric eigenproblem – Direct meth- ods................................ 232 5.6.2. Tridiagonal QR iteration................... 235 5.6.3. Rayleigh quotient iteration................. 240 5.6.4. Divide-and-conquer...................... 241 5.7. Homework............................... 242 6 Iterative Methods for Linear Systems 245 6.1. A Model Problem........................... 246 6.2. Relaxation Methods.......................... 247 vi Contents 6.3. Krylov Subspace Methods...................... 248 6.4. Domain Decomposition Methods.................. 249 6.5. Multigrid Methods.......................... 250 6.6. Homework............................... 251 Chapter 1 Introduction to Matrix Analysis 1 2 CHAPTER 1. INTRODUCTION TO MATRIX ANALYSIS 1.1. Basic Algorithms and Notation 1.1.1. Matrix notation Let R be the set of real numbers. We denote the vector space of all m × n matrices by Rm×n: 2 3 a11 ··· a1n m×n 6 . 7 A 2 R , A = (aij) = 4 . 5 ; aij 2 R: (1.1) am1 ··· amn The subscript ij refers to the (i; j) entry. 1.1.2. Matrix operations • transpose (Rm×n ! Rn×m) T C = A ) cij = aji • addition (Rm×n × Rm×n ! Rm×n) C = A + B ) cij = aij + bij • scalar-matrix multiplication (R × Rm×n ! Rm×n) C = αA ) cij = α aij • matrix-matrix multiplication (Rm×p × Rp×n ! Rm×n) p X C = AB ) cij = aikbkj k=1 These are the building blocks of matrix computations. 1.1. Basic Algorithms and Notation 3 1.1.3. Vector notation Let Rn denote the vector space of real n-vectors: 2 3 x1 n 6 . 7 x 2 R , x = 4 . 5 ; xi 2 R: (1.2) xn n n×1 We refer to xi as the ith component of x. We identify R with R so that the members of Rn are column vectors. On the other hand, the elements of R1×n are row vectors: 1×n y 2 R , y = (y1; ··· ; yn): If x 2 Rn, then y = xT is a row vector. 4 CHAPTER 1. INTRODUCTION TO MATRIX ANALYSIS 1.1.4. Vector operations Assume a 2 R and x; y 2 Rn. • scalar-vector multiplication z = a x ) zi = a xi • vector addition z = x + y ) zi = xi + yi • dot product (or, inner product) n T X c = x y(= x · y) ) c = xiyi i=1 • vector multiply (or, Hadamard product) z = x: ∗ y ) zi = xiyi • saxpy (“scalar a x plus y"): a LAPACK routine z = a x + y ) zi = a xi + yi 1.1.5. Matrix-vector multiplication & “gaxpy" • gaxpy (“generalized saxpy"): a LAPACK routine n X z = A x + y ) zi = aijxj + yi j=1 1.1. Basic Algorithms and Notation 5 1.1.6. The colon notation A handy way to specify a column or row of a matrix is with the “colon" nota- tion. Let A 2 Rm×n. Then, 2 3 a1j 6 . 7 A(i; :) = [ai1; ··· ; ain];A(:; j) = 4 . 5 amj Example 1.1. With these conventions, the gaxpy can be written as for i=1:m z(i)=y(i)+A(i,:)x (1.3) end or, in column version, z=y for j=1:n (1.4) z=z+x(j)A(:,j) end Matlab-code 1.2. The gaxpy can be implemented as simple as z=A*x+y; (1.5) 6 CHAPTER 1. INTRODUCTION TO MATRIX ANALYSIS 1.1.7. Flops A flop (floating point operation) is any mathematical operation (such as +, -, ∗, /) or assignment that involves floating-point numbers Thus, the gaxpy (1.3) or (1.4) requires 2mn flops. The following will be frequently utilized in counting flops: n n X n(n + 1) X n(n + 1)(2n + 1) k = ; k2 = ; 2 6 k=1 k=1 (1.6) n n X X 2 n2(n + 1)2 k3 = k = : 4 k=1 k=1 1.2. Vector and Matrix Norms 7 1.2. Vector and Matrix Norms 1.2.1. Vector norms Definition 1.3. A norm (or, vector norm) on Rn is a function that assigns to each x 2 Rn a nonnegative real number kxk, called the norm of x, such that the following three properties are satisfied: for all x; y 2 Rn and λ 2 R, kxk > 0 if x 6= 0 (positive definiteness) kλxk = jλj kxk (homogeneity) (1.7) kx + yk ≤ kxk + kyk (triangle inequality) Example 1.4. The most common norms are 1=p X p kxkp = jxij ; 1 ≤ p < 1; (1.8) i which we call the p-norms, and kxk1 = max jxij; (1.9) i which is called the infinity-norm or maximum-norm. Two of frequently used p-norms are 1=2 P P 2 kxk1 = i jxij; kxk2 = i jxij The 2-norm is also called the Euclidean norm, often denoted by k · k. Remark 1.5. One may consider the infinity-norm as the limit of p-norms, as p ! 1; see Homework 1.1. 8 CHAPTER 1. INTRODUCTION TO MATRIX ANALYSIS Theorem 1.6. (Cauchy-Schwarz inequality) For all x; y 2 Rn, n n n 1=2 1=2 X X 2 X 2 xiyi ≤ xi yi (1.10) i=1 i=1 i=1 Note:(1.10) can be rewritten as jx · yj ≤ kxk kyk (1.11) which is clearly true. Example 1.7. Given a positive definite matrix A 2 Rn×n, define the A-norm on Rn by T 1=2 kxkA = (x Ax) Note: When A = I, the A-norm is just the Euclidean norm. The A-norm is indeed a norm; see Homework 1.2. Lemma 1.8. All p-norms on Rn are equivalent to each other. In particular, p kxk2 ≤ kxk1 ≤ n kxk2 p kxk1 ≤ kxk2 ≤ n kxk1 (1.12) kxk1 ≤ kxk1 ≤ n kxk1 Note: For all x 2 Rn, p kxk1 ≤ kxk2 ≤ kxk1 ≤ n kxk2 ≤ n kxk1 (1.13) 1.2.
Recommended publications
  • 2 Homework Solutions 18.335 " Fall 2004

    2 Homework Solutions 18.335 " Fall 2004

    2 Homework Solutions 18.335 - Fall 2004 2.1 Count the number of ‡oating point operations required to compute the QR decomposition of an m-by-n matrix using (a) Householder re‡ectors (b) Givens rotations. 2 (a) See Trefethen p. 74-75. Answer: 2mn2 n3 ‡ops. 3 (b) Following the same procedure as in part (a) we get the same ‘volume’, 1 1 namely mn2 n3: The only di¤erence we have here comes from the 2 6 number of ‡opsrequired for calculating the Givens matrix. This operation requires 6 ‡ops (instead of 4 for the Householder re‡ectors) and hence in total we need 3mn2 n3 ‡ops. 2.2 Trefethen 5.4 Let the SVD of A = UV : Denote with vi the columns of V , ui the columns of U and i the singular values of A: We want to …nd x = (x1; x2) and such that: 0 A x x 1 = 1 A 0 x2 x2 This gives Ax2 = x1 and Ax1 = x2: Multiplying the 1st equation with A 2 and substitution of the 2nd equation gives AAx2 = x2: From this we may conclude that x2 is a left singular vector of A: The same can be done to see that x1 is a right singular vector of A: From this the 2m eigenvectors are found to be: 1 v x = i ; i = 1:::m p2 ui corresponding to the eigenvalues = i. Therefore we get the eigenvalue decomposition: 1 0 A 1 VV 0 1 VV = A 0 p2 U U 0 p2 U U 3 2.3 If A = R + uv, where R is upper triangular matrix and u and v are (column) vectors, describe an algorithm to compute the QR decomposition of A in (n2) time.
  • A Fast Algorithm for the Recursive Calculation of Dominant Singular Subspaces N

    A Fast Algorithm for the Recursive Calculation of Dominant Singular Subspaces N

    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Elsevier - Publisher Connector Journal of Computational and Applied Mathematics 218 (2008) 238–246 www.elsevier.com/locate/cam A fast algorithm for the recursive calculation of dominant singular subspaces N. Mastronardia,1, M. Van Barelb,∗,2, R. Vandebrilb,2 aIstituto per le Applicazioni del Calcolo, CNR, via Amendola122/D, 70126, Bari, Italy bDepartment of Computer Science, Katholieke Universiteit Leuven, Celestijnenlaan 200A, 3001 Leuven, Belgium Received 26 September 2006 Abstract In many engineering applications it is required to compute the dominant subspace of a matrix A of dimension m × n, with m?n. Often the matrix A is produced incrementally, so all the columns are not available simultaneously. This problem arises, e.g., in image processing, where each column of the matrix A represents an image of a given sequence leading to a singular value decomposition-based compression [S. Chandrasekaran, B.S. Manjunath, Y.F. Wang, J. Winkeler, H. Zhang, An eigenspace update algorithm for image analysis, Graphical Models and Image Process. 59 (5) (1997) 321–332]. Furthermore, the so-called proper orthogonal decomposition approximation uses the left dominant subspace of a matrix A where a column consists of a time instance of the solution of an evolution equation, e.g., the flow field from a fluid dynamics simulation. Since these flow fields tend to be very large, only a small number can be stored efficiently during the simulation, and therefore an incremental approach is useful [P. Van Dooren, Gramian based model reduction of large-scale dynamical systems, in: Numerical Analysis 1999, Chapman & Hall, CRC Press, London, Boca Raton, FL, 2000, pp.
  • The Generalized Triangular Decomposition

    The Generalized Triangular Decomposition

    MATHEMATICS OF COMPUTATION Volume 77, Number 262, April 2008, Pages 1037–1056 S 0025-5718(07)02014-5 Article electronically published on October 1, 2007 THE GENERALIZED TRIANGULAR DECOMPOSITION YI JIANG, WILLIAM W. HAGER, AND JIAN LI Abstract. Given a complex matrix H, we consider the decomposition H = QRP∗,whereR is upper triangular and Q and P have orthonormal columns. Special instances of this decomposition include the singular value decompo- sition (SVD) and the Schur decomposition where R is an upper triangular matrix with the eigenvalues of H on the diagonal. We show that any diag- onal for R can be achieved that satisfies Weyl’s multiplicative majorization conditions: k k K K |ri|≤ σi, 1 ≤ k<K, |ri| = σi, i=1 i=1 i=1 i=1 where K is the rank of H, σi is the i-th largest singular value of H,andri is the i-th largest (in magnitude) diagonal element of R. Given a vector r which satisfies Weyl’s conditions, we call the decomposition H = QRP∗,whereR is upper triangular with prescribed diagonal r, the generalized triangular decom- position (GTD). A direct (nonrecursive) algorithm is developed for computing the GTD. This algorithm starts with the SVD and applies a series of permu- tations and Givens rotations to obtain the GTD. The numerical stability of the GTD update step is established. The GTD can be used to optimize the power utilization of a communication channel, while taking into account qual- ity of service requirements for subchannels. Another application of the GTD is to inverse eigenvalue problems where the goal is to construct matrices with prescribed eigenvalues and singular values.
  • CSE 275 Matrix Computation

    CSE 275 Matrix Computation

    CSE 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 13 1 / 22 Overview Eigenvalue problem Schur decomposition Eigenvalue algorithms 2 / 22 Reading Chapter 24 of Numerical Linear Algebra by Llyod Trefethen and David Bau Chapter 7 of Matrix Computations by Gene Golub and Charles Van Loan 3 / 22 Eigenvalues and eigenvectors Let A 2 Cm×m be a square matrix, a nonzero x 2 Cm is an eigenvector of A, and λ 2 C is its corresponding eigenvalue if Ax = λx Idea: the action of a matrix A on a subspace S 2 Cm may sometimes mimic scalar multiplication When it happens, the special subspace S is called an eigenspace, and any nonzero x 2 S is an eigenvector The set of all eigenvalues of a matrix A is the spectrum of A, a subset of C denoted by Λ(A) 4 / 22 Eigenvalues and eigenvectors (cont'd) Ax = λx Algorithmically: simplify solutions of certain problems by reducing a coupled system to a collection of scalar problems Physically: give insight into the behavior of evolving systems governed by linear equations, e.g., resonance (of musical instruments when struck or plucked or bowed), stability (of fluid flows with small perturbations) 5 / 22 Eigendecomposition An eigendecomposition (eigenvalue decomposition) of a square matrix A is a factorization A = X ΛX −1 where X is a nonsingular and Λ is diagonal Equivalently, AX = X Λ 2 3 λ1 6 λ2 7 A x x ··· x = x x ··· x 6 7 1 2 m 1 2 m 6 .
  • The Schur Decomposition Week 5 UCSB 2014

    The Schur Decomposition Week 5 UCSB 2014

    Math 108B Professor: Padraic Bartlett Lecture 5: The Schur Decomposition Week 5 UCSB 2014 Repeatedly through the past three weeks, we have taken some matrix A and written A in the form A = UBU −1; where B was a diagonal matrix, and U was a change-of-basis matrix. However, on HW #2, we saw that this was not always possible: in particular, you proved 1 1 in problem 4 that for the matrix A = , there was no possible basis under which A 0 1 would become a diagonal matrix: i.e. you proved that there was no diagonal matrix D and basis B = f(b11; b21); (b12; b22)g such that b b b b −1 A = 11 12 · D · 11 12 : b21 b22 b21 b22 This is a bit of a shame, because diagonal matrices (for reasons discussed earlier) are pretty fantastic: they're easy to raise to large powers and calculate determinants of, and it would have been nice if every linear transformation was diagonal in some basis. So: what now? Do we simply assume that some matrices cannot be written in a \nice" form in any basis, and that we should assume that operations like matrix exponentiation and finding determinants is going to just be awful in many situations? The answer, as you may have guessed by the fact that these notes have more pages after this one, is no! In particular, while diagonalization1 might not always be possible, there is something fairly close that is - the Schur decomposition. Our goal for this week is to prove this, and study its applications.
  • Finite-Dimensional Spectral Theory Part I: from Cn to the Schur Decomposition

    Finite-Dimensional Spectral Theory Part I: from Cn to the Schur Decomposition

    Finite-dimensional spectral theory part I: from Cn to the Schur decomposition Ed Bueler MATH 617 Functional Analysis Spring 2020 Ed Bueler (MATH 617) Finite-dimensional spectral theory Spring 2020 1 / 26 linear algebra versus functional analysis these slides are about linear algebra, i.e. vector spaces of finite dimension, and linear operators on those spaces, i.e. matrices one definition of functional analysis might be: “rigorous extension of linear algebra to 1-dimensional topological vector spaces” ◦ it is important to understand the finite-dimensional case! the goal of these part I slides is to prove the Schur decomposition and the spectral theorem for matrices good references for these slides: ◦ L. Trefethen & D. Bau, Numerical Linear Algebra, SIAM Press 1997 ◦ G. Strang, Introduction to Linear Algebra, 5th ed., Wellesley-Cambridge Press, 2016 ◦ G. Golub & C. van Loan, Matrix Computations, 4th ed., Johns Hopkins University Press 2013 Ed Bueler (MATH 617) Finite-dimensional spectral theory Spring 2020 2 / 26 the spectrum of a matrix the spectrum σ(A) of a square matrix A is its set of eigenvalues ◦ reminder later about the definition of eigenvalues ◦ σ(A) is a subset of the complex plane C ◦ the plural of spectrum is spectra; the adjectival is spectral graphing σ(A) gives the matrix a personality ◦ example below: random, nonsymmetric, real 20 × 20 matrix 6 4 2 >> A = randn(20,20); ) >> lam = eig(A); λ 0 >> plot(real(lam),imag(lam),’o’) Im( >> grid on >> xlabel(’Re(\lambda)’) -2 >> ylabel(’Im(\lambda)’) -4 -6 -6 -4 -2 0 2 4 6 Re(λ) Ed Bueler (MATH 617) Finite-dimensional spectral theory Spring 2020 3 / 26 Cn is an inner product space we use complex numbers C from now on ◦ why? because eigenvalues can be complex even for a real matrix ◦ recall: if α = x + iy 2 C then α = x − iy is the conjugate let Cn be the space of (column) vectors with complex entries: 2v13 = 6 .
  • Numerical Linear Algebra Revised February 15, 2010 4.1 the LU

    Numerical Linear Algebra Revised February 15, 2010 4.1 the LU

    Numerical Linear Algebra Revised February 15, 2010 4.1 The LU Decomposition The Elementary Matrices and badgauss In the previous chapters we talked a bit about the solving systems of the form Lx = b and Ux = b where L is lower triangular and U is upper triangular. In the exercises for Chapter 2 you were asked to write a program x=lusolver(L,U,b) which solves LUx = b using forsub and backsub. We now address the problem of representing a matrix A as a product of a lower triangular L and an upper triangular U: Recall our old friend badgauss. function B=badgauss(A) m=size(A,1); B=A; for i=1:m-1 for j=i+1:m a=-B(j,i)/B(i,i); B(j,:)=a*B(i,:)+B(j,:); end end The heart of badgauss is the elementary row operation of type 3: B(j,:)=a*B(i,:)+B(j,:); where a=-B(j,i)/B(i,i); Note also that the index j is greater than i since the loop is for j=i+1:m As we know from linear algebra, an elementary opreration of type 3 can be viewed as matrix multipli- cation EA where E is an elementary matrix of type 3. E looks just like the identity matrix, except that E(j; i) = a where j; i and a are as in the MATLAB code above. In particular, E is a lower triangular matrix, moreover the entries on the diagonal are 1's. We call such a matrix unit lower triangular.
  • Massively Parallel Poisson and QR Factorization Solvers

    Massively Parallel Poisson and QR Factorization Solvers

    Computers Math. Applic. Vol. 31, No. 4/5, pp. 19-26, 1996 Pergamon Copyright~)1996 Elsevier Science Ltd Printed in Great Britain. All rights reserved 0898-1221/96 $15.00 + 0.00 0898-122 ! (95)00212-X Massively Parallel Poisson and QR Factorization Solvers M. LUCK£ Institute for Control Theory and Robotics, Slovak Academy of Sciences DdbravskA cesta 9, 842 37 Bratislava, Slovak Republik utrrluck@savba, sk M. VAJTERSIC Institute of Informatics, Slovak Academy of Sciences DdbravskA cesta 9, 840 00 Bratislava, P.O. Box 56, Slovak Republic kaifmava©savba, sk E. VIKTORINOVA Institute for Control Theoryand Robotics, Slovak Academy of Sciences DdbravskA cesta 9, 842 37 Bratislava, Slovak Republik utrrevka@savba, sk Abstract--The paper brings a massively parallel Poisson solver for rectangle domain and parallel algorithms for computation of QR factorization of a dense matrix A by means of Householder re- flections and Givens rotations. The computer model under consideration is a SIMD mesh-connected toroidal n x n processor array. The Dirichlet problem is replaced by its finite-difference analog on an M x N (M + 1, N are powers of two) grid. The algorithm is composed of parallel fast sine transform and cyclic odd-even reduction blocks and runs in a fully parallel fashion. Its computational complexity is O(MN log L/n2), where L = max(M + 1, N). A parallel proposal of QI~ factorization by the Householder method zeros all subdiagonal elements in each column and updates all elements of the given submatrix in parallel. For the second method with Givens rotations, the parallel scheme of the Sameh and Kuck was chosen where the disjoint rotations can be computed simultaneously.
  • The Non–Symmetric Eigenvalue Problem

    The Non–Symmetric Eigenvalue Problem

    Chapter 4 of Calculus++: The Non{symmetric Eigenvalue Problem by Eric A Carlen Professor of Mathematics Georgia Tech c 2003 by the author, all rights reserved 1-1 Table of Contents Overview ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 1-3 Section 1: Schur factorization 1.1 The non{symmetric eigenvalue problem :::::::::::::::::::::::::::::::::::::: 1-4 1.2 What is the Schur factorization? ::::::::::::::::::::::::::::::::::::::::::::: 1-4 1.3 The 2 × 2 case ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 1-5 Section 2: Complex eigenvectors and the geometry of Cn 2.1 Why get complicated? ::::::::::::::::::::::::::::::::::::::::::::::::::::::: 1-8 2.2 Algebra and geometry in Cn ::::::::::::::::::::::::::::::::::::::::::::::::: 1-8 2.3 Unitary matrices :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 1-11 2.4 Schur factorization in general ::::::::::::::::::::::::::::::::::::::::::::::: 1-11 Section: 3 Householder reflections 3.1 Reflection matrices ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::1-15 3.2 The n × n case ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::1-17 3.3 Householder reflection matrices and the QR factorization :::::::::::::::::::: 1-19 3.4 The complex case ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 1-23 Section: 4 The QR iteration 4.1 What the QR iteration is ::::::::::::::::::::::::::::::::::::::::::::::::::: 1-25 4.2 When and how QR iteration works :::::::::::::::::::::::::::::::::::::::::: 1-28 4.3 What to do when QR iteration
  • Parallel Eigenvalue Reordering in Real Schur Forms∗

    Parallel Eigenvalue Reordering in Real Schur Forms∗

    Parallel eigenvalue reordering in real Schur forms∗ R. Granat,y B. K˚agstr¨om,z and D. Kressnerx May 12, 2008 Abstract A parallel algorithm for reordering the eigenvalues in the real Schur form of a matrix is pre- sented and discussed. Our novel approach adopts computational windows and delays multiple outside-window updates until each window has been completely reordered locally. By using multiple concurrent windows the parallel algorithm has a high level of concurrency, and most work is level 3 BLAS operations. The presented algorithm is also extended to the generalized real Schur form. Experimental results for ScaLAPACK-style Fortran 77 implementations on a Linux cluster confirm the efficiency and scalability of our algorithms in terms of more than 16 times of parallel speedup using 64 processor for large scale problems. Even on a single processor our implementation is demonstrated to perform significantly better compared to the state-of-the-art serial implementation. Keywords: Parallel algorithms, eigenvalue problems, invariant subspaces, direct reorder- ing, Sylvester matrix equations, condition number estimates 1 Introduction The solution of large-scale matrix eigenvalue problems represents a frequent task in scientific computing. For example, the asymptotic behavior of a linear or linearized dynamical system is determined by the right-most eigenvalue of the system matrix. Despite the advance of iterative methods { such as Arnoldi and Jacobi-Davidson algorithms [3] { there are problems where a transformation method { usually the QR algorithm [14] { is preferred, even in a large-scale setting. In the example quoted above, an iterative method may fail to detect the right-most eigenvalue and, in the worst case, misleadingly predict stability even though the system is unstable [33].
  • The Unsymmetric Eigenvalue Problem

    The Unsymmetric Eigenvalue Problem

    Jim Lambers CME 335 Spring Quarter 2010-11 Lecture 4 Supplemental Notes The Unsymmetric Eigenvalue Problem Properties and Decompositions Let A be an n × n matrix. A nonzero vector x is called an eigenvector of A if there exists a scalar λ such that Ax = λx: The scalar λ is called an eigenvalue of A, and we say that x is an eigenvector of A corresponding to λ. We see that an eigenvector of A is a vector for which matrix-vector multiplication with A is equivalent to scalar multiplication by λ. We say that a nonzero vector y is a left eigenvector of A if there exists a scalar λ such that λyH = yH A: The superscript H refers to the Hermitian transpose, which includes transposition and complex conjugation. That is, for any matrix A, AH = AT . An eigenvector of A, as defined above, is sometimes called a right eigenvector of A, to distinguish from a left eigenvector. It can be seen that if y is a left eigenvector of A with eigenvalue λ, then y is also a right eigenvector of AH , with eigenvalue λ. Because x is nonzero, it follows that if x is an eigenvector of A, then the matrix A − λI is singular, where λ is the corresponding eigenvalue. Therefore, λ satisfies the equation det(A − λI) = 0: The expression det(A−λI) is a polynomial of degree n in λ, and therefore is called the characteristic polynomial of A (eigenvalues are sometimes called characteristic values). It follows from the fact that the eigenvalues of A are the roots of the characteristic polynomial that A has n eigenvalues, which can repeat, and can also be complex, even if A is real.
  • Schur Triangularization and the Spectral Decomposition(S)

    Schur Triangularization and the Spectral Decomposition(S)

    Advanced Linear Algebra – Week 7 Schur Triangularization and the Spectral Decomposition(s) This week we will learn about: • Schur triangularization, • The Cayley–Hamilton theorem, • Normal matrices, and • The real and complex spectral decompositions. Extra reading and watching: • Section 2.1 in the textbook • Lecture videos 25, 26, 27, 28, and 29 on YouTube • Schur decomposition at Wikipedia • Normal matrix at Wikipedia • Spectral theorem at Wikipedia Extra textbook problems: ? 2.1.1, 2.1.2, 2.1.5 ?? 2.1.3, 2.1.4, 2.1.6, 2.1.7, 2.1.9, 2.1.17, 2.1.19 ??? 2.1.8, 2.1.11, 2.1.12, 2.1.18, 2.1.21 A 2.1.22, 2.1.26 1 Advanced Linear Algebra – Week 7 2 We’re now going to start looking at matrix decompositions, which are ways of writing down a matrix as a product of (hopefully simpler!) matrices. For example, we learned about diagonalization at the end of introductory linear algebra, which said that... While diagonalization let us do great things with certain matrices, it also raises some new questions: Over the next few weeks, we will thoroughly investigate these types of questions, starting with this one: Advanced Linear Algebra – Week 7 3 Schur Triangularization We know that we cannot hope in general to get a diagonal matrix via unitary similarity (since not every matrix is diagonalizable via any similarity). However, the following theorem says that we can get partway there and always get an upper triangular matrix. Theorem 7.1 — Schur Triangularization Suppose A ∈ Mn(C).