Daniel Kressner Numerical Methods for General and Structured Eigenvalue Problems SPIN Springer's internal project number, if known { Monograph { April 4, 2005 Springer Berlin Heidelberg NewYork Hong Kong London Milan Paris Tokyo Immer wenn es regnet: : : Preface The purpose of this book is to describe recent developments in solving eigen- value problems, in particular with respect to the QR and QZ algorithms as well as structured matrices. Outline Mathematically speaking, the eigenvalues of a square matrix A are the roots of its characteristic polynomial det(A λI). An invariant subspace is a linear subspace that stays invariant under the− action of A. In realistic applications, it usually takes a long process of simplifications, linearizations and discretiza- tions before one comes up with the problem of computing the eigenvalues of a matrix. In some cases, the eigenvalues have an intrinsic meaning, e.g., for the expected long-time behavior of a dynamical system; in others they are just meaningless intermediate values of a computational method. The same applies to invariant subspaces, which for example can describe sets of initial states for which a dynamical system produces exponentially decaying states. Computing eigenvalues has a long history, dating back to at least 1846 when Jacobi [172] wrote his famous paper on solving symmetric eigenvalue problems. Detailed historical accounts of this subject can be found in two papers by Golub and van der Vorst [140, 327]. Chapter 1 of this book is concerned with the QR algorithm, which was introduced by Francis [128] and Kublanovskaya [206] in 1961{1962, partly based on earlier work by Rutishauser [278]. The QR algorithm is a general- purpose, numerically backward stable method for computing all eigenvalues of a non-symmetric matrix. It has undergone only a few modification during the following 40 years, see [348] for a complete overview of the practical QR algo- rithm as it is currently implemented in LAPACK [10, 17]. An award-winning improvement was made in 2002 when Braman, Byers, and Mathias [62] pre- sented their aggressive early deflation strategy. The combination of this de- flation strategy with a tiny-bulge multishift QR algorithm [61, 208] leads to VIII Preface a variant of the QR algorithm, which can, for sufficiently large matrices, re- quire less than 10% of the computing time needed by the current LAPACK implementation. Similar techniques can also be used to significantly improve the performance of the post-processing step necessary to compute invariant subspaces from the output of the QR algorithm. Besides these algorithmic im- provements, Chapter 1 summarizes well-known and also some recent material related to the perturbation analysis of eigenvalues and invariant subspaces; local and global convergence properties of the QR algorithm; and the failure of the large-bulge multishift QR algorithm in finite-precision arithmetic. The subject of Chapter 2 is the QZ algorithm, a popular method for com- puting the generalized eigenvalues of a matrix pair (A; B), i.e., the roots of the bivariate polynomial det(βA αB). The QZ algorithm was developed by Moler and Stewart [248] in 1973.−Its probably most notable modification has been the high-performance pipelined QZ algorithm developed by Dackland and K˚agstr¨om [96]. One topic of Chapter 2 is the use of Householder matrices within the QZ algorithm. The wooly role of infinite eigenvalues is investigated and a tiny-bulge multishift QZ algorithm with aggressive early deflation in the spirit of [61, 208] is described. Numerical experiments illustrate the per- formance improvements to be gained from these recent developments. This book is not so much about solving large-scale eigenvalue problems. The practically important aspect of parallelization is completely omitted; we refer to the ScaLAPACK users' guide [49]. Also, methods for computing a few eigenvalues of a large matrix, such as Arnoldi, Lanczos or Jacobi-Davidson methods, are only partially covered. In Chapter 3, we focus on a descendant of the Arnoldi method, the recently introduced Krylov-Schur algorithm by Stewart [307]. Later on, in Chapter 4, it is explained how this algorithm can be adapted to some structured eigenvalue problems in a considerably simple manner. Another subject of Chapter 3 is the balancing of sparse matrices for eigenvalue computations [91]. In many cases, the eigenvalue problem under consideration is known to be structured. Preserving this structure can help preserve induced eigenvalue symmetries in finite-precision arithmetic and may improve the accuracy and efficiency of an eigenvalue computation. Chapter 4 provides an overview of some of the recent developments in the area of structured eigenvalue prob- lems. Particular attention is paid to the concept of structured condition num- bers for eigenvalues and invariant subspaces. A detailed treatment of the- ory, algorithms and applications is given for product, Hamiltonian and skew- Hamiltonian eigenvalue problems, while other structures (skew-symmetric, persymmetric, orthogonal, palindromic) are only briefly discussed. Appendix B contains an incomplete list of publicly available software for solving general and structured eigenvalue problems. A more complete and regularly updated list can be found at http://www.math.tu-berlin.de/ ∼kressner/book/, the web page of this book. Preface IX Prerequisites Readers of this text need to be familiar with the basic concepts from nu- merical analysis and linear algebra. Those are covered by any of the text books [103, 141, 304, 305, 354]. Concepts from systems and control theory are occasionally used; either because an algorithm for computing eigenvalues is better understood in a control theoretic setting or such an algorithm can be used for the analysis and design of linear control systems. Knowledge of systems and control theory is not assumed, everything that is needed can be picked up from Appendix A, which contains a brief introduction to this area. Nevertheless, for getting a more complete picture, it might be wise to com- plement the reading with a state space oriented book on control theory. The monographs [148, 265, 285, 329, 368] are particularly suited for this purpose with respect to content and style of presentation. Acknowledgments This book is largely based on my PhD thesis and, once again, I thank all who supported the writing of the thesis, in particular my supervisor Volker Mehrmann and my parents. Turning the thesis into a book would not have been possible without the encouragement and patience of Thanh-Ha Le Thi from Springer-Verlag. I have benefited a lot from ongoing joint work and discussions with Ulrike Baur, Peter Benner, Ralph Byers, Heike Faßben- der, Michiel Hochstenbach, Bo K˚agstr¨om, Michael Karow, Emre Mengi, and Fran¸coise Tisseur. Furthermore, I am indebted to Gene Golub, Robert Granat, Nick Higham, Damien Lemonnier, J¨org Liesen, Christian Mehl, Bor Plesten- jak, Christian Schr¨oder, Vasile Sima, Valeria Simoncini, Tanja Stykel, Ji-guang Sun, Paul Van Dooren, Kreˇsimir Veseli´c, David Watkins, and many others for helpful and illuminating discussions. The work on this book was supported by the DFG Research Center Matheon \Mathematics for key technologies" in Berlin. Berlin, Daniel Kressner April 2005 Contents 1 The QR Algorithm . 1 1.1 The Standard Eigenvalue Problem . 2 1.2 Perturbation Analysis . 3 1.2.1 Spectral Projectors and Separation . 4 1.2.2 Eigenvalues and Eigenvectors . 6 1.2.3 Eigenvalue Clusters and Invariant Subspaces . 10 1.2.4 Global Perturbation Bounds . 15 1.3 The Basic QR Algorithm . 18 1.3.1 Local Convergence. 19 1.3.2 Hessenberg Form . 24 1.3.3 Implicit Shifted QR Iteration . 27 1.3.4 Deflation . 30 1.3.5 The Overall Algorithm . 31 1.3.6 Failure of Global Converge . 34 1.4 Balancing . 35 1.4.1 Isolating Eigenvalues . 35 1.4.2 Scaling . 36 1.4.3 Merits of Balancing . 39 1.5 Block Algorithms . 39 1.5.1 Compact WY Representation . 40 1.5.2 Block Hessenberg Reduction . 41 1.5.3 Multishifts and Bulge Pairs . 44 1.5.4 Connection to Pole Placement . 45 1.5.5 Tightly Coupled Tiny Bulges . 48 1.6 Advanced Deflation Techniques. 53 1.7 Computation of Invariant Subspaces . 57 1.7.1 Swapping Two Diagonal Blocks . 58 1.7.2 Reordering . 60 1.7.3 Block Algorithm . 60 1.8 Case Study: Solution of an Optimal Control Problem . 63 XII Contents 2 The QZ Algorithm . 67 2.1 The Generalized Eigenvalue Problem . 68 2.2 Perturbation Analysis . 70 2.2.1 Spectral Projectors and Dif . 70 2.2.2 Local Perturbation Bounds . 72 2.2.3 Global Perturbation Bounds . 75 2.3 The Basic QZ Algorithm . 76 2.3.1 Hessenberg-Triangular Form . 76 2.3.2 Implicit Shifted QZ Iteration. 79 2.3.3 On the Use of Householder Matrices . 82 2.3.4 Deflation . 86 2.3.5 The Overall Algorithm . 89 2.4 Balancing . 91 2.4.1 Isolating Eigenvalues . 91 2.4.2 Scaling . 91 2.5 Block Algorithms . 93 2.5.1 Reduction to Hessenberg-Triangular Form . 94 2.5.2 Multishifts and Bulge Pairs . 99 2.5.3 Deflation of Infinite Eigenvalues Revisited . 101 2.5.4 Tightly Coupled Tiny Bulge Pairs . 102 2.6 Aggressive Early Deflation . 105 2.7 Computation of Deflating Subspaces . 108 3 The Krylov-Schur Algorithm . 113 3.1 Basic Tools . 114 3.1.1 Krylov Subspaces . 114 3.1.2 The Arnoldi Method . 116 3.2 Restarting and the Krylov-Schur Algorithm . 119 3.2.1 Restarting an Arnoldi Decomposition . 120 3.2.2 The Krylov Decomposition . 121 3.2.3 Restarting a Krylov Decomposition . 122 3.2.4 Deflating a Krylov Decomposition . 124 3.3 Balancing Sparse Matrices . 126 3.3.1 Irreducible Forms . 127 3.3.2 Krylov-Based Balancing . 128 4 Structured Eigenvalue Problems .