Mathematics & Statistics Auburn University, Alabama, USA QR History Dec 17, 2010 Asymptotic result QR iteration QR decomposition: History and its EE Applications Home Page Title Page Tin-Yau Tam èèèUUUÎÎÎ JJ II J I Page 1 of 37 Æâ§w Go Back fÆêÆÆÆ Full Screen Close email: [email protected] Website: www.auburn.edu/∼tamtiny Quit 1. QR decomposition Recall the QR decomposition of A ∈ GLn(C): QR A = QR History Asymptotic result QR iteration where Q ∈ GLn(C) is unitary and R ∈ GLn(C) is upper ∆ with positive EE diagonal entries. Such decomposition is unique. Set Home Page a(A) := diag (r11, . , rnn) Title Page where A is written in column form JJ II J I A = (a1| · · · |an) Page 2 of 37 Go Back Geometric interpretation of a(A): Full Screen rii is the distance (w.r.t. 2-norm) between ai and span {a1, . , ai−1}, Close i = 2, . , n. Quit Example: 12 −51 4 6/7 −69/175 −58/175 14 21 −14 6 167 −68 = 3/7 158/175 6/175 0 175 −70 . QR −4 24 −41 −2/7 6/35 −33/35 0 0 35 History Asymptotic result QR iteration EE • QR decomposition is the matrix version of the Gram-Schmidt orthonor- Home Page malization process. Title Page JJ II • QR decomposition can be extended to rectangular matrices, i.e., if A ∈ J I m×n with m ≥ n (tall matrix) and full rank, then C Page 3 of 37 A = QR Go Back Full Screen where Q ∈ Cm×n has orthonormal columns and R ∈ Cn×n is upper ∆ Close with positive “diagonal” entries. Quit 2. QR history QR • When Erhard Schmidt presented the formulae on p. 442 of his History Asymptotic result E. Schmidt, Zur Theorie der linearen und nichtlinearen Integralgle- QR iteration EE ichungen. I. Teil: Entwicklung willkulicher¨ Funktionen nach Systemen vorgeschriebener, Math. Ann., 63 (1907) 433–476. Home Page he said that essentially the same formulae were in Title Page J. P. Gram, Ueber die Entwickelung reeler Funtionen in Reihen mittelst JJ II der Methode der kleinsten Quadrate, Jrnl. fur¨ die reine und angewandte J I Math. 94 (1883) 71–73. Page 4 of 37 Go Back • Modern writers, however, distinguish the two procedures, sometimes us- Full Screen ing the term “Gram-Schmidt” for the Schmidt form and “modified Gram- Close Schmidt” for the Gram version. Quit • But Gram-Schmidt orthonormalization appeared earlier in the work of QR History Laplace and Cauchy. Asymptotic result QR iteration EE • In the theory of semisimple Lie grougs, Gram-Schmidt process is extended to the Iwasawa decomposition G = KAN. Home Page Title Page • A JSTOR search: the term “Gram-Schmidt orthogonalization process” JJ II first appears on p.57 of J I Y. K. Wong, An Application of Orthogonalization Process to the Theory Page 5 of 37 of Least Squares,” Annals of Mathematical Statistics, 6 (1935), 53–75. Go Back Full Screen Close Quit • In 1801 Gauss predicted the orbit of the steroid Ceres using the method of least squares. Since then, the principle of least squares has been the standard procedure for the analysis of scientific data. QR History • Least squares problem Asymptotic result Ax ≈ b QR iteration EE i.e., finding xˆ that would yield Home Page min kAx − bk2. x Title Page The solution is characterized by r ⊥ R(A), where r = b − Ax is the JJ II residual vector, or equivalently, given by the normal equation J I Page 6 of 37 ∗ ∗ A Ax = A b. Go Back Full Screen • With A = QR and A ∈ Cm×n tall with full rank Close ∗ ∗ ∗ ∗ ∗ R Rx = (QR) QRx = R Q b ⇔ Rx = Q b. Quit There are several methods for computing the QR decomposition: QR • GS or modified GS, History Asymptotic result • Givens rotations (real A), or QR iteration EE • Householder reflections. Home Page Disadvantage of GS: sensitive to rounding error (orthogonality of the Title Page computed vectors can be lost quickly or may even be completely lost) → JJ II modified Gram-Schmidt. J I Page 7 of 37 Idea of modified GS: do the projection step with a number of projections Go Back which will be against the errors introduced in computation. Full Screen Close Quit Example: 1 + 1 1 A = 1 1 + 1 QR History 1 1 1 + Asymptotic result 2 QR iteration with very small such that 3+2 will be computed accurately but 3+2+ EE will be computed as 3 + 2. For example, || < 10−10. Then Home Page √1+ √−1 √−1 3+2 2 2 Title Page Q = √ 1 √1 0 3+2 2 JJ II √ 1 0 √1 3+2 2 J I and cos θ12 and cos θ13 ≈ π/2 but and cos θ23 ≈ π/3. Page 8 of 37 Go Back See http://www-db.stanford.edu/TR/CS-TR-69-122.html for a heuristic Full Screen analysis of why Gram-Schmidt not stable. Close Quit Computing QR by Householder reflections A Householder reflection is a reflection about some hyperplane. Consider ∗ QR Qv = I − 2vv , kvk2 = 1. History Asymptotic result QR iteration Qv sends v to −v and fix pointwise the hyperplane ⊥ to v. Householder EE reflections are Hermitian and unitary. T Let e1 = (1, 0, ..., 0) . Recall Home Page Title Page A = (a1| · · · |an) ∈ GLn(C) JJ II If ka1k2 = α1, set J I u ∗ Page 9 of 37 u = a1 − α1e1, v = ,Q1 = I − 2vv kuk2 Go Back so that Full Screen T Close Qa1 = (α1, 0, ··· , 0) = α1e1. Quit Then α1 ∗ ... ∗ 0 Q1A = . QR . A1 History 0 Asymptotic result QR iteration After t iterations of this process, t ≤ n − 1, EE R = Qt ··· Q2Q1A Home Page is upper ∆. So, with Title Page Q = Q1Q2 ··· Qt JJ II A = QR is the QR decomposition of A. J I Page 10 of 37 Go Back • This method has greater numerical stability than GS. Full Screen • On the other hand, GS produces the qj vector after the jth iteration, Close while Householder reflections produces all the vectors only at the end. Quit 3. An Asymptotic result Theorem 3.1. (Huang and Tam 2007) Given A ∈ GLn(C). Let A = QR Y −1JY be the Jordan decomposition of A, where J is the Jordan form of History Asymptotic result A, QR iteration EE diag J = diag (λ1, . , λn) satisfying |λ | ≥ · · · ≥ |λ |. Then 1 n Home Page m 1/m Title Page lim a(A ) = diag (|λω(1)|,..., |λω(n)|), m→∞ JJ II where the permutation ω is uniquely determined by the Gelfand-Naimark J I decomposition of Y = LωU: Page 11 of 37 Go Back rank ω(i|j) = rank (Y )(i|j), 1 ≤ i, j ≤ n. Full Screen Here ω(i|j) denotes the submatrix formed by the first i rows and the first Close j columns of ω, 1 ≤ i, j ≤ n. Quit • Gelfand-Naimark decomposition of Y = LωU is different from the T Guassian decomposition Y = P LU obtained by Gaussian elimination QR History with row exchanges. Asymptotic result QR iteration EE • None of P , U and L in the Gaussian decomposition is unique. Home Page • But ω and diag U are unique in the Gelfand-Naimark decomposition. Title Page JJ II H. Huang and T.Y. Tam, An asymptotic behavior of QR decomposition, J I Linear Algebra and Its Applications, 424 (2007) 96-107. Page 12 of 37 H. Huang and T.Y. Tam, An asymptotic result on the a-component in Iwa- Go Back sawa decomposition, Journal of Lie Theory, 17 (2007) 469-479. Full Screen Close Quit Numerical experiments: Computing the discrepancy between QR m 1/m [a(A )] and |λ(A)| History Asymptotic result QR iteration of randomly generated A ∈ GLn(C). EE The graph of Home Page m 1/m 100 k[a(A )] − diag (|λ1|,..., |λn|)k2 Title Page JJ II 80 versus m (m = 1,..., 100) J I 60 Page 13 of 37 40 Go Back Full Screen 20 0 20 40 60 80 100 Close Quit If we consider m 1/m |a1(A ) − |λ1(A)|| QR m 1/m History instead of k[a(A )] − diag (|λ1|,..., |λn|)k2 for the above example, Asymptotic result convergence occurs. QR iteration EE The graph of Home Page 70 m 1/m |a1(A ) − |λ1(A)|| Title Page 60 50 versus m (m = 1,..., 100) JJ II 40 J I 30 Page 14 of 37 20 Go Back 10 Full Screen 0 20 40 60 80 100 Close Quit 4. QR iteration • Because of Abel’s theorem (1824), the roots of a general fifth order polynomial cannot be solved by radicals. Thus the computation of eigen- QR History values of A ∈ Cn×n has to be approximative. Asymptotic result QR iteration EE • Given A ∈ GLn(C), define a sequence {Ak}k∈N of matrices with Home Page A1 := A = Q1R1 Title Page and if Ak = QkRk,, k = 1, 2,... then JJ II J I A := R Q = Q∗Q R Q = Q∗A Q k+1 k k k k k k k k k Page 15 of 37 So the eigenvalues are fixed in the process. Go Back Full Screen Close • One hopes to have “some sort of convergence” of the sequence {Ak}k∈N Quit so that the “limit” would provide the eigenvalue approximation of A. Theorem 4.1. (Francis (1961/62), Kublanovskaja (1961), Huang and Tam (2005)) Suppose that the moduli of the eigenvalues λ1, . , λn of A ∈ GLn(C) are distinct: QR |λ1| > |λ2| > ··· > |λn| (> 0). History Asymptotic result QR iteration Let EE −1 A = Y diag (λ1, . , λn)Y. Assume Home Page Y = LωU, Title Page JJ II where ω is a permutation, L is lower ∆ and U is unit upper ∆.