Mathematics & Statistics Auburn University, Alabama, USA
QR History Dec 17, 2010 Asymptotic result QR iteration QR decomposition: History and its EE
Applications Home Page
Title Page Tin-Yau Tam èèèUUUÎÎÎ JJ II J I
Page 1 of 37
Æâ§w Go Back fÆêÆÆÆ Full Screen
Close email: [email protected] Website: www.auburn.edu/∼tamtiny Quit 1. QR decomposition
Recall the QR decomposition of A ∈ GLn(C): QR A = QR History Asymptotic result QR iteration where Q ∈ GLn(C) is unitary and R ∈ GLn(C) is upper ∆ with positive EE diagonal entries. Such decomposition is unique. Set
Home Page a(A) := diag (r11, . . . , rnn) Title Page where A is written in column form JJ II
J I
A = (a1| · · · |an) Page 2 of 37
Go Back Geometric interpretation of a(A): Full Screen rii is the distance (w.r.t. 2-norm) between ai and span {a1, . . . , ai−1}, Close i = 2, . . . , n. Quit Example: 12 −51 4 6/7 −69/175 −58/175 14 21 −14 6 167 −68 = 3/7 158/175 6/175 0 175 −70 . QR −4 24 −41 −2/7 6/35 −33/35 0 0 35 History Asymptotic result QR iteration EE
• QR decomposition is the matrix version of the Gram-Schmidt orthonor- Home Page malization process. Title Page
JJ II • QR decomposition can be extended to rectangular matrices, i.e., if A ∈ J I m×n with m ≥ n (tall matrix) and full rank, then C Page 3 of 37
A = QR Go Back
Full Screen where Q ∈ Cm×n has orthonormal columns and R ∈ Cn×n is upper ∆ Close with positive “diagonal” entries. Quit 2. QR history
QR • When Erhard Schmidt presented the formulae on p. 442 of his History Asymptotic result E. Schmidt, Zur Theorie der linearen und nichtlinearen Integralgle- QR iteration EE ichungen. I. Teil: Entwicklung willkulicher¨ Funktionen nach Systemen vorgeschriebener, Math. Ann., 63 (1907) 433–476. Home Page he said that essentially the same formulae were in Title Page J. P. Gram, Ueber die Entwickelung reeler Funtionen in Reihen mittelst JJ II der Methode der kleinsten Quadrate, Jrnl. fur¨ die reine und angewandte J I Math. 94 (1883) 71–73. Page 4 of 37
Go Back • Modern writers, however, distinguish the two procedures, sometimes us- Full Screen ing the term “Gram-Schmidt” for the Schmidt form and “modified Gram- Close
Schmidt” for the Gram version. Quit • But Gram-Schmidt orthonormalization appeared earlier in the work of QR History Laplace and Cauchy. Asymptotic result QR iteration EE • In the theory of semisimple Lie grougs, Gram-Schmidt process is
extended to the Iwasawa decomposition G = KAN. Home Page
Title Page • A JSTOR search: the term “Gram-Schmidt orthogonalization process” JJ II
first appears on p.57 of J I
Y. K. Wong, An Application of Orthogonalization Process to the Theory Page 5 of 37 of Least Squares,” Annals of Mathematical Statistics, 6 (1935), 53–75. Go Back
Full Screen
Close
Quit • In 1801 Gauss predicted the orbit of the steroid Ceres using the method of least squares. Since then, the principle of least squares has been the standard procedure for the analysis of scientific data.
QR History • Least squares problem Asymptotic result Ax ≈ b QR iteration EE i.e., finding xˆ that would yield
Home Page min kAx − bk2. x Title Page
The solution is characterized by r ⊥ R(A), where r = b − Ax is the JJ II residual vector, or equivalently, given by the normal equation J I
Page 6 of 37 ∗ ∗ A Ax = A b. Go Back
Full Screen • With A = QR and A ∈ Cm×n tall with full rank Close ∗ ∗ ∗ ∗ ∗ R Rx = (QR) QRx = R Q b ⇔ Rx = Q b. Quit There are several methods for computing the QR decomposition:
QR • GS or modified GS, History Asymptotic result • Givens rotations (real A), or QR iteration EE • Householder reflections.
Home Page
Disadvantage of GS: sensitive to rounding error (orthogonality of the Title Page computed vectors can be lost quickly or may even be completely lost) → JJ II modified Gram-Schmidt. J I
Page 7 of 37
Idea of modified GS: do the projection step with a number of projections Go Back which will be against the errors introduced in computation. Full Screen
Close
Quit Example: 1 + 1 1 A = 1 1 + 1 QR History 1 1 1 + Asymptotic result 2 QR iteration with very small such that 3+2 will be computed accurately but 3+2+ EE will be computed as 3 + 2. For example, || < 10−10. Then
Home Page √1+ √−1 √−1 3+2 2 2 Title Page Q = √ 1 √1 0 3+2 2 JJ II √ 1 0 √1 3+2 2 J I and cos θ12 and cos θ13 ≈ π/2 but and cos θ23 ≈ π/3. Page 8 of 37
Go Back
See http://www-db.stanford.edu/TR/CS-TR-69-122.html for a heuristic Full Screen analysis of why Gram-Schmidt not stable. Close
Quit Computing QR by Householder reflections A Householder reflection is a reflection about some hyperplane. Consider
∗ QR Qv = I − 2vv , kvk2 = 1. History Asymptotic result QR iteration Qv sends v to −v and fix pointwise the hyperplane ⊥ to v. Householder EE reflections are Hermitian and unitary. T Let e1 = (1, 0, ..., 0) . Recall Home Page
Title Page A = (a1| · · · |an) ∈ GLn(C) JJ II
If ka1k2 = α1, set J I
u ∗ Page 9 of 37 u = a1 − α1e1, v = ,Q1 = I − 2vv kuk2 Go Back so that Full Screen
T Close Qa1 = (α1, 0, ··· , 0) = α1e1.
Quit Then α1 ∗ ... ∗ 0 Q1A = . QR . A1 History 0 Asymptotic result QR iteration After t iterations of this process, t ≤ n − 1, EE
R = Qt ··· Q2Q1A Home Page is upper ∆. So, with Title Page
Q = Q1Q2 ··· Qt JJ II A = QR is the QR decomposition of A. J I Page 10 of 37
Go Back • This method has greater numerical stability than GS. Full Screen • On the other hand, GS produces the qj vector after the jth iteration, Close while Householder reflections produces all the vectors only at the end. Quit 3. An Asymptotic result
Theorem 3.1. (Huang and Tam 2007) Given A ∈ GLn(C). Let A = QR Y −1JY be the Jordan decomposition of A, where J is the Jordan form of History Asymptotic result A, QR iteration EE diag J = diag (λ1, . . . , λn) satisfying |λ | ≥ · · · ≥ |λ |. Then 1 n Home Page
m 1/m Title Page lim a(A ) = diag (|λω(1)|,..., |λω(n)|), m→∞ JJ II where the permutation ω is uniquely determined by the Gelfand-Naimark J I decomposition of Y = LωU: Page 11 of 37
Go Back rank ω(i|j) = rank (Y )(i|j), 1 ≤ i, j ≤ n. Full Screen
Here ω(i|j) denotes the submatrix formed by the first i rows and the first Close j columns of ω, 1 ≤ i, j ≤ n. Quit • Gelfand-Naimark decomposition of Y = LωU is different from the T Guassian decomposition Y = P LU obtained by Gaussian elimination QR History with row exchanges. Asymptotic result QR iteration EE • None of P , U and L in the Gaussian decomposition is unique.
Home Page
• But ω and diag U are unique in the Gelfand-Naimark decomposition. Title Page
JJ II
H. Huang and T.Y. Tam, An asymptotic behavior of QR decomposition, J I
Linear Algebra and Its Applications, 424 (2007) 96-107. Page 12 of 37
H. Huang and T.Y. Tam, An asymptotic result on the a-component in Iwa- Go Back sawa decomposition, Journal of Lie Theory, 17 (2007) 469-479. Full Screen
Close
Quit Numerical experiments: Computing the discrepancy between
QR m 1/m [a(A )] and |λ(A)| History Asymptotic result QR iteration of randomly generated A ∈ GLn(C). EE
The graph of Home Page m 1/m 100 k[a(A )] − diag (|λ1|,..., |λn|)k2 Title Page
JJ II 80 versus m (m = 1,..., 100) J I 60 Page 13 of 37
40 Go Back
Full Screen 20
0 20 40 60 80 100 Close
Quit If we consider m 1/m |a1(A ) − |λ1(A)|| QR m 1/m History instead of k[a(A )] − diag (|λ1|,..., |λn|)k2 for the above example, Asymptotic result convergence occurs. QR iteration EE The graph of
Home Page 70 m 1/m |a1(A ) − |λ1(A)|| Title Page 60
50 versus m (m = 1,..., 100) JJ II
40 J I
30 Page 14 of 37
20 Go Back 10 Full Screen 0 20 40 60 80 100 Close
Quit 4. QR iteration
• Because of Abel’s theorem (1824), the roots of a general fifth order polynomial cannot be solved by radicals. Thus the computation of eigen- QR History values of A ∈ Cn×n has to be approximative. Asymptotic result QR iteration EE
• Given A ∈ GLn(C), define a sequence {Ak}k∈N of matrices with
Home Page A1 := A = Q1R1 Title Page and if Ak = QkRk,, k = 1, 2,... then JJ II
J I A := R Q = Q∗Q R Q = Q∗A Q k+1 k k k k k k k k k Page 15 of 37
So the eigenvalues are fixed in the process. Go Back Full Screen
Close • One hopes to have “some sort of convergence” of the sequence {Ak}k∈N Quit so that the “limit” would provide the eigenvalue approximation of A. Theorem 4.1. (Francis (1961/62), Kublanovskaja (1961), Huang and Tam
(2005)) Suppose that the moduli of the eigenvalues λ1, . . . , λn of A ∈
GLn(C) are distinct: QR |λ1| > |λ2| > ··· > |λn| (> 0). History Asymptotic result QR iteration Let EE −1 A = Y diag (λ1, . . . , λn)Y.
Assume Home Page Y = LωU, Title Page JJ II where ω is a permutation, L is lower ∆ and U is unit upper ∆. Then J I
1. the strictly lower ∆ part of Ak converge to zero. Page 16 of 37
Go Back 2. diag Ak → diag (λω(1), . . . , λω(n)). Full Screen
H. Huang and T.Y. Tam, On the QR iterations of real matrices, Linear Close
Algebra and Its Applications, 408 (2005) 161-176. Quit • It rates as one of the most important algorithmic developments of the QR History past century Asymptotic result QR iteration • Parlett (2000): “The QR algorithm solves the eigenvalue problem in EE a very satisfactory way ... What makes experts in matrix computations happy is that this algorithm is a genuinely new contribution to the field Home Page of numerical analysis and not just a refinement of ideas given by Newton, Title Page
Gauss, Hadamard, or Schur.” JJ II
• Higham (2003): “The QR algorithm for solving the nonsymmetric J I eigenvalue problem is one of the jewels in the crown of matrix compu- Page 17 of 37 tations.” Go Back
Full Screen
Close
Quit A common misconception • Horn and Johnson’s “Matrix Analysis” (p.114):
Under some circumstances (for example, if all the eigenvalues of A0 has distinct absolute values), the QR iterates Ak will converge to an upper QR triangular matrix as k → ∞.... History Asymptotic result QR iteration EE • Quarteroni, Sacco, and Saleri’s “Numerical Mathematics” (p.201-202): n×n Let A ∈ R be a matrix with real eigenvalues such that |λ1| > |λ2| > Home Page ··· > |λn|. Then Title Page λ1 t12 . . . t1n JJ II (k) 0 λ2 t23 ... J I lim T = k→∞ 0 ··· ... Page 18 of 37 Go Back 0 ··· λn
Full Screen • Wikipedia: Under certain conditions, the matrices Ak converge to a Close triangular matrix, the Schur form of A. Quit QR D. Serre’s “Theory of Matrices” (p.176-177): History Asymptotic result Let us recall that the sequence Ak is not always convergent. For example, QR iteration if A is already triangular, its QR factorization is Q = D, R = D−1A, with EE −1 dj = ajj/|ajj|. Hence, A1 = D AD is triangular, with the same diagonal
Home Page as that of A. By induction, Ak is triangular, with the same diagonal as that −k k Title Page of A so that Ak = D AD .... Hence, the part above the diagonal of Ak does not converge. JJ II Summing up, a convergence theorem may concern only the diagonal of J I Page 19 of 37 Ak and what is below it. Go Back
Full Screen
Close
Quit More on QR algorithm
• The QR algorithm is numerically stable because it proceeds by orthog- onal/unitary similarity transforms. QR History Asymptotic result • QR iteration EE A → Hessenberg form → QR algorithm
It would be cost effective if we convert A to an upper Hessenberg form, Home Page with a finite sequence of orthogonal similarities. Then determine the QR Title Page 2 decomposition of an upper Hessenberg matrix costs 6n +O(n) arithmetic JJ II operations. J I
• A practical algorithm will use shifts to increase separation and accelerate Page 20 of 37 convergence. Go Back • The QR iteration is further extended to semisimple Lie groups: Full Screen H. Huang. R.R. Holmes and T.Y.Tam, Asymptotic behavior of Iwasawa Close and Cholesky iterations, manuscript. Quit 5. Application in MIMO QR History In radio, multiple-input and multiple-output, or MIMO, is the use of mul- Asymptotic result QR iteration tiple antennas at both the transmitter and receiver to improve communica- EE tion performance.
Home Page
Title Page
JJ II
J I
Page 21 of 37
Go Back
Full Screen
Close
Quit QR History Asymptotic result QR iteration EE
Home Page
Title Page
JJ II • MIMO is one of several forms of smart antenna technology. J I • MIMO offers significant increases in data throughput and link range Page 22 of 37 Go Back without additional bandwidth or transmit power. Full Screen • MIMO is an important part of modern wireless communication stan- Close dards such as IEEE 802.11n (Wifi) and 4G. Quit • In MIMO systems, a transmitter sends multiple streams by multiple transmit antennas. The transmit streams go through a matrix channel which consists of all NtNr paths between the Nt transmit antennas at the QR transmitter and Nr receive antennas at the receiver. History Asymptotic result QR iteration EE
Home Page
Title Page
JJ II
J I
Page 23 of 37
Go Back • Then, the receiver gets the received signal vectors by the multiple re- Full Screen ceive antennas and decodes the received signal vectors into the original Close information. Quit Mathematical description T Let x = (x1, . . . , xn) be a vector to be transmitted over a noisy channel.
Each xi is chosen from a finite-size alphabet X .
QR History A general MIMO system is modeled as Asymptotic result QR iteration r = Hx + ξ EE
• H is the m × n full column rank channel (tall, i.e., m ≥ n) matrix Home Page
(known to the receiver) Title Page T ∗ 2 • ξ = (ξ1, . . . , ξm) is a white Gaussian noise vector where E(ξξ ) = σ I JJ II T • r = (r1, . . . , rm) is the observed received vector. J I T n • Our task is to detect/estimate the vector ˆx = (xˆ1,..., xˆn) ∈ X given Page 24 of 37 the noisy observation r. Go Back
Full Screen
• QR decomposition of the channel matrix H can be used to form the Close back-cancellation detector. Quit A. Successive Cancellation Detection Using QR Decomposition (1) QR decomposition
Let H = QR be the QR decomposition of the m × n channel matrix H, QR i.e., History Asymptotic result • Q = m × n matrix with orthonormal columns QR iteration EE • R = n × n upper ∆ matrix. So Home Page r = Hx + ξ ⇒ Q∗r = Rx + Q∗ξ Title Page
Set JJ II ∗ ˜ ∗ ˜r := Q r, ξ := Q ξ J I So Page 25 of 37 ˜ Go Back r˜1 r11 r12 ··· r1n x1 ξ1
˜ Full Screen r˜2 0 r22 ··· r2n x2 ξ2 . = . . . . + . . ...... . . Close ˜ Quit r˜n 0 0 ··· rnn xn ξn (2) Hard decision
Estimate xn by making the hard decision: QR r˜n History xˆn := Quant [ ] Asymptotic result rnn QR iteration EE where Quant (t) is the element X that is closest w.r.t. 2-norm to t. (3) Cancellation Home Page
Title Page Backward substitutions: JJ II r˜n xˆn := Quant [ ] J I rnn Page 26 of 37 Pn r˜k − i=k+1 rkmxˆi Go Back xˆk := Quant [ ], k = n − 1, n − 2,..., 1 rkk Full Screen • The algorithm is essentially least squares solution via QR decomposi- Close tion. Quit B. Optimally Ordered Detection • Golden et al proposed a vertical Bell Laboratories layered space-time (V-BLAST) system with an optimal ordered detection algorithm that QR maximizes the signal-to-noise ratio (SNR). History Asymptotic result QR iteration G.D. Golden, G.J. Foschini, R.A. Valenzuela, and P.W. Wolniansky, EE Detection algorithm and initial laboratory results using V-BLAST space- time communication architecture, Electron. Lett., vol. 35, pp. 14õ15, Home Page Jan. 1999. Title Page
JJ II • The idea is (equivalently) to find an n × n permutation matrix P such J I that if Page 27 of 37 H0 := HP = QR Go Back Full Screen then the distance between h0 and the other columns h0 ,..., h0 is max- n 1 n−1 Close imal. Then repeat the process by stripping off the column vectors one. Quit • Doing so amounts to finding a subchannel whose SNR is the highest QR among all n possible subchannels. Geometrically it is to find the column History of H whose distance to the span of the other columns of H is maximum. Asymptotic result QR iteration Then do it recursively. EE • Equivalently 0 0 r = H x + ξ Home Page where Title Page H0 := HP, x = Px0 JJ II J I i.e., if we precode a vector with the permutation matrix P, and apply Algo- Page 28 of 37 0 rithm A to detect x , we get the optimally ordered successive-cancellation Go Back detector of Golden et al. Full Screen
Close
Quit A recent optimal QR decomposition, called equal-diagonal QR decompo- QR History sition, or briefly the QRS decomposition in introduced in Asymptotic result QR iteration EE J.K. Zhang, A. Kavciˇ c´ and K. M. Wong, Equal-Diagonal QR Decompo-
sition and its Application to Precoder Design for Successive-Cancellation Home Page
Detection, IEEE Transactions on Information Theory, 51 (2005) 154–172. Title Page
JJ II
The QRS decomposition is applied to precode successive-cancellation de- J I tection, where we assume that both the transmitter and the receiver have Page 29 of 37 perfect channel knowledge. Go Back
Full Screen
Close
Quit Theorem 5.1. (Zhang, Kavciˇ c´ and Wong 2005) For any channel matrix H, there exists a unitary precoder S, such that the nonzero diagonal entries of QR the upper ∆ matrix R are all equal where History Asymptotic result QR iteration HS = QR. EE
Nice properties The precoder S and the resulting successive-cancellation detector have Home Page Title Page many nice properties. • The minimum Euclidean distance between two signal points at the JJ II J I channel output is equal to the minimum Euclidean distance between two Page 30 of 37 constellation points at the precoder input up to a multiplicative factor that Go Back equals the diagonal entry in the R-factor. Full Screen
Close
Quit • The superchannel HS naturally exhibits an optimally ordered column permutation, i.e., the optimal detection order for the vertical Bell Labs QR layered spaceõtime (V-BLAST) detector is the natural order. History Asymptotic result • The precoder S minimizes the block error probability of the QR succes- QR iteration sive cancellation detector. EE • A lower and an upper bound for the free distance at the channel output is expressible in terms of the diagonal entries of the R-factor in the QR Home Page decomposition of a channel matrix. Title Page • The precoder S maximizes the lower bound of the channels free distance JJ II subject to a power constraint. J I Page 31 of 37 • For the optimal precoder S, the performance of the QR detector is Go Back asymptotically (at large signal-to-noise ratios (SNRs)) equivalent to that Full Screen of the maximum-likelihood detector (MLD) that uses the same precoder. Close
Quit • Recall the QRS decomposition HS = QR Let
H = UΣV s1 QR .. History . ∗ be the SVD of H where Σ = . Then set S = V Sˆ Asymptotic result s QR iteration n O EE ! diag (s , . . . , s )Sˆ HS = HV∗Sˆ = UΣSˆ = U 1 n Home Page 0 Title Page where the unitary Sˆ is to be determined so that R has equal diagonal entries. JJ II Clearly the R factors of ΣSˆ and UΣSˆ are the same. J I
WLOG, assume that m = n, i.e., Σ = diag (s1, . . . , sn). The problem is reduced Page 32 of 37 to finding a unitary S such that Go Back
Full Screen ΣS = QR Close and R has identical diagonal entries. Quit s1 .. Theorem 5.2. (Kostant 1973) Let Σ = . ∈ Cn×n s1 ≥ · · · ≥ QR s History n Asymptotic result sn > 0. For any unitary S ∈ Cn×n, if ΣS = QR, where Q ∈ Cm×n has QR iteration EE orthonormal columns, R ∈ Cn×n is upper ∆, then k k Y 0 Y Home Page rii ≤ si, k = 1, . . . , n − 1 (1) i=1 i=1 Title Page n n Y 0 Y JJ II rii = si, (2) i=1 i=1 J I 0 0 Page 33 of 37 where r11 ≥ · · · ≥ rnn are the rearrangement of r11, . . . , rnn. Conversely Go Back if (1) and (2) are satisfied, then there is an n × n unitary S such that the Full Screen diagonal entries of R in ΣS = QR are those rii, i = 1, . . . , n. Close
Quit B. Kostant, On convexity, the Weyl group and Iwasawa decomposition. QR Ann. Sci. Ecole Norm. Sup. (4), 6 (1973) 413–460. History Asymptotic result • Clearly Kostant (1973) ⇒ Zhang, Kavciˇ c´ and Wong (2005). QR iteration • The proof of Kostant is not constructive. Indeed his result is true for EE semisimple Lie groups.
Home Page
• The construction of the S precoder in [ZKW] is not very cost effective Title Page and involves a number of steps. Determinant is also involved. JJ II J I
Page 34 of 37 Shuangchi He and Tin-Yau Tam, On equal-diagonal QR decomposition, Go Back manuscript. Full Screen
Close
Quit Use induction. WLOG assume that m = n. 2 × 2 case: ! ! ! QR s1 0 cos θ − sin θ s1 cos θ −s1 sin θ History ΣS = = Asymptotic result 0 s2 sin θ cos θ s2 sin θ s2 cos θ QR iteration EE Then for any s1 ≥ µ ≥ s2,
T 2 2 k(s1 cos θ, s2 sin θ) k2 = s1 cos θ + s2 sin θ = µ Home Page √ Title Page for some θ ∈ . In particular µ = s1s2 R JJ II Suppose that the statement holds true for n − 1. J I
For r = n > 2, suppose that k is the smallest index such that sk−1 ≥ λ ≥ Page 35 of 37 sk, where Go Back n Y 1/n λ := ( si) . Full Screen
i=1 Close
Quit There is a unitary S1 ∈ C2×2 such that ! λ ∗ diag (s1, sk)S1 = A1 := . s1sk 0 λ QR History Set S2 = S1 ⊕ In−2. Then Asymptotic result QR iteration EE A2 := diag (s1, sk, s2, ··· , sk−1, sk+1, ··· , sn) S2
= A1 ⊕ diag (s2, ··· , sk−1, sk+1, ··· , sn) . Home Page
Note that Title Page n s1sk Y λn−1 = s . JJ II λ i i=2,i6=k J I
Page 36 of 37 By the inductive hypothesis, there exists a unitary S3 ∈ C(n−1)×(n−1) such that Go Back
s1sk Full Screen A3 = diag , s2, ··· , sk−1, sk+1, ··· , sn S3 λ Close is upper ∆ with equal-diagonal entries λ. Then S4 = 1 ⊕ S3. Quit T HAN K YOU FOR YOURATTENTION
QR History Asymptotic result QR iteration EE
Home Page
Title Page
JJ II
J I
Page 37 of 37
Go Back
Full Screen
Close
Quit