AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 13: Householder Reflectors; Updating QR Factorization Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 14 Outline 1 Householder Reflectors 2 Givens Rotations and Updating QR Factorization Xiangmin Jiao Numerical Analysis I 2 / 14 Gram-Schmidt as Triangular Orthogonalization Every step of Gram-Schmidt can be viewed as multiplication with triangular matrix. For example, at first step: 2 1 −r12 −r13 ··· 3 r11 r11 r11 6 1 7 h i 6 7 (2) (2) [v1jv2j · · · jvn] 6 1 7 = q1jv2 j · · · jvn ; 4 . 5 .. | {z } R1 Gram-Schmidt therefore multiplies triangular matrices to orthogonalize column vectors, and in turns can be viewed as triangular orthogonalization AR1R2 ··· Rn = Qb | {z } Rb−1 where Ri is triangular matrix A “dual” approach would be orthogonal triangularization, i.e., multiply A by orthogonal matrices to make it triangular matrix Xiangmin Jiao Numerical Analysis I 3 / 14 Householder Triangularization Method introduced by Alston Scott Householder in 1958 It multiplies orthogonal matrices to make column triangular, e.g. 2 × × × 3 2 x x x 3 2 × × × 3 2 × × × 3 6 × × × 7 6 0 x x 7 6 x x 7 6 × × 7 6 7 Q1 6 7 Q2 6 7 Q3 6 7 6 × × × 7 ! 6 0 x x 7 ! 6 0 x 7 ! 6 x 7 6 7 6 7 6 7 6 7 4 × × × 5 4 0 x x 5 4 0 x 5 4 0 5 × × × 0 x x 0 x 0 A Q1A Q2Q1A Q3Q2Q1A After n steps, we get a product of orthogonal matrices Qn ··· Q2Q1 A = R | {z } QT and in turn we get full QR factorization A = QR Qk introduces zeros below diagonal of kth column while preserving zeros below diagonal in preceding columns The key question is how to find Qk Xiangmin Jiao Numerical Analysis I 4 / 14 Householder Reflectors T First, consider Q1: Q1a1 = ka1ke1, where e1 = (1; 0; ··· ; 0) . Why the length is ka1k? Q1 reflects a1 across hyperplane H orthogonal to v = ka1ke1 − a1, and therefore vv T Q = I − 2 1 v T v More generally, I 0 Q = k 0 F where I is (k − 1) × (k − 1) and F is (m − k + 1) × (m − k + 1) such T that Fx = kxk2e1, where x is (ak;k;ak+1;k ; ··· ; am;k ) What is F ? It has similar form as Q1 with v = kxke1 − x. Xiangmin Jiao Numerical Analysis I 5 / 14 Choice of Reflectors We could choose F such that Fx = −kxke1 instead of Fx = kxke1, or more generally, Fx = zkxke1 with jzj = 1 If z 2 C, this leads to an infinite number of QR factorizations If we require z 2 R, we still have two choices Numerically, it is undesirable for v T v to be close to zero for v = zkxke1 − x, and kvk is larger if z = −sign(x1) vv T Therefore, v = −sign(x1)kxke1 − x. Since I − 2 v T v is independent of sign, clear out the factor −1 and obtain v = sign(x1)kxke1 + x For completeness, if x1 = 0, set z to 1 (instead of 0) Xiangmin Jiao Numerical Analysis I 6 / 14 Question: Can A be reused to store both R and vk completely? Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed. Question: What happens if vk is 0 in line 3 of the loop? Householder Algorithm Householder QR Factorization for k = 1 to n x = Ak:m;k vk = sign(x1)kxke1 + x vk = vk =kvk k T Ak:m;k:n = Ak:m;k:n − 2vk (vk Ak:m;k:n) Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A Matrix Q is not formed explicitly but reflection vector vk is stored Xiangmin Jiao Numerical Analysis I 7 / 14 Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed. Question: What happens if vk is 0 in line 3 of the loop? Householder Algorithm Householder QR Factorization for k = 1 to n x = Ak:m;k vk = sign(x1)kxke1 + x vk = vk =kvk k T Ak:m;k:n = Ak:m;k:n − 2vk (vk Ak:m;k:n) Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A Matrix Q is not formed explicitly but reflection vector vk is stored Question: Can A be reused to store both R and vk completely? Xiangmin Jiao Numerical Analysis I 7 / 14 Question: What happens if vk is 0 in line 3 of the loop? Householder Algorithm Householder QR Factorization for k = 1 to n x = Ak:m;k vk = sign(x1)kxke1 + x vk = vk =kvk k T Ak:m;k:n = Ak:m;k:n − 2vk (vk Ak:m;k:n) Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A Matrix Q is not formed explicitly but reflection vector vk is stored Question: Can A be reused to store both R and vk completely? Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed. Xiangmin Jiao Numerical Analysis I 7 / 14 Householder Algorithm Householder QR Factorization for k = 1 to n x = Ak:m;k vk = sign(x1)kxke1 + x vk = vk =kvk k T Ak:m;k:n = Ak:m;k:n − 2vk (vk Ak:m;k:n) Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A Matrix Q is not formed explicitly but reflection vector vk is stored Question: Can A be reused to store both R and vk completely? Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed. Question: What happens if vk is 0 in line 3 of the loop? Xiangmin Jiao Numerical Analysis I 7 / 14 Compute Qx = Q1Q2 ··· Qnx Implicit calculation of Qx for k = n downto 1 T xk:m = xk:m − 2vk (vk xk:m) Question: How to form Q and Q^ , respectively? Answer: Apply x = Im×m or first n columns of I , respectively Applying or Forming Q T Compute Q b = Qn ··· Q1b Implicit calculation of QT b for k = 1 to n T bk:m = bk:m − 2vk (vk bk:m) Xiangmin Jiao Numerical Analysis I 8 / 14 Answer: Apply x = Im×m or first n columns of I , respectively Applying or Forming Q T Compute Q b = Qn ··· Q1b Implicit calculation of QT b for k = 1 to n T bk:m = bk:m − 2vk (vk bk:m) Compute Qx = Q1Q2 ··· Qnx Implicit calculation of Qx for k = n downto 1 T xk:m = xk:m − 2vk (vk xk:m) Question: How to form Q and Q^ , respectively? Xiangmin Jiao Numerical Analysis I 8 / 14 Applying or Forming Q T Compute Q b = Qn ··· Q1b Implicit calculation of QT b for k = 1 to n T bk:m = bk:m − 2vk (vk bk:m) Compute Qx = Q1Q2 ··· Qnx Implicit calculation of Qx for k = n downto 1 T xk:m = xk:m − 2vk (vk xk:m) Question: How to form Q and Q^ , respectively? Answer: Apply x = Im×m or first n columns of I , respectively Xiangmin Jiao Numerical Analysis I 8 / 14 Operation Count T Most work done at step Ak:m;k:n = Ak:m;k:n − 2vk (vk Ak:m;k:n) Flops per iteration: T I ∼ 2(m − k)(n − k) for dot products vk Ak:m;k:n I ∼ (m − k)(n − k) for outer product 2vk (··· ) I ∼ (m − k)(n − k) for subtraction I ∼ 4(m − k)(n − k) total Including outer loop, total flops is n n X X 4(m − k)(n − k) = 4 (mn − km − kn + k2) k=1 k=1 ∼ 4mn2 − 4(m + n)n2=2 + 4n3=3 2 = 2mn2 − n3 3 If m ≈ n, it is more efficient than Gram-Schmidt method, but if m n, similar to Gram-Schmidt Xiangmin Jiao Numerical Analysis I 9 / 14 Outline 1 Householder Reflectors 2 Givens Rotations and Updating QR Factorization Xiangmin Jiao Numerical Analysis I 10 / 14 Givens Rotations Instead of using reflection, we can rotate x to obtain kxke1 cos θ − sin θ A Given rotation R = rotates x 2 2 sin θ cos θ R counterclockwise by θ T T Choose θ to be angle between (xi ; xj ) and (1; 0) , and we have " q # cos θ − sin θ x x2 + x2 i = i j sin θ cos θ xj 0 where x −x cos θ = i ; sin θ = j q 2 2 q 2 2 xi + xj xi + xj Xiangmin Jiao Numerical Analysis I 11 / 14 Givens QR Introduce zeros in column bottom-up, one zero at a time 2 × × × 3 2 × × × 3 2 × × × 3 6 × × × 7 6 × × × 7 6 × × × 7 6 7 (4;5) 6 7 (3;4) 6 7 6 × × × 7 ! 6 × × × 7 ! 6 x x x 7 ::: 6 7 6 7 6 7 4 × × × 5 4 x x x 5 4 0 x x 5 × × × 0 x x × × To zero aij , left-multiply matrix F with Fi:i+1;i:i+1 being rotation matrix and Fkk = 1 for k 6= i; i + 1 Flop count of Givens QR is 3mn2 − n3, which is about 50% more expensive than Householder QR Xiangmin Jiao Numerical Analysis I 12 / 14 Adding a Row m×n Suppose A 2 R with m ≥ n, and A has full rank 2 3 A1 A1 Let A~ = 4 zT 5, where A = and zT is a new row inserted A2 A2 Obtain A~ = Q~ R~ from A = QR efficiently using Givens rotation: A1 Q1 I Suppose A = = R.