AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 13: Householder Reflectors; Updating QR Factorization
Xiangmin Jiao
Stony Brook University
Xiangmin Jiao Numerical Analysis I 1 / 14 Outline
1 Householder Reflectors
2 Givens Rotations and Updating QR Factorization
Xiangmin Jiao Numerical Analysis I 2 / 14 Gram-Schmidt as Triangular Orthogonalization Every step of Gram-Schmidt can be viewed as multiplication with triangular matrix. For example, at first step: 1 −r12 −r13 ··· r11 r11 r11 1 h i (2) (2) [v1|v2| · · · |vn] 1 = q1|v2 | · · · |vn , . .. | {z } R1 Gram-Schmidt therefore multiplies triangular matrices to orthogonalize column vectors, and in turns can be viewed as triangular orthogonalization AR1R2 ··· Rn = Qb | {z } Rb−1 where Ri is triangular matrix A “dual” approach would be orthogonal triangularization, i.e., multiply A by orthogonal matrices to make it triangular matrix
Xiangmin Jiao Numerical Analysis I 3 / 14 Householder Triangularization Method introduced by Alston Scott Householder in 1958 It multiplies orthogonal matrices to make column triangular, e.g. × × × x x x × × × × × × × × × 0 x x x x × × Q1 Q2 Q3 × × × → 0 x x → 0 x → x × × × 0 x x 0 x 0 × × × 0 x x 0 x 0 A Q1A Q2Q1A Q3Q2Q1A After n steps, we get a product of orthogonal matrices
Qn ··· Q2Q1 A = R | {z } QT and in turn we get full QR factorization A = QR Qk introduces zeros below diagonal of kth column while preserving zeros below diagonal in preceding columns The key question is how to find Qk
Xiangmin Jiao Numerical Analysis I 4 / 14 Householder Reflectors
T First, consider Q1: Q1a1 = ka1ke1, where e1 = (1, 0, ··· , 0) . Why the length is ka1k?
Q1 reflects a1 across hyperplane H orthogonal to v = ka1ke1 − a1, and therefore vv T Q = I − 2 1 v T v More generally, I 0 Q = k 0 F where I is (k − 1) × (k − 1) and F is (m − k + 1) × (m − k + 1) such T that Fx = kxk2e1, where x is (ak,k,ak+1,k , ··· , am,k )
What is F ? It has similar form as Q1 with v = kxke1 − x.
Xiangmin Jiao Numerical Analysis I 5 / 14 Choice of Reflectors
We could choose F such that Fx = −kxke1 instead of Fx = kxke1, or more generally, Fx = zkxke1 with |z| = 1 If z ∈ C, this leads to an infinite number of QR factorizations If we require z ∈ R, we still have two choices Numerically, it is undesirable for v T v to be close to zero for v = zkxke1 − x, and kvk is larger if z = −sign(x1) vv T Therefore, v = −sign(x1)kxke1 − x. Since I − 2 v T v is independent of sign, clear out the factor −1 and obtain v = sign(x1)kxke1 + x
For completeness, if x1 = 0, set z to 1 (instead of 0)
Xiangmin Jiao Numerical Analysis I 6 / 14 Question: Can A be reused to store both R and vk completely? Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed.
Question: What happens if vk is 0 in line 3 of the loop?
Householder Algorithm
Householder QR Factorization for k = 1 to n x = Ak:m,k vk = sign(x1)kxke1 + x vk = vk /kvk k T Ak:m,k:n = Ak:m,k:n − 2vk (vk Ak:m,k:n)
Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A
Matrix Q is not formed explicitly but reflection vector vk is stored
Xiangmin Jiao Numerical Analysis I 7 / 14 Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed.
Question: What happens if vk is 0 in line 3 of the loop?
Householder Algorithm
Householder QR Factorization for k = 1 to n x = Ak:m,k vk = sign(x1)kxke1 + x vk = vk /kvk k T Ak:m,k:n = Ak:m,k:n − 2vk (vk Ak:m,k:n)
Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A
Matrix Q is not formed explicitly but reflection vector vk is stored
Question: Can A be reused to store both R and vk completely?
Xiangmin Jiao Numerical Analysis I 7 / 14 Question: What happens if vk is 0 in line 3 of the loop?
Householder Algorithm
Householder QR Factorization for k = 1 to n x = Ak:m,k vk = sign(x1)kxke1 + x vk = vk /kvk k T Ak:m,k:n = Ak:m,k:n − 2vk (vk Ak:m,k:n)
Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A
Matrix Q is not formed explicitly but reflection vector vk is stored
Question: Can A be reused to store both R and vk completely? Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed.
Xiangmin Jiao Numerical Analysis I 7 / 14 Householder Algorithm
Householder QR Factorization for k = 1 to n x = Ak:m,k vk = sign(x1)kxke1 + x vk = vk /kvk k T Ak:m,k:n = Ak:m,k:n − 2vk (vk Ak:m,k:n)
Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A
Matrix Q is not formed explicitly but reflection vector vk is stored
Question: Can A be reused to store both R and vk completely? Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed.
Question: What happens if vk is 0 in line 3 of the loop?
Xiangmin Jiao Numerical Analysis I 7 / 14 Compute Qx = Q1Q2 ··· Qnx
Implicit calculation of Qx for k = n downto 1 T xk:m = xk:m − 2vk (vk xk:m)
Question: How to form Q and Qˆ , respectively?
Answer: Apply x = Im×m or first n columns of I , respectively
Applying or Forming Q
T Compute Q b = Qn ··· Q1b
Implicit calculation of QT b for k = 1 to n T bk:m = bk:m − 2vk (vk bk:m)
Xiangmin Jiao Numerical Analysis I 8 / 14 Answer: Apply x = Im×m or first n columns of I , respectively
Applying or Forming Q
T Compute Q b = Qn ··· Q1b
Implicit calculation of QT b for k = 1 to n T bk:m = bk:m − 2vk (vk bk:m)
Compute Qx = Q1Q2 ··· Qnx
Implicit calculation of Qx for k = n downto 1 T xk:m = xk:m − 2vk (vk xk:m)
Question: How to form Q and Qˆ , respectively?
Xiangmin Jiao Numerical Analysis I 8 / 14 Applying or Forming Q
T Compute Q b = Qn ··· Q1b
Implicit calculation of QT b for k = 1 to n T bk:m = bk:m − 2vk (vk bk:m)
Compute Qx = Q1Q2 ··· Qnx
Implicit calculation of Qx for k = n downto 1 T xk:m = xk:m − 2vk (vk xk:m)
Question: How to form Q and Qˆ , respectively?
Answer: Apply x = Im×m or first n columns of I , respectively
Xiangmin Jiao Numerical Analysis I 8 / 14 Operation Count
T Most work done at step Ak:m,k:n = Ak:m,k:n − 2vk (vk Ak:m,k:n) Flops per iteration: T I ∼ 2(m − k)(n − k) for dot products vk Ak:m,k:n I ∼ (m − k)(n − k) for outer product 2vk (··· ) I ∼ (m − k)(n − k) for subtraction I ∼ 4(m − k)(n − k) total Including outer loop, total flops is
n n X X 4(m − k)(n − k) = 4 (mn − km − kn + k2) k=1 k=1 ∼ 4mn2 − 4(m + n)n2/2 + 4n3/3 2 = 2mn2 − n3 3 If m ≈ n, it is more efficient than Gram-Schmidt method, but if m n, similar to Gram-Schmidt Xiangmin Jiao Numerical Analysis I 9 / 14 Outline
1 Householder Reflectors
2 Givens Rotations and Updating QR Factorization
Xiangmin Jiao Numerical Analysis I 10 / 14 Givens Rotations
Instead of using reflection, we can rotate x to obtain kxke1 cos θ − sin θ A Given rotation R = rotates x ∈ 2 sin θ cos θ R counterclockwise by θ T T Choose θ to be angle between (xi , xj ) and (1, 0) , and we have " q # cos θ − sin θ x x2 + x2 i = i j sin θ cos θ xj 0
where x −x cos θ = i , sin θ = j q 2 2 q 2 2 xi + xj xi + xj
Xiangmin Jiao Numerical Analysis I 11 / 14 Givens QR
Introduce zeros in column bottom-up, one zero at a time
× × × × × × × × × × × × × × × × × × (4,5) (3,4) × × × → × × × → x x x ... × × × x x x 0 x x × × × 0 x x × ×
To zero aij , left-multiply matrix F with Fi:i+1,i:i+1 being rotation matrix and Fkk = 1 for k 6= i, i + 1 Flop count of Givens QR is 3mn2 − n3, which is about 50% more expensive than Householder QR
Xiangmin Jiao Numerical Analysis I 12 / 14 Adding a Row
m×n Suppose A ∈ R with m ≥ n, and A has full rank A1 A1 Let A˜ = zT , where A = and zT is a new row inserted A2 A2 Obtain A˜ = Q˜ R˜ from A = QR efficiently using Givens rotation: A1 Q1 I Suppose A = = R. A2 Q2 A1 0 Q1 T T T z I Then A˜ = z = 1 0 R A2 0 Q2 T T T T z I Perform series of Givens rotation R˜ = U ... U U , and n 2 1 R 0 Q1 ˜ T then Q = 1 0 U1U2 ... Un 0 Q2 2 I Updating R˜ costs 3n flops, and updating Q˜ costs 6mn flops
Xiangmin Jiao Numerical Analysis I 13 / 14 Adding a Column
m×n Suppose A ∈ R with m ≥ n, and A has full rank Let A˜ = A1 z A2 , where A = A1 A2 and z is new column Obtain A˜ = Q˜ R˜ from A = QR efficiently using Givens rotation: I Suppose A = A1 A2 = Q R1 R2 T I Then A˜ = A1 z A2 = Q R1 w R2 , where w = Q z I Perform series of Givens rotation R˜ = Uk+1 ··· Un R1 w R2 , where Un performs on rows n and n − 1, Un−1 performs on rows n − 1 and n − 2, etc. ˜ T T I Q = QUn ··· Uk+1 I It takes O(mn) time overall
Xiangmin Jiao Numerical Analysis I 14 / 14