AMS526: Numerical Analysis I () Lecture 13: Householder Reflectors; Updating QR Factorization

Xiangmin Jiao

Stony Brook University

Xiangmin Jiao Numerical Analysis I 1 / 14 Outline

1 Householder Reflectors

2 Givens Rotations and Updating QR Factorization

Xiangmin Jiao Numerical Analysis I 2 / 14 Gram-Schmidt as Triangular Orthogonalization Every step of Gram-Schmidt can be viewed as multiplication with triangular . For example, at first step:  1 −r12 −r13 ···  r11 r11 r11  1  h i   (2) (2) [v1|v2| · · · |vn]  1  = q1|v2 | · · · |vn ,  .  .. | {z } R1 Gram-Schmidt therefore multiplies triangular matrices to orthogonalize column vectors, and in turns can be viewed as triangular orthogonalization AR1R2 ··· Rn = Qb | {z } Rb−1 where Ri is A “dual” approach would be orthogonal triangularization, i.e., multiply A by orthogonal matrices to make it triangular matrix

Xiangmin Jiao Numerical Analysis I 3 / 14 Householder Triangularization Method introduced by Alston Scott Householder in 1958 It multiplies orthogonal matrices to make column triangular, e.g.  × × ×   x x x   × × ×   × × ×   × × ×   0 x x   x x   × ×    Q1   Q2   Q3    × × ×  →  0 x x  →  0 x  →  x           × × ×   0 x x   0 x   0  × × × 0 x x 0 x 0 A Q1A Q2Q1A Q3Q2Q1A After n steps, we get a product of orthogonal matrices

Qn ··· Q2Q1 A = R | {z } QT and in turn we get full QR factorization A = QR Qk introduces zeros below diagonal of kth column while preserving zeros below diagonal in preceding columns The key question is how to find Qk

Xiangmin Jiao Numerical Analysis I 4 / 14 Householder Reflectors

T First, consider Q1: Q1a1 = ka1ke1, where e1 = (1, 0, ··· , 0) . Why the length is ka1k?

Q1 reflects a1 across hyperplane H orthogonal to v = ka1ke1 − a1, and therefore vv T Q = I − 2 1 v T v More generally,  I 0  Q = k 0 F where I is (k − 1) × (k − 1) and F is (m − k + 1) × (m − k + 1) such T that Fx = kxk2e1, where x is (ak,k,ak+1,k , ··· , am,k )

What is F ? It has similar form as Q1 with v = kxke1 − x.

Xiangmin Jiao Numerical Analysis I 5 / 14 Choice of Reflectors

We could choose F such that Fx = −kxke1 instead of Fx = kxke1, or more generally, Fx = zkxke1 with |z| = 1 If z ∈ C, this leads to an infinite number of QR factorizations If we require z ∈ R, we still have two choices Numerically, it is undesirable for v T v to be close to zero for v = zkxke1 − x, and kvk is larger if z = −sign(x1) vv T Therefore, v = −sign(x1)kxke1 − x. Since I − 2 v T v is independent of sign, clear out the factor −1 and obtain v = sign(x1)kxke1 + x

For completeness, if x1 = 0, set z to 1 (instead of 0)

Xiangmin Jiao Numerical Analysis I 6 / 14 Question: Can A be reused to store both R and vk completely? Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed.

Question: What happens if vk is 0 in line 3 of the loop?

Householder Algorithm

Householder QR Factorization for k = 1 to n x = Ak:m,k vk = sign(x1)kxke1 + x vk = vk /kvk k T Ak:m,k:n = Ak:m,k:n − 2vk (vk Ak:m,k:n)

Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A

Matrix Q is not formed explicitly but reflection vector vk is stored

Xiangmin Jiao Numerical Analysis I 7 / 14 Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed.

Question: What happens if vk is 0 in line 3 of the loop?

Householder Algorithm

Householder QR Factorization for k = 1 to n x = Ak:m,k vk = sign(x1)kxke1 + x vk = vk /kvk k T Ak:m,k:n = Ak:m,k:n − 2vk (vk Ak:m,k:n)

Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A

Matrix Q is not formed explicitly but reflection vector vk is stored

Question: Can A be reused to store both R and vk completely?

Xiangmin Jiao Numerical Analysis I 7 / 14 Question: What happens if vk is 0 in line 3 of the loop?

Householder Algorithm

Householder QR Factorization for k = 1 to n x = Ak:m,k vk = sign(x1)kxke1 + x vk = vk /kvk k T Ak:m,k:n = Ak:m,k:n − 2vk (vk Ak:m,k:n)

Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A

Matrix Q is not formed explicitly but reflection vector vk is stored

Question: Can A be reused to store both R and vk completely? Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed.

Xiangmin Jiao Numerical Analysis I 7 / 14 Householder Algorithm

Householder QR Factorization for k = 1 to n x = Ak:m,k vk = sign(x1)kxke1 + x vk = vk /kvk k T Ak:m,k:n = Ak:m,k:n − 2vk (vk Ak:m,k:n)

Note that sign(x) = 1 if x ≥ 0 and = −1 if x < 0 Leave R in place of A

Matrix Q is not formed explicitly but reflection vector vk is stored

Question: Can A be reused to store both R and vk completely? Answer: We can use lower diagonal portion of A to store all but one entry in each vk . So an additional array of size n is needed.

Question: What happens if vk is 0 in line 3 of the loop?

Xiangmin Jiao Numerical Analysis I 7 / 14 Compute Qx = Q1Q2 ··· Qnx

Implicit calculation of Qx for k = n downto 1 T xk:m = xk:m − 2vk (vk xk:m)

Question: How to form Q and Qˆ , respectively?

Answer: Apply x = Im×m or first n columns of I , respectively

Applying or Forming Q

T Compute Q b = Qn ··· Q1b

Implicit calculation of QT b for k = 1 to n T bk:m = bk:m − 2vk (vk bk:m)

Xiangmin Jiao Numerical Analysis I 8 / 14 Answer: Apply x = Im×m or first n columns of I , respectively

Applying or Forming Q

T Compute Q b = Qn ··· Q1b

Implicit calculation of QT b for k = 1 to n T bk:m = bk:m − 2vk (vk bk:m)

Compute Qx = Q1Q2 ··· Qnx

Implicit calculation of Qx for k = n downto 1 T xk:m = xk:m − 2vk (vk xk:m)

Question: How to form Q and Qˆ , respectively?

Xiangmin Jiao Numerical Analysis I 8 / 14 Applying or Forming Q

T Compute Q b = Qn ··· Q1b

Implicit calculation of QT b for k = 1 to n T bk:m = bk:m − 2vk (vk bk:m)

Compute Qx = Q1Q2 ··· Qnx

Implicit calculation of Qx for k = n downto 1 T xk:m = xk:m − 2vk (vk xk:m)

Question: How to form Q and Qˆ , respectively?

Answer: Apply x = Im×m or first n columns of I , respectively

Xiangmin Jiao Numerical Analysis I 8 / 14 Operation Count

T Most work done at step Ak:m,k:n = Ak:m,k:n − 2vk (vk Ak:m,k:n) Flops per iteration: T I ∼ 2(m − k)(n − k) for dot products vk Ak:m,k:n I ∼ (m − k)(n − k) for outer product 2vk (··· ) I ∼ (m − k)(n − k) for subtraction I ∼ 4(m − k)(n − k) total Including outer loop, total flops is

n n X X 4(m − k)(n − k) = 4 (mn − km − kn + k2) k=1 k=1 ∼ 4mn2 − 4(m + n)n2/2 + 4n3/3 2 = 2mn2 − n3 3 If m ≈ n, it is more efficient than Gram-Schmidt method, but if m  n, similar to Gram-Schmidt Xiangmin Jiao Numerical Analysis I 9 / 14 Outline

1 Householder Reflectors

2 Givens Rotations and Updating QR Factorization

Xiangmin Jiao Numerical Analysis I 10 / 14 Givens Rotations

Instead of using reflection, we can rotate x to obtain kxke1  cos θ − sin θ  A Given R = rotates x ∈ 2 sin θ cos θ R counterclockwise by θ T T Choose θ to be angle between (xi , xj ) and (1, 0) , and we have " q #  cos θ − sin θ   x  x2 + x2 i = i j sin θ cos θ xj 0

where x −x cos θ = i , sin θ = j q 2 2 q 2 2 xi + xj xi + xj

Xiangmin Jiao Numerical Analysis I 11 / 14 Givens QR

Introduce zeros in column bottom-up, one zero at a time

 × × ×   × × ×   × × ×   × × ×   × × ×   × × ×    (4,5)   (3,4)    × × ×  →  × × ×  →  x x x  ...        × × ×   x x x   0 x x  × × × 0 x x × ×

To zero aij , left-multiply matrix F with Fi:i+1,i:i+1 being and Fkk = 1 for k 6= i, i + 1 Flop count of Givens QR is 3mn2 − n3, which is about 50% more expensive than Householder QR

Xiangmin Jiao Numerical Analysis I 12 / 14 Adding a Row

m×n Suppose A ∈ R with m ≥ n, and A has full rank   A1   A1 Let A˜ =  zT , where A = and zT is a new row inserted A2 A2 Obtain A˜ = Q˜ R˜ from A = QR efficiently using Givens rotation:     A1 Q1 I Suppose A = = R. A2 Q2     A1 0 Q1  T  T T z I Then A˜ = z = 1 0     R A2 0 Q2  T  T T T z I Perform series of Givens rotation R˜ = U ... U U , and n 2 1 R   0 Q1 ˜ T then Q =  1 0  U1U2 ... Un 0 Q2 2 I Updating R˜ costs 3n flops, and updating Q˜ costs 6mn flops

Xiangmin Jiao Numerical Analysis I 13 / 14 Adding a Column

m×n Suppose A ∈ R with m ≥ n, and A has full rank     Let A˜ = A1 z A2 , where A = A1 A2 and z is new column Obtain A˜ = Q˜ R˜ from A = QR efficiently using Givens rotation:     I Suppose A = A1 A2 = Q R1 R2     T I Then A˜ = A1 z A2 = Q R1 w R2 , where w = Q z   I Perform series of Givens rotation R˜ = Uk+1 ··· Un R1 w R2 , where Un performs on rows n and n − 1, Un−1 performs on rows n − 1 and n − 2, etc. ˜ T T I Q = QUn ··· Uk+1 I It takes O(mn) time overall

Xiangmin Jiao Numerical Analysis I 14 / 14