High-Performance Up-and-Downdating via Householder-like Transformations Robert A. van de Geijn Field G. Van Zee Department of Computer Science The University of Texas at Austin Austin, TX 78712 rvdg,[email protected] Draft February 1, 2010 Abstract We present high-performance algorithms for up-and-downdating a Cholesky factor or QR factor- ization. The method uses Householder-like transformations, sometimes called hyperbolic Householder transformations, that are accumulated so that most computation can be cast in terms of high-performance matrix-matrix operations. The resulting algorithms can then be used as building blocks for an algorithm- by-blocks that allows computation to be conveniently scheduled to multithreaded architectures like mul- ticore processors. Performance is shown to be similar to that achieved by a blocked QR factorization via Householder transformations. 1 Introduction Consider the Linear Least-Squares problem that, given a matrix A 2 Cm×n with linearly independent m n columns and y 2 C , computes x 2 C that minimizes kAx − yk2. This problem is typically solved via one of two methods: Method of Normal Equations: Solve AH Ax = AH y by computing the Cholesky factor of AH A, upper triangular matrix R, followed by forward and backward substitution to solve RH Rx = AH y. QR factorization (via Householder transformations): Compute the QR factorization A = QR where Q is an orthogonal m × n matix and R is an upper triangular n × n matrix. Solve Rx = QH y. In this paper, we concern ourselves with the following prototypical scenario: Let rows of the appended system A y represent observations that have been taken, for example, over time. These observations 0 B b 1 A y B C c C can be partitioned into three groups: = @ A where the Cholesky factor corresponding to D d ! B has already been computed: BH B + DH D = RH R. Now, the rows of D represent old data that we D would like to remove while the rows of C represent new data that we would like to add to the Linear Least- ! B Squares problem. Thus, we would like to compute the Cholesky factor corresponding to leveraging C the already computed R. The right-hand side has to be updated correspondingly, which is discussed in Section 6. 1 In [14] hyperbolic Householder transformations are reviewed for this problem and analyzed both from an algorithmic and numerical stability point of view. In that paper, references to the literature can also be found. The present paper builds on the insights in that paper and combines it with insights from other papers [1, 13, 7, 16, 8, 17, 15] that focus on aggregrating multiple Householder-like transformations into a block transformation. The contribution of the present paper is a practical high-performance algorithm for up- and/or downdating that can be implemented as a library routine using the level-3 BLAS [5]. The remainder of the paper is structured as follows. In Section 2 we discuss a family of Householder-like transformations and how to accumulate them into a block transformation. Updating and downdating are discussed separately in Sections 3 and 4, respectively, and then combined in Section 5 in which a blocked algorithm is also given. How to use an up- and/or downdated system to solve the new Linear Least-Squares problem is discussed in Section 6. A brief overview of the algorithm-by-blocks concept is given in Section 7. Performance is reported in Section 8 and concluding remarks can be found in the final section. 2 A Family of Householder Transformations In the following discussion, we will let Σ 2 Rn×n with Σ = diag(1; ±1; ··· ; ±1) so that ΣΣ = I. Such a matrix is referred to as a signature matrix. We make the choice that the first diagonal element equals to one so as to simplify our discussion. Then, by design, Σe0 = e0, where e0 is the first column of the identity matrix. n H 1 H 1 H H Theorem 1 Let w 2 C and τ = w Σw=2 6= 0. Then (I − τ Σww )Σ(I − τ Σww ) = Σ. Proof: Let w 2 Cn and τ = wH Σw=2 6= 0. Then 1 1 1 2 (I − ΣwwH )Σ(I − ΣwwH )H = Σ − 2 ΣwwH Σ + ΣwwH Σ = Σ: τ τ τ τ endofproof When Σ = I the (I − 1 ΣwwH ) in the above theorem is the traditional Householder transformation or τ ! I 0 reflector. If Σ = it is referred to as a hyperbolic Householder Transformation. 0 −I p n T 2 H ¯ Theorem 2 Let x 2 C , χ0 = e0 x be its first element, jλj = x Σx chosen so that λχ0(=χ ¯0λ) is real, wH Σw 1 H H w = x + λe0, and τ = 2 6= 0. Then (I − τ Σww ) x = −λe0. Proof: Under the assumptions of the theorem 2wH Σx 2(x + λe )H Σx 2(xH Σx + λχ¯ ) = 0 = 0 = 1 H H H ¯ 2 w Σw (x + λe0) Σ(x + λe0) x Σx + 2λχ0 + jλj and 1 1 2wH Σx (I − ΣwwH )H x = x − wwH Σx = x − (x + λe ) = −λe : τ τ wH Σw 0 0 endofproof p H Corollary 3 Under the assumptions of Thm. 2, if χ0 is real, then λ = ± x Σx. The Cholesky factor R that is being updated and/or downdated often has real diagonal elements, and vector w can be normalized by dividing by a nonzero scalar, in which case the following steps provide a robust way of computing w and λ so that w has a unit first element: p H • λ := sign(χ0) x Σx. w := x + λe0. (Note: the choice of the sign means that λ and χ0 have the same sign, thus avoiding catastrophic cancellation that can lead to unnecessary numerical inaccuracy). T • If !0 equals zero, then w = e0 else w := w=!0. (Here !0 = e0 w equals the first element of w. This step normalizes w so that it has a unit first element.) 2 Definition 4 Let x 2 Cn be such that xH Σx 6= 0. We define the function [~χ0; w1; τ] := GeneralHouse (Σ; χ0; x1) 1 H H χ0 1 wH Σw so that I − τ Σww x =χ ~0e0, where x = , w = , and τ = 2 . x1 w1 n×k Theorem 5 Let the matrix Wk−1 2 C have linearly independent columns. Partition Wk−1 by columns as Wk−1 = w0 w1 ··· wk−1 and let τi 6= 0, 0 ≤ i < k. Then for 0 ≤ j < k there exists a j × j nonsingular upper triangular matrix Tj such that 1 H 1 H 1 H −1 H I − Σw0w0 I − Σw1w1 ··· I − Σwj−1wj−1 = I − ΣWj−1Tj−1Wj−1 τ0 τ1 τj−1 H Tj−1 Wj−1Σwj The matrices Tj is given by the recurrence T0 = τ0 and Tj = for 1 ≤ j < k. 0 τj Proof: Proof by induction on j. Base case. j = 0: Trivially true. Inductive step. Induction Hypothesis (I.H.): Assume that 1 H 1 H 1 H −1 H I − Σw0w0 ··· I − Σw1w1 I − Σwj−1wj−1 = I − ΣWj−1Tj−1Wj−1 τ0 τ1 τj−1 1 H 1 H 1 H −1 H We need to show that I − Σw0w ··· I − Σwj−1w I − Σwjw = I − ΣWjT W : τ0 0 τj−1 j−1 τj j j j H ! 1 H 1 H −1 H Σwjwj I − Σw0w0 ··· I − Σwjwj = I − ΣWj−1Tj−1Wj−1 I − τ0 τj τj −1 −1 H Tj−1 −Tj−1Wj−1Σwj/τj H = I − Σ Wj−1 wj Wj−1 wj 0 1/τj H −1 Tj−1 Wj−1Σwj H −1 H = I − Σ Wj−1 wj Wj−1 wj = I − ΣWjTj Wj : 0 τj By the Principle of Mathematical Induction the desired result holds. endofproof Theorem 6 Let W 2 Cn×k be a matrix with linearly independent columns such that W H ΣW has nonzero di- agonal elements. Then there exists a unique nonsingular upper triangular matrix with real diagonal elements T 2 Ck×k such that (I −ΣWT −1W H )Σ(I −ΣWT −1W H )H = Σ. This matrix T satisfies T +T H = W H ΣW H 1 H so that T = striu(W ΣW ) + 2 diag(W ΣW ). Proof: Theorem 5 provides a proof of existence. (The wi and τi's in that theorem equal the columns of W and diagonal elements of W T ΣW , respectively.) Now, Σ = (I−ΣWT −1W H )Σ(I−ΣWT −1W H )H = Σ−ΣWT −1W H Σ−ΣWT −H W H Σ+ΣWT −1W H ΣWT −H W H Σ so that 0 = Σ(WT −1W H + WT −H W H − WT −1W H ΣWT −H W H )Σ: Thus, 0 = W H Σ0ΣW = W H ΣΣ(WT −1W H + WT −H W H − WT −1W H ΣWT −H W H )ΣΣW or, equivalently, (since W H ΣΣW = W H W is nonsingular) 0 = T −1 + T −H − T −1W H ΣWT −H from which we conclude that T H + T = W H ΣW . Now, if T is upper triangular and has real valued diagonal elements, then T = striu(W H ΣW ) + 1 H 2 diag(W ΣW ). endofproof 3 3 Updating ! B Let us consider A = with AH A = BH B + CH C = R~H R~, where R~ is the Cholesky factor of AH A. C Here we will assume that both A and B have linearly independent columns. The question is whether, if we know that the Cholesky factor of BH B is R, we can cheaply compute the Cholesky factor of AH A.