QR Decomposition on Gpus
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
The Multishift Qr Algorithm. Part I: Maintaining Well-Focused Shifts and Level 3 Performance∗
SIAM J. MATRIX ANAL. APPL. c 2002 Society for Industrial and Applied Mathematics Vol. 23, No. 4, pp. 929–947 THE MULTISHIFT QR ALGORITHM. PART I: MAINTAINING WELL-FOCUSED SHIFTS AND LEVEL 3 PERFORMANCE∗ KAREN BRAMAN† , RALPH BYERS† , AND ROY MATHIAS‡ Abstract. This paper presents a small-bulge multishift variation of the multishift QR algorithm that avoids the phenomenon ofshiftblurring, which retards convergence and limits the number of simultaneous shifts. It replaces the large diagonal bulge in the multishift QR sweep with a chain of many small bulges. The small-bulge multishift QR sweep admits nearly any number ofsimultaneous shifts—even hundreds—without adverse effects on the convergence rate. With enough simultaneous shifts, the small-bulge multishift QR algorithm takes advantage ofthe level 3 BLAS, which is a special advantage for computers with advanced architectures. Key words. QR algorithm, implicit shifts, level 3 BLAS, eigenvalues, eigenvectors AMS subject classifications. 65F15, 15A18 PII. S0895479801384573 1. Introduction. This paper presents a small-bulge multishift variation of the multishift QR algorithm [4] that avoids the phenomenon of shift blurring, which re- tards convergence and limits the number of simultaneous shifts that can be used effec- tively. The small-bulge multishift QR algorithm replaces the large diagonal bulge in the multishift QR sweep with a chain of many small bulges. The small-bulge multishift QR sweep admits nearly any number of simultaneous shifts—even hundreds—without adverse effects on the convergence rate. It takes advantage of the level 3 BLAS by organizing nearly all the arithmetic workinto matrix-matrix multiplies. This is par- ticularly efficient on most modern computers and especially efficient on computers with advanced architectures. -
The SVD Algorithm
Jim Lambers CME 335 Spring Quarter 2010-11 Lecture 6 Notes The SVD Algorithm Let A be an m × n matrix. The Singular Value Decomposition (SVD) of A, A = UΣV T ; where U is m × m and orthogonal, V is n × n and orthogonal, and Σ is an m × n diagonal matrix with nonnegative diagonal entries σ1 ≥ σ2 ≥ · · · ≥ σp; p = minfm; ng; known as the singular values of A, is an extremely useful decomposition that yields much informa- tion about A, including its range, null space, rank, and 2-norm condition number. We now discuss a practical algorithm for computing the SVD of A, due to Golub and Kahan. Let U and V have column partitions U = u1 ··· um ;V = v1 ··· vn : From the relations T Avj = σjuj;A uj = σjvj; j = 1; : : : ; p; it follows that T 2 A Avj = σj vj: That is, the squares of the singular values are the eigenvalues of AT A, which is a symmetric matrix. It follows that one approach to computing the SVD of A is to apply the symmetric QR algorithm T T T T to A A to obtain a decomposition A A = V Σ ΣV . Then, the relations Avj = σjuj, j = 1; : : : ; p, can be used in conjunction with the QR factorization with column pivoting to obtain U. However, this approach is not the most practical, because of the expense and loss of information incurred from computing AT A. Instead, we can implicitly apply the symmetric QR algorithm to AT A. As the first step of the symmetric QR algorithm is to use Householder reflections to reduce the matrix to tridiagonal form, we can use Householder reflections to instead reduce A to upper bidiagonal form 2 3 d1 f1 6 d2 f2 7 6 7 T 6 . -
QR Decomposition: History and Its Applications
Mathematics & Statistics Auburn University, Alabama, USA QR History Dec 17, 2010 Asymptotic result QR iteration QR decomposition: History and its EE Applications Home Page Title Page Tin-Yau Tam èèèUUUÎÎÎ JJ II J I Page 1 of 37 Æâ§w Go Back fÆêÆÆÆ Full Screen Close email: [email protected] Website: www.auburn.edu/∼tamtiny Quit 1. QR decomposition Recall the QR decomposition of A ∈ GLn(C): QR A = QR History Asymptotic result QR iteration where Q ∈ GLn(C) is unitary and R ∈ GLn(C) is upper ∆ with positive EE diagonal entries. Such decomposition is unique. Set Home Page a(A) := diag (r11, . , rnn) Title Page where A is written in column form JJ II J I A = (a1| · · · |an) Page 2 of 37 Go Back Geometric interpretation of a(A): Full Screen rii is the distance (w.r.t. 2-norm) between ai and span {a1, . , ai−1}, Close i = 2, . , n. Quit Example: 12 −51 4 6/7 −69/175 −58/175 14 21 −14 6 167 −68 = 3/7 158/175 6/175 0 175 −70 . QR −4 24 −41 −2/7 6/35 −33/35 0 0 35 History Asymptotic result QR iteration EE • QR decomposition is the matrix version of the Gram-Schmidt orthonor- Home Page malization process. Title Page JJ II • QR decomposition can be extended to rectangular matrices, i.e., if A ∈ J I m×n with m ≥ n (tall matrix) and full rank, then C Page 3 of 37 A = QR Go Back Full Screen where Q ∈ Cm×n has orthonormal columns and R ∈ Cn×n is upper ∆ Close with positive “diagonal” entries. -
Lecture 4: Applications of Orthogonality: QR Decompositions Week 4 UCSB 2014
Math 108B Professor: Padraic Bartlett Lecture 4: Applications of Orthogonality: QR Decompositions Week 4 UCSB 2014 In our last class, we described the following method for creating orthonormal bases, known as the Gram-Schmidt method: ~ ~ Theorem. Suppose that V is a k-dimensional space with a basis B = fb1;::: bkg. The following process (called the Gram-Schmidt process) creates an orthonormal basis for V : 1. First, create the following vectors f ~v1; : : : ~vkg: • ~u1 = b~1: • ~u2 = b~2 − proj(b~2 onto ~u1). • ~u3 = b~3 − proj(b~3 onto ~u1) − proj(b~3 onto ~u2). • ~u4 = b~4 − proj(b~4 onto ~u1) − proj(b~4 onto ~u2) − proj(b~4 onto ~u3). ~ ~ ~ • ~uk = bk − proj(bk onto ~u1) − ::: − proj(bk onto k~u −1). 2. Now, take each of the vectors ~ui, and rescale them so that they are unit length: i.e. ~ui redefine each ~ui as the rescaled vector . jj ~uijj In this class, I want to talk about a useful application of the Gram-Schmidt method: the QR decomposition! We define this here: 1 The QR decomposition Definition. We say that an n×n matrix Q is orthogonal if its columns form an orthonor- n mal basis for R . As a side note: we've studied these matrices before! In class on Friday, we proved that for any such matrix, the relation QT · Q = I held; to see this, we just looked at the (i; j)-th entry of the product QT · Q, which by definition was the i-th row of QT dotted with the j-th column of A. -
Numerical Linear Algebra Revised February 15, 2010 4.1 the LU
Numerical Linear Algebra Revised February 15, 2010 4.1 The LU Decomposition The Elementary Matrices and badgauss In the previous chapters we talked a bit about the solving systems of the form Lx = b and Ux = b where L is lower triangular and U is upper triangular. In the exercises for Chapter 2 you were asked to write a program x=lusolver(L,U,b) which solves LUx = b using forsub and backsub. We now address the problem of representing a matrix A as a product of a lower triangular L and an upper triangular U: Recall our old friend badgauss. function B=badgauss(A) m=size(A,1); B=A; for i=1:m-1 for j=i+1:m a=-B(j,i)/B(i,i); B(j,:)=a*B(i,:)+B(j,:); end end The heart of badgauss is the elementary row operation of type 3: B(j,:)=a*B(i,:)+B(j,:); where a=-B(j,i)/B(i,i); Note also that the index j is greater than i since the loop is for j=i+1:m As we know from linear algebra, an elementary opreration of type 3 can be viewed as matrix multipli- cation EA where E is an elementary matrix of type 3. E looks just like the identity matrix, except that E(j; i) = a where j; i and a are as in the MATLAB code above. In particular, E is a lower triangular matrix, moreover the entries on the diagonal are 1's. We call such a matrix unit lower triangular. -
Communication-Optimal Parallel and Sequential QR and LU Factorizations
Communication-optimal parallel and sequential QR and LU factorizations James Demmel Laura Grigori Mark Frederick Hoemmen Julien Langou Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No. UCB/EECS-2008-89 http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-89.html August 4, 2008 Copyright © 2008, by the author(s). All rights reserved. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission. Communication-optimal parallel and sequential QR and LU factorizations James Demmel, Laura Grigori, Mark Hoemmen, and Julien Langou August 4, 2008 Abstract We present parallel and sequential dense QR factorization algorithms that are both optimal (up to polylogarithmic factors) in the amount of communication they perform, and just as stable as Householder QR. Our first algorithm, Tall Skinny QR (TSQR), factors m × n matrices in a one-dimensional (1-D) block cyclic row layout, and is optimized for m n. Our second algorithm, CAQR (Communication-Avoiding QR), factors general rectangular matrices distributed in a two-dimensional block cyclic layout. It invokes TSQR for each block column factorization. The new algorithms are superior in both theory and practice. We have extended known lower bounds on communication for sequential and parallel matrix multiplication to provide latency lower bounds, and show these bounds apply to the LU and QR decompositions. -
Massively Parallel Poisson and QR Factorization Solvers
Computers Math. Applic. Vol. 31, No. 4/5, pp. 19-26, 1996 Pergamon Copyright~)1996 Elsevier Science Ltd Printed in Great Britain. All rights reserved 0898-1221/96 $15.00 + 0.00 0898-122 ! (95)00212-X Massively Parallel Poisson and QR Factorization Solvers M. LUCK£ Institute for Control Theory and Robotics, Slovak Academy of Sciences DdbravskA cesta 9, 842 37 Bratislava, Slovak Republik utrrluck@savba, sk M. VAJTERSIC Institute of Informatics, Slovak Academy of Sciences DdbravskA cesta 9, 840 00 Bratislava, P.O. Box 56, Slovak Republic kaifmava©savba, sk E. VIKTORINOVA Institute for Control Theoryand Robotics, Slovak Academy of Sciences DdbravskA cesta 9, 842 37 Bratislava, Slovak Republik utrrevka@savba, sk Abstract--The paper brings a massively parallel Poisson solver for rectangle domain and parallel algorithms for computation of QR factorization of a dense matrix A by means of Householder re- flections and Givens rotations. The computer model under consideration is a SIMD mesh-connected toroidal n x n processor array. The Dirichlet problem is replaced by its finite-difference analog on an M x N (M + 1, N are powers of two) grid. The algorithm is composed of parallel fast sine transform and cyclic odd-even reduction blocks and runs in a fully parallel fashion. Its computational complexity is O(MN log L/n2), where L = max(M + 1, N). A parallel proposal of QI~ factorization by the Householder method zeros all subdiagonal elements in each column and updates all elements of the given submatrix in parallel. For the second method with Givens rotations, the parallel scheme of the Sameh and Kuck was chosen where the disjoint rotations can be computed simultaneously. -
A Fast and Accurate Matrix Completion Method Based on QR Decomposition and L 2,1-Norm Minimization Qing Liu, Franck Davoine, Jian Yang, Ying Cui, Jin Zhong, Fei Han
A Fast and Accurate Matrix Completion Method based on QR Decomposition and L 2,1-Norm Minimization Qing Liu, Franck Davoine, Jian Yang, Ying Cui, Jin Zhong, Fei Han To cite this version: Qing Liu, Franck Davoine, Jian Yang, Ying Cui, Jin Zhong, et al.. A Fast and Accu- rate Matrix Completion Method based on QR Decomposition and L 2,1-Norm Minimization. IEEE Transactions on Neural Networks and Learning Systems, IEEE, 2019, 30 (3), pp.803-817. 10.1109/TNNLS.2018.2851957. hal-01927616 HAL Id: hal-01927616 https://hal.archives-ouvertes.fr/hal-01927616 Submitted on 20 Nov 2018 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Page 1 of 20 1 1 2 3 A Fast and Accurate Matrix Completion Method 4 5 based on QR Decomposition and L -Norm 6 2,1 7 8 Minimization 9 Qing Liu, Franck Davoine, Jian Yang, Member, IEEE, Ying Cui, Zhong Jin, and Fei Han 10 11 12 13 14 Abstract—Low-rank matrix completion aims to recover ma- I. INTRODUCTION 15 trices with missing entries and has attracted considerable at- tention from machine learning researchers. -
Notes on Orthogonal Projections, Operators, QR Factorizations, Etc. These Notes Are Largely Based on Parts of Trefethen and Bau’S Numerical Linear Algebra
Notes on orthogonal projections, operators, QR factorizations, etc. These notes are largely based on parts of Trefethen and Bau’s Numerical Linear Algebra. Please notify me of any typos as soon as possible. 1. Basic lemmas and definitions Throughout these notes, V is a n-dimensional vector space over C with a fixed Her- mitian product. Unless otherwise stated, it is assumed that bases are chosen so that the Hermitian product is given by the conjugate transpose, i.e. that (x, y) = y∗x where y∗ is the conjugate transpose of y. Recall that a projection is a linear operator P : V → V such that P 2 = P . Its complementary projection is I − P . In class we proved the elementary result that: Lemma 1. Range(P ) = Ker(I − P ), and Range(I − P ) = Ker(P ). Another fact worth noting is that Range(P ) ∩ Ker(P ) = {0} so the vector space V = Range(P ) ⊕ Ker(P ). Definition 1. A projection is orthogonal if Range(P ) is orthogonal to Ker(P ). Since it is convenient to use orthonormal bases, but adapt them to different situations, we often need to change basis in an orthogonality-preserving way. I.e. we need the following type of operator: Definition 2. A nonsingular matrix Q is called an unitary matrix if (x, y) = (Qx, Qy) for every x, y ∈ V . If Q is real, it is called an orthogonal matrix. An orthogonal matrix Q can be thought of as an orthogonal change of basis matrix, in which each column is a new basis vector. Lemma 2. -
GRAM SCHMIDT and QR FACTORIZATION Math 21B, O. Knill
GRAM SCHMIDT AND QR FACTORIZATION Math 21b, O. Knill GRAM-SCHMIDT PROCESS. Let ~v1; : : : ;~vn be a basis in V . Let ~w1 = ~v1 and ~u1 = ~w1=jj~w1jj. The Gram-Schmidt process recursively constructs from the already constructed orthonormal set ~u1; : : : ; ~ui−1 which spans a linear space Vi−1 the new vector ~wi = (~vi − projVi−1 (~vi)) which is orthogonal to Vi−1, and then normalizes ~wi to get ~ui = ~wi=jj~wijj. Each vector ~wi is orthonormal to the linear space Vi−1. The vectors f~u1; : : : ; ~un g form then an orthonormal basis in V . EXAMPLE. 2 2 3 2 1 3 2 1 3 Find an orthonormal basis for ~v1 = 4 0 5, ~v2 = 4 3 5 and ~v3 = 4 2 5. 0 0 5 SOLUTION. 2 1 3 1. ~u1 = ~v1=jj~v1jj = 4 0 5. 0 2 0 3 2 0 3 3 1 2. ~w2 = (~v2 − projV1 (~v2)) = ~v2 − (~u1 · ~v2)~u1 = 4 5. ~u2 = ~w2=jj~w2jj = 4 5. 0 0 2 0 3 2 0 3 0 0 3. ~w3 = (~v3 − projV2 (~v3)) = ~v3 − (~u1 · ~v3)~u1 − (~u2 · ~v3)~u2 = 4 5, ~u3 = ~w3=jj~w3jj = 4 5. 5 1 QR FACTORIZATION. The formulas can be written as ~v1 = jj~v1jj~u1 = r11~u1 ··· ~vi = (~u1 · ~vi)~u1 + ··· + (~ui−1 · ~vi)~ui−1 + jj~wijj~ui = r1i~u1 + ··· + rii~ui ··· ~vn = (~u1 · ~vn)~u1 + ··· + (~un−1 · ~vn)~un−1 + jj~wnjj~un = r1n~u1 + ··· + rnn~un which means in matrix form 2 3 2 3 2 3 j j · j j j · j r11 r12 · r1n A = 4 ~v1 ···· ~vn 5 = 4 ~u1 ···· ~un 5 4 0 r22 · r2n 5 = QR ; j j · j j j · j 0 0 · rnn where A and Q are m × n matrices and R is a n × n matrix which has rij = ~vj · ~ui, for i < j and vii = j~vij THE GRAM-SCHMIDT PROCESS PROVES: Any matrix A with linearly independent columns ~vi can be decomposed as A = QR, where Q has orthonormal column vectors and where R is an upper triangular square matrix with the same number of columns than A. -
How Unique Is QR? Full Rank, M = N
LECTUREAPPENDIX · purdue university cs 51500 Kyle Kloster matrix computations October 12, 2014 How unique is QR? Full rank, m = n In class we looked at the special case of full rank, n × n matrices, and showed that the QR decomposition is unique up to a factor of a diagonal matrix with entries ±1. Here we'll see that the other full rank cases follow the m = n case somewhat closely. Any full rank QR decomposition involves a square, upper- triangular partition R within the larger (possibly rectangular) m × n matrix. The gist of these uniqueness theorems is that R is unique, up to multiplication by a diagonal matrix of ±1s; the extent to which the orthogonal matrix is unique depends on its dimensions. Theorem (m = n) If A = Q1R1 = Q2R2 are two QR decompositions of full rank, square A, then Q2 = Q1S R2 = SR1 for some square diagonal S with entries ±1. If we require the diagonal entries of R to be positive, then the decomposition is unique. Theorem (m < n) If A = Q1 R1 N 1 = Q2 R2 N 2 are two QR decom- positions of a full rank, m × n matrix A with m < n, then Q2 = Q1S; R2 = SR1; and N 2 = SN 1 for square diagonal S with entries ±1. If we require the diagonal entries of R to be positive, then the decomposition is unique. R R Theorem (m > n) If A = Q U 1 = Q U 2 are two QR 1 1 0 2 2 0 decompositions of a full rank, m × n matrix A with m > n, then Q2 = Q1S; R2 = SR1; and U 2 = U 1T for square diagonal S with entries ±1, and square orthogonal T . -
QR Factorization for the CELL Processor 1 Introduction
QR Factorization for the CELL Processor { LAPACK Working Note 201 Jakub Kurzak Department of Electrical Engineering and Computer Science, University of Tennessee Jack Dongarra Department of Electrical Engineering and Computer Science, University of Tennessee Computer Science and Mathematics Division, Oak Ridge National Laboratory School of Mathematics & School of Computer Science, University of Manchester ABSTRACT ScaLAPACK [4] are good examples. These imple- The QR factorization is one of the most mentations work on square or rectangular submatri- important operations in dense linear alge- ces in their inner loops, where operations are encap- bra, offering a numerically stable method for sulated in calls to Basic Linear Algebra Subroutines solving linear systems of equations includ- (BLAS) [5], with emphasis on expressing the compu- ing overdetermined and underdetermined sys- tation as level 3 BLAS (matrix-matrix type) opera- tems. Classic implementation of the QR fac- tions. torization suffers from performance limita- The fork-and-join parallelization model of these li- tions due to the use of matrix-vector type op- braries has been identified as the main obstacle for erations in the phase of panel factorization. achieving scalable performance on new processor ar- These limitations can be remedied by using chitectures. The arrival of multi-core chips increased the idea of updating of QR factorization, ren- the demand for new algorithms, exposing much more dering an algorithm, which is much more scal- thread-level parallelism of much finer granularity. able and much more suitable for implementa- This paper presents an implementation of the QR tion on a multi-core processor. It is demon- factorization based on the idea of updating the QR strated how the potential of the CELL proces- factorization.