DOCSLIB.ORG

High-Performance SVD for Big Data

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
High-Performance SVD for Big Data High-Performance SVD for big data Andreas Stathopoulos Eloy Romero Alcalde Computer Science Department, College of William & Mary, USA Bigdata'18 High-Performance SVD for big data Computer Science Department College of William & Mary 1/50 The SVD problem Given a matrix A with dimensions m × n find k singular triplets (σi ; ui ; vi ) such that Avi ≈ σi ui and fui g; fvi g are orthonormal bases. Applications: applied math (norm computation, reduction of dimensionality or variance) combinatorial scientific computing, social network analysis computational statistics and Machine Learning (PCA, SVM, LSI) low dimensionality embeddings (ISOMAP, MDS) SVD updating of streaming data High-Performance SVD for big data Computer Science Department College of William & Mary 2/50 Notation for SVD σi , i-th singular value, Σ = diag([σ1 : : : σk ]), σ1 ≤ · · · ≤ σk ui , i-th left singular vector, U = [u1 ::: uk ] vi , i-th right singular vector, V = [v1 ::: vk ] Full decomposition, Partial decomposition, k = minfm; ng: k < minfm; ng: k minfm;ng T X T T X T A ≈ UΣV = ui σi vi A = UΣV = ui σi vi i=1 i=1 Given vi , then σi = kAvi k2, and ui = Avi /σi . T T Given ui , then σi = kA ui k2, and vi = A ui /σi . High-Performance SVD for big data Computer Science Department College of William & Mary 3/50 Relevant Aspects The performance of various methods depends on the next aspects: Matrix dimensions, m rows and n columns Sparsity Number of singular triplets sought If few singular triplets are sought, which ones Accuracy required Spectral distribution/gaps/decay High-Performance SVD for big data Computer Science Department College of William & Mary 4/50 Accuracy of singular values Accuracy of σi : distance between the exact singular value σi and an approximationσ ~i It can be bound as q l 2 r 2 jσi − σ~i j ≤ kri k2 + kri k2 kAT Av~ − σ~ v~ k ≤ i i i σi +σ ~i kAAT u~ − σ~ u~ k ≤ i i i ; σi +σ ~i l r T where ri = Av~i − σi u~i and ri = A u~i − σi v~i We say a method obtains accuracy kAk2f () if jσi − σ~i j . kAk2f () where is the maximum of the rounding error of the arithmetic used in the method and the matrix High-Performance SVD for big data Computer Science Department College of William & Mary 5/50 Accuracy of singular vectors Individual vectors: Angle between the exact singular vector vi and an approximation v~i q l 2 r 2 kri k + kri k Bound: sin \(v~i ; vi ) . minfjσi − σi−1j; jσi − σi+1jg Not useful when singular values are close As low rank approximations, A ≈ U~U~T A: T kA − U~U~ Ak2: the largest singular value not included in U~ T kA − U~U~ AkF : the sum of all singular values not included in U~ These norms do not converge to zero unless U~ captures the entire rank of A Accuracy means closeness to kA − UUT Ak or smaller than a certain fraction of kAk. q X 2 Note that kAk2 = max σi and kAkF = σi High-Performance SVD for big data Computer Science Department College of William & Mary 6/50 Methods High-Performance SVD for big data Computer Science Department College of William & Mary 7/50 Direct Methods: Bidiagonal reduction Aka svd() in Matlab/Octave and R, numpy.linalg.svd() in NumPy Based on reducing the input matrix into a bidiagonal matrix by similarity transformations Cost: spatial O(mn), temporal O(mn2) Limitations: Challenging parallelization in shared and distributed memory Densification. The nonzero structure of the input matrix is not exploited Cost does not reduce if few singular triplets needed or if matrix is low rank Best used when: Small matrix dimensions (m = n = 4000 takes ∼ 20 sec on a 2018 MacBook Pro) More than 10% of the spectrum is needed High-Performance SVD for big data Computer Science Department College of William & Mary 8/50 Direct Methods: Squaring the matrix T 1 Form M = A A T 2 Decompose M = V~Λ~V~ 1 − 1 3 Solution: V = V~, Σ = Λ~ 2 , U = AV~Λ~ 2 Cost: spatial O(n2 + mk + nk), temporal O(mn2), for k wanted singular triplets Improvement: Easy parallelization The nonzero structure can be exploited The spatial cost reduces with k and if left and/or right singular vectors are not required Limitations: kAk2 Accuracy is kσi − σ~i k2 ≤ , where is the rounding error for A σi Best used when: One of the dimensions is small, n . 4000 Approximate solutions are enough High-Performance SVD for big data Computer Science Department College of William & Mary 9/50 Direct Methods: QR 1 Decompose A = QR T 2 Decompose R = U~Σ~V~ 3 Solution: V = V~, Σ = Σ,~ U = QU~ Improvement: Easy parallelization using Tall Skinny QR (TSQR) Better accuracy than squaring the matrix using proper variant of TSQR Limitations: More expensive than the previous method for sparse matrices Best used when: Dense input matrix One of the dimensions is small, n . 4000 High-accurate solutions are required (TSQR variant may affect accuracy) High-Performance SVD for big data Computer Science Department College of William & Mary 10/50 Iterative Methods Warning: Risk of missing triplets. See perturbation analysis In practice, this is usually not an issue, probably because of the random nature of the methods They introduce options that have impact on time and quality of the computed solution Oversampling/restarting parameters Stopping criterion Critical kernels: Matrix-multi vector product, AX and AT X Orthogonalization High-Performance SVD for big data Computer Science Department College of William & Mary 11/50 Subspace Iteration (aka Randomized SVD) Basic method: 0 0 T V0 = orth(rand(n; k )), for k ≥ k, Vi = orth(A AVi−1) T Vi converges towards the k largest singular vectors of A A Detailed method that achieve full precision: 0 1 Set i = 0 and V0 = orth(rand(n; k )) 2 Set Ui = orth(AVi ) 3 Stop if Vi and Ui are accurate enough T 4 Set Vi+1 = orth(A Ui ) 5 Increment i and go to step 2 Stopping criterion: After some number of iterations Monitor the convergence of singular values or kA − U~U~T Ak T Monitor the residual minR kA Ui − Vi Rk2, where R is a k × k matrix High-Performance SVD for big data Computer Science Department College of William & Mary 12/50 Subspace Iteration (aka Randomized SVD) 0 1 Set i = 0 and V0 = orth(rand(n; k )) 2 Set Ui = orth(AVi ) 3 Stop if Vi and Ui are accurate enough T 4 Set Vi+1 = orth(A Ui ) 5 Increment i and go to step 2 Comments: Requires a high-performance orthogonalization kernel, ie., TSQR or SVQB Can take advantage of high-performance matrix-multivector product Can be accelerated by replacing A by p(A), where the polynomial p boosts the wanted part of the spectrum, eg, p(x) = x 3 Tuning of oversampling, k0 ≥ k, may be required High-Performance SVD for big data Computer Science Department College of William & Mary 12/50 RSVD Oversampling: Example Convergence of the 10 largest singular values with RSVD Matrix's spectrum 1 F 0 k 10 A ) T ~ U ~ U 10−2 − T UU 0:5 ( k0 = 10 k 10−4 k0 = 20 0 Error, k = 30 k0 = 40 10−6 0 5 10 15 0 20 40 60 Iterations v u 10 T ~ ~T uX 2 2 Note: k(UU − UU )AkF = t σi − σ~i i=1 High-Performance SVD for big data Computer Science Department College of William & Mary 13/50 Lanczos Bidiagonalization Basic method: T T 2 T i v0 = orth(rand(n; 1)), Vi = orth([v0; A A v0; (A A) v0;:::; (A A) v0]) Vi converges towards the largest and smallest singular vectors of A Comments: Robust implementations are challenging: several choices for orthogonalization and restarting Popular variant: IRLBA Suitable for finding the largest and the smallest singular values Usually converges in less MV than RSVD, but not necessarily faster in time High-Performance SVD for big data Computer Science Department College of William & Mary 14/50 RSVD vs LBD ~ ~T T Goal: satisfy kA − UU Ak2=F = kA − UU Ak2=F (1 + ) Gap Convergence Analysis: ! n σ −1 RSVD converges in O log k − 1 iterations [Musco & Musco,'15] σk+1 ! n σ −1=2 Block Lanczos converges in O log k − 1 iterations [Saad] σk+1 Gap-Free Convergence Analysis (with high probability): log n RSVD converges in O iterations [Rokhlin et al.'09, Halko et al.'11] log n Block Lanczos in O p iterations [Musco & Musco,'15] High-Performance SVD for big data Computer Science Department College of William & Mary 15/50 The interplay between rank of the random block and gaps If p is oversampling, q the # of RSVD iterations, then [Ming Gu, '16] ~ ~T 2 T 2 2 4q kA − UU Ak2=F − kA − UU Ak2=F σk+p+1 σk+p+1 2 ≤ k C F F σk Normalization choices explain the behavior of different norms: σk+p+1 T σk+p+1 F = kAk2 gives F = kA − UU Ak2 gives σ1 σk+1 σk+p+1 T σk+p+1 F = kAkF gives P F = kA − UU AkF gives P i σi i>k σi More importantly, the σk+p+1 factor can be much smaller than σk+p+1 F σk For low accuracy, sometimes it is faster to just increase p! High-Performance SVD for big data Computer Science Department College of William & Mary 16/50 RSVD vs LBD: Example of Matrix-Vector Cost Convergence of the 10 largest singular values with RSVD 101 RSVD k0 = 10 0 F RSVD k = 20 k 0 A RSVD k = 30 ) T 0 0 ~ 10 U RSVD k = 40 ~ U Unrestarted LBD − Restarted LBD 10,20 T 10−1 UU ( k −2 Error, 10 50 100 150 200 250 300 Number of matrix-vector products High-Performance SVD for big data Computer Science Department College of William & Mary 17/50 RSVD vs LBD: Example of Orthogonalization Cost (flops) Convergence of the 10 largest singular values with RSVD F 0 k 10 A ) T ~ U ~ U 10−2 − RSVD k0 = 10 T RSVD k0 = 20 UU ( 0 k RSVD k = 30 10−4 RSVD k0 = 40 Error, Unrestarted LBD Restarted LBD 10,20 10−6 100 101 102 103 104 Number of inner products in orthogonalization (proportional to flops) High-Performance SVD for big data Computer Science Department College of William & Mary 18/50 RSVD vs LBD: Example of Orthogonalization Cost (loads)
Load more

Recommended publications
  • Accelerating the LOBPCG Method on Gpus Using a Blocked Sparse Matrix Vector Product
    Accelerating the LOBPCG method on GPUs using a blocked Sparse Matrix Vector Product Hartwig Anzt and Stanimire Tomov and Jack Dongarra Innovative Computing Lab University of Tennessee Knoxville, USA Email: [email protected], [email protected], [email protected] Abstract— the computing power of today’s supercomputers, often accel- erated by coprocessors like graphics processing units (GPUs), This paper presents a heterogeneous CPU-GPU algorithm design and optimized implementation for an entire sparse iter- becomes challenging. ative eigensolver – the Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) – starting from low-level GPU While there exist numerous efforts to adapt iterative lin- data structures and kernels to the higher-level algorithmic choices ear solvers like Krylov subspace methods to coprocessor and overall heterogeneous design. Most notably, the eigensolver technology, sparse eigensolvers have so far remained out- leverages the high-performance of a new GPU kernel developed side the main focus. A possible explanation is that many for the simultaneous multiplication of a sparse matrix and a of those combine sparse and dense linear algebra routines, set of vectors (SpMM). This is a building block that serves which makes porting them to accelerators more difficult. Aside as a backbone for not only block-Krylov, but also for other from the power method, algorithms based on the Krylov methods relying on blocking for acceleration in general. The subspace idea are among the most commonly used general heterogeneous LOBPCG developed here reveals the potential of eigensolvers [1]. When targeting symmetric positive definite this type of eigensolver by highly optimizing all of its components, eigenvalue problems, the recently developed Locally Optimal and can be viewed as a benchmark for other SpMM-dependent applications.
    [Show full text]
  • ARM HPC Ecosystem
    ARM HPC Ecosystem Darren Cepulis HPC Segment Manager ARM Business Segment Group HPC Forum, Santa Fe, NM 19th April 2017 © ARM 2017 ARM Collaboration for Exascale Programs United States Japan ARM is currently a participant in Fujitsu and RIKEN announced that the two Department of Energy funded Post-K system targeted at Exascale will pre-Exascale projects: Data be based on ARMv8 with new Scalable Movement Dominates and Fast Vector Extensions. Forward 2. European Union China Through FP7 and Horizon 2020, James Lin, vice director for the Center ARM has been involved in several of HPC at Shanghai Jiao Tong University funded pre-Exascale projects claims China will build three pre- including the Mont Blanc program Exascale prototypes to select the which deployed one of the first architecture for their Exascale system. ARM prototype HPC systems. The three prototypes are based on AMD, SunWei TaihuLight, and ARMv8. 2 © ARM 2017 ARM HPC deployments starting in 2H2017 Tw o recent announcements about ARM in HPC in Europe: 3 © ARM 2017 Japan Exascale slides from Fujitsu at ISC’16 4 © ARM 2017 Foundational SW Ecosystem for HPC . Linux OS’s – RedHat, SUSE, CENTOS, UBUNTU,… Commercial . Compilers – ARM, GNU, LLVM,… . Libraries – ARM, OpenBLAS, BLIS, ATLAS, FFTW… . Parallelism – OpenMP, OpenMPI, MVAPICH2,… . Debugging – Allinea, RW Totalview, GDB,… Open-source . Analysis – ARM, Allinea, HPCToolkit, TAU,… . Job schedulers – LSF, PBS Pro, SLURM,… . Cluster mgmt – Bright, CMU, warewulf,… Predictable Baseline 5 © ARM 2017 – now on ARM Functional Supported packages / components Areas OpenHPC defines a baseline. It is a community effort to Base OS RHEL/CentOS 7.1, SLES 12 provide a common, verified set of open source packages for Administrative Conman, Ganglia, Lmod, LosF, ORCM, Nagios, pdsh, HPC deployments Tools prun ARM’s participation: Provisioning Warewulf Resource Mgmt.
    [Show full text]
  • 0 BLIS: a Modern Alternative to the BLAS
    0 BLIS: A Modern Alternative to the BLAS FIELD G. VAN ZEE and ROBERT A. VAN DE GEIJN, The University of Texas at Austin We propose the portable BLAS-like Interface Software (BLIS) framework which addresses a number of short- comings in both the original BLAS interface and present-day BLAS implementations. The framework allows developers to rapidly instantiate high-performance BLAS-like libraries on existing and new architectures with relatively little effort. The key to this achievement is the observation that virtually all computation within level-2 and level-3 BLAS operations may be expressed in terms of very simple kernels. Higher-level framework code is generalized so that it can be reused and/or re-parameterized for different operations (as well as different architectures) with little to no modification. Inserting high-performance kernels into the framework facilitates the immediate optimization of any and all BLAS-like operations which are cast in terms of these kernels, and thus the framework acts as a productivity multiplier. Users of BLAS-dependent applications are supported through a straightforward compatibility layer, though calling sequences must be updated for those who wish to access new functionality. Experimental performance of level-2 and level-3 operations is observed to be competitive with two mature open source libraries (OpenBLAS and ATLAS) as well as an established commercial product (Intel MKL). Categories and Subject Descriptors: G.4 [Mathematical Software]: Efficiency General Terms: Algorithms, Performance Additional Key Words and Phrases: linear algebra, libraries, high-performance, matrix, BLAS ACM Reference Format: ACM Trans. Math. Softw. 0, 0, Article 0 ( 0000), 31 pages.
    [Show full text]
  • 18 Restructuring the Tridiagonal and Bidiagonal QR Algorithms for Performance
    18 Restructuring the Tridiagonal and Bidiagonal QR Algorithms for Performance FIELD G. VAN ZEE and ROBERT A. VAN DE GEIJN, The University of Texas at Austin GREGORIO QUINTANA-ORT´I, Universitat Jaume I We show how both the tridiagonal and bidiagonal QR algorithms can be restructured so that they become rich in operations that can achieve near-peak performance on a modern processor. The key is a novel, cache-friendly algorithm for applying multiple sets of Givens rotations to the eigenvector/singular vec- tor matrix. This algorithm is then implemented with optimizations that: (1) leverage vector instruction units to increase floating-point throughput, and (2) fuse multiple rotations to decrease the total number of memory operations. We demonstrate the merits of these new QR algorithms for computing the Hermitian eigenvalue decomposition (EVD) and singular value decomposition (SVD) of dense matrices when all eigen- vectors/singular vectors are computed. The approach yields vastly improved performance relative to tradi- tional QR algorithms for these problems and is competitive with two commonly used alternatives—Cuppen’s Divide-and-Conquer algorithm and the method of Multiple Relatively Robust Representations—while inherit- ing the more modest O(n) workspace requirements of the original QR algorithms. Since the computations performed by the restructured algorithms remain essentially identical to those performed by the original methods, robust numerical properties are preserved. Categories and Subject Descriptors: G.4 [Mathematical Software]: Efficiency General Terms: Algorithms, Performance Additional Key Words and Phrases: Eigenvalues, singular values, tridiagonal, bidiagonal, EVD, SVD, QR algorithm, Givens rotations, linear algebra, libraries, high performance ACM Reference Format: Van Zee, F.
    [Show full text]
  • Solving Symmetric Semi-Definite (Ill-Conditioned)
    Solving Symmetric Semi-definite (ill-conditioned) Generalized Eigenvalue Problems Zhaojun Bai University of California, Davis Berkeley, August 19, 2016 Symmetric definite generalized eigenvalue problem I Symmetric definite generalized eigenvalue problem Ax = λBx where AT = A and BT = B > 0 I Eigen-decomposition AX = BXΛ where Λ = diag(λ1; λ2; : : : ; λn) X = (x1; x2; : : : ; xn) XT BX = I: I Assume λ1 ≤ λ2 ≤ · · · ≤ λn LAPACK solvers I LAPACK routines xSYGV, xSYGVD, xSYGVX are based on the following algorithm (Wilkinson'65): 1. compute the Cholesky factorization B = GGT 2. compute C = G−1AG−T 3. compute symmetric eigen-decomposition QT CQ = Λ 4. set X = G−T Q I xSYGV[D,X] could be numerically unstable if B is ill-conditioned: −1 jλbi − λij . p(n)(kB k2kAk2 + cond(B)jλbij) · and −1 1=2 kB k2kAk2(cond(B)) + cond(B)jλbij θ(xbi; xi) . p(n) · specgapi I User's choice between the inversion of ill-conditioned Cholesky decomposition and the QZ algorithm that destroys symmetry Algorithms to address the ill-conditioning 1. Fix-Heiberger'72 (Parlett'80): explicit reduction 2. Chang-Chung Chang'74: SQZ method (QZ by Moler and Stewart'73) 3. Bunse-Gerstner'84: MDR method 4. Chandrasekaran'00: \proper pivoting scheme" 5. Davies-Higham-Tisseur'01: Cholesky+Jacobi 6. Working notes by Kahan'11 and Moler'14 This talk Three approaches: 1. A LAPACK-style implementation of Fix-Heiberger algorithm 2. An algebraic reformulation 3. Locally accelerated block preconditioned steepest descent (LABPSD) This talk Three approaches: 1. A LAPACK-style implementation of Fix-Heiberger algorithm Status: beta-release 2.
    [Show full text]
  • 0 BLIS: a Framework for Rapidly Instantiating BLAS Functionality
    0 BLIS: A Framework for Rapidly Instantiating BLAS Functionality FIELD G. VAN ZEE and ROBERT A. VAN DE GEIJN, The University of Texas at Austin The BLAS-like Library Instantiation Software (BLIS) framework is a new infrastructure for rapidly in- stantiating Basic Linear Algebra Subprograms (BLAS) functionality. Its fundamental innovation is that virtually all computation within level-2 (matrix-vector) and level-3 (matrix-matrix) BLAS operations can be expressed and optimized in terms of very simple kernels. While others have had similar insights, BLIS reduces the necessary kernels to what we believe is the simplest set that still supports the high performance that the computational science community demands. Higher-level framework code is generalized and imple- mented in ISO C99 so that it can be reused and/or re-parameterized for different operations (and different architectures) with little to no modification. Inserting high-performance kernels into the framework facili- tates the immediate optimization of any BLAS-like operations which are cast in terms of these kernels, and thus the framework acts as a productivity multiplier. Users of BLAS-dependent applications are given a choice of using the the traditional Fortran-77 BLAS interface, a generalized C interface, or any other higher level interface that builds upon this latter API. Preliminary performance of level-2 and level-3 operations is observed to be competitive with two mature open source libraries (OpenBLAS and ATLAS) as well as an established commercial product (Intel MKL). Categories and Subject Descriptors: G.4 [Mathematical Software]: Efficiency General Terms: Algorithms, Performance Additional Key Words and Phrases: linear algebra, libraries, high-performance, matrix, BLAS ACM Reference Format: ACM Trans.
    [Show full text]
  • On the Performance and Energy Efficiency of Sparse Linear Algebra on Gpus
    Original Article The International Journal of High Performance Computing Applications 1–16 On the performance and energy efficiency Ó The Author(s) 2016 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav of sparse linear algebra on GPUs DOI: 10.1177/1094342016672081 hpc.sagepub.com Hartwig Anzt1, Stanimire Tomov1 and Jack Dongarra1,2,3 Abstract In this paper we unveil some performance and energy efficiency frontiers for sparse computations on GPU-based super- computers. We compare the resource efficiency of different sparse matrix–vector products (SpMV) taken from libraries such as cuSPARSE and MAGMA for GPU and Intel’s MKL for multicore CPUs, and develop a GPU sparse matrix–matrix product (SpMM) implementation that handles the simultaneous multiplication of a sparse matrix with a set of vectors in block-wise fashion. While a typical sparse computation such as the SpMV reaches only a fraction of the peak of current GPUs, we show that the SpMM succeeds in exceeding the memory-bound limitations of the SpMV. We integrate this kernel into a GPU-accelerated Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) eigensolver. LOBPCG is chosen as a benchmark algorithm for this study as it combines an interesting mix of sparse and dense linear algebra operations that is typical for complex simulation applications, and allows for hardware-aware optimizations. In a detailed analysis we compare the performance and energy efficiency against a multi-threaded CPU counterpart. The reported performance and energy efficiency results are indicative of sparse computations on supercomputers. Keywords sparse eigensolver, LOBPCG, GPU supercomputer, energy efficiency, blocked sparse matrix–vector product 1Introduction GPU-based supercomputers.
    [Show full text]
  • DD2358 – Introduction to HPC Linear Algebra Libraries & BLAS
    DD2358 – Introduction to HPC Linear Algebra Libraries & BLAS Stefano Markidis, KTH Royal Institute of Technology After this lecture, you will be able to • Understand the importance of numerical libraries in HPC • List a series of key numerical libraries including BLAS • Describe which kind of operations BLAS supports • Experiment with OpenBLAS and perform a matrix-matrix multiply using BLAS 2021-02-22 2 Numerical Libraries are the Foundation for Application Developments • While these applications are used in a wide variety of very different disciplines, their underlying computational algorithms are very similar to one another. • Application developers do not have to waste time redeveloping supercomputing software that has already been developed elsewhere. • Libraries targeting numerical linear algebra operations are the most common, given the ubiquity of linear algebra in scientific computing algorithms. 2021-02-22 3 Libraries are Tuned for Performance • Numerical libraries have been highly tuned for performance, often for more than a decade – It makes it difficult for the application developer to match a library’s performance using a homemade equivalent. • Because they are relatively easy to use and their highly tuned performance across a wide range of HPC platforms – The use of scientific computing libraries as software dependencies in computational science applications has become widespread. 2021-02-22 4 HPC Community Standards • Apart from acting as a repository for software reuse, libraries serve the important role of providing a
    [Show full text]
  • The BLAS API of BLASFEO: Optimizing Performance for Small Matrices
    The BLAS API of BLASFEO: optimizing performance for small matrices Gianluca Frison1, Tommaso Sartor1, Andrea Zanelli1, Moritz Diehl1;2 University of Freiburg, 1 Department of Microsystems Engineering (IMTEK), 2 Department of Mathematics email: [email protected] February 5, 2020 This research was supported by the German Federal Ministry for Economic Affairs and Energy (BMWi) via eco4wind (0324125B) and DyConPV (0324166B), and by DFG via Research Unit FOR 2401. Abstract BLASFEO is a dense linear algebra library providing high-performance implementations of BLAS- and LAPACK-like routines for use in embedded optimization and other applications targeting relatively small matrices. BLASFEO defines an API which uses a packed matrix format as its native format. This format is analogous to the internal memory buffers of optimized BLAS, but it is exposed to the user and it removes the packing cost from the routine call. For matrices fitting in cache, BLASFEO outperforms optimized BLAS implementations, both open-source and proprietary. This paper investigates the addition of a standard BLAS API to the BLASFEO framework, and proposes an implementation switching between two or more algorithms optimized for different matrix sizes. Thanks to the modular assembly framework in BLASFEO, tailored linear algebra kernels with mixed column- and panel-major arguments are easily developed. This BLAS API has lower performance than the BLASFEO API, but it nonetheless outperforms optimized BLAS and especially LAPACK libraries for matrices fitting in cache. Therefore, it can boost a wide range of applications, where standard BLAS and LAPACK libraries are employed and the matrix size is moderate. In particular, this paper investigates the benefits in scientific programming languages such as Octave, SciPy and Julia.
    [Show full text]
  • Introduchon to Arm for Network Stack Developers
    Introducon to Arm for network stack developers Pavel Shamis/Pasha Principal Research Engineer Mvapich User Group 2017 © 2017 Arm Limited Columbus, OH Outline • Arm Overview • HPC SoLware Stack • Porng on Arm • Evaluaon 2 © 2017 Arm Limited Arm Overview © 2017 Arm Limited An introduc1on to Arm Arm is the world's leading semiconductor intellectual property supplier. We license to over 350 partners, are present in 95% of smart phones, 80% of digital cameras, 35% of all electronic devices, and a total of 60 billion Arm cores have been shipped since 1990. Our CPU business model: License technology to partners, who use it to create their own system-on-chip (SoC) products. We may license an instrucBon set architecture (ISA) such as “ARMv8-A”) or a specific implementaon, such as “Cortex-A72”. …and our IP extends beyond the CPU Partners who license an ISA can create their own implementaon, as long as it passes the compliance tests. 4 © 2017 Arm Limited A partnership business model A business model that shares success Business Development • Everyone in the value chain benefits Arm Licenses technology to Partner • Long term sustainability SemiCo Design once and reuse is fundamental IP Partner Licence fee • Spread the cost amongst many partners Provider • Technology reused across mulBple applicaons Partners develop • Creates market for ecosystem to target chips – Re-use is also fundamental to the ecosystem Royalty Upfront license fee OEM • Covers the development cost Customer Ongoing royalBes OEM sells • Typically based on a percentage of chip price
    [Show full text]
  • High Efficiency Spectral Analysis and BLAS-3
    High Efficiency Spectral Analysis and BLAS-3 Randomized QRCP with Low-Rank Approximations by Jed Alma Duersch A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Applied Mathematics in the Graduate Division of the University of California, Berkeley Committee in charge: Professor Ming Gu, Chair Assistant Professor Lin Lin Professor Kameshwar Poolla Fall 2015 High Efficiency Spectral Analysis and BLAS-3 Randomized QRCP with Low-Rank Approximations Copyright 2015 by Jed Alma Duersch i Contents Contents i 1 A Robust and Efficient Implementation of LOBPCG 2 1.1 Contributions and results . 2 1.2 Introduction to LOBPCG . 3 1.3 The basic LOBPCG algorithm . 4 1.4 Numerical stability and basis selection . 6 1.5 Stability improvements . 10 1.6 Efficiency improvements . 13 1.7 Analysis . 22 1.8 Numerical examples . 25 2 Spectral Target Residual Descent 32 2.1 Contributions and results . 32 2.2 Introduction to interior eigenvalue targeting . 33 2.3 Spectral targeting analysis . 34 2.4 Notes on generalized spectral targeting . 42 2.5 Numerical experiments . 48 3 True BLAS-3 Performance QRCP using Random Sampling 53 3.1 Contributions and results . 53 3.2 Introduction to QRCP . 54 3.3 Sample QRCP . 58 3.4 Sample updates . 64 3.5 Full randomized QRCP . 66 3.6 Parallel implementation notes . 71 3.7 Truncated approximate Singular Value Decomposition . 73 3.8 Experimental performance . 77 ii Bibliography 93 iii Acknowledgments I gratefully thank my coauthors, Dr. Meiyue Shao and Dr. Chao Yang, for their time and expertise. I would also like to thank Assistant Professor Lin Lin, Dr.
    [Show full text]
  • Algorithms and Optimization Techniques for High-Performance Matrix-Matrix Multiplications of Very Small Matrices
    Algorithms and Optimization Techniques for High-Performance Matrix-Matrix Multiplications of Very Small Matrices I. Masliahc, A. Abdelfattaha, A. Haidara, S. Tomova, M. Baboulinb, J. Falcoub, J. Dongarraa,d aInnovative Computing Laboratory, University of Tennessee, Knoxville, TN, USA bUniversity of Paris-Sud, France cInria Bordeaux, France dUniversity of Manchester, Manchester, UK Abstract Expressing scientific computations in terms of BLAS, and in particular the general dense matrix-matrix multiplication (GEMM), is of fundamental impor- tance for obtaining high performance portability across architectures. However, GEMMs for small matrices of sizes smaller than 32 are not sufficiently optimized in existing libraries. We consider the computation of many small GEMMs and its performance portability for a wide range of computer architectures, includ- ing Intel CPUs, ARM, IBM, Intel Xeon Phi, and GPUs. These computations often occur in applications like big data analytics, machine learning, high-order finite element methods (FEM), and others. The GEMMs are grouped together in a single batched routine. For these cases, we present algorithms and their op- timization techniques that are specialized for the matrix sizes and architectures of interest. We derive a performance model and show that the new develop- ments can be tuned to obtain performance that is within 90% of the optimal for any of the architectures of interest. For example, on a V100 GPU for square matrices of size 32, we achieve an execution rate of about 1; 600 gigaFLOP/s in double-precision arithmetic, which is 95% of the theoretically derived peak for this computation on a V100 GPU. We also show that these results outperform I Preprint submitted to Journal of Parallel Computing September 25, 2018 currently available state-of-the-art implementations such as vendor-tuned math libraries, including Intel MKL and NVIDIA CUBLAS, as well as open-source libraries like OpenBLAS and Eigen.
    [Show full text]