An Efficient GPU-Accelerated Implementation of Genomic Short Read Mapping with BWA-MEM Ernst Joachim Houtgast1;2, Vlad-Mihai Sima2, Koen Bertels1 and Zaid Al-Ars1 1 Computer Engineering Lab, TU Delft, Mekelweg 4, 2628 CD Delft, The Netherlands 2 Bluebee, Laan van Zuid Hoorn 57, 2289 DC Rijswijk, The Netherlands Corresponding author: [email protected] ABSTRACT These variants, or mutations, are generally what is of inte- Next Generation Sequencing techniques have resulted in an rest, as such a mutation could give insight on which is the exponential growth in the generation of genetics data, the most effective treatment to follow for the particular illness a amount of which will soon rival, if not overtake, other Big patient has. The mapping stage takes a significant portion Data fields, such as astronomy and streaming video services. of processing time for a typical pipeline execution, around To become useful, this data requires processing by a complex 30%-40%, depending on data set and platform. pipeline of algorithms, taking multiple days even on large A sequencing run on an Illumina HiSeq X, a state-of-the- clusters. The mapping stage of such genomics pipelines, art NGS sequencer, produces data in the order of 450 GB. which maps the short reads onto a reference genome, takes For cancer data sets, this data requires multiple days of up a significant portion of execution time. BWA-MEM is processing, even on high performance computing clusters. the de-facto industry-standard for the mapping stage. The extreme scale of data and processing requires enormous Here, a GPU-accelerated implementation of BWA-MEM computing capabilities to make the analysis feasible within is proposed. The Seed Extension phase, one of the three a realistic time frame. As heterogeneous computing holds main BWA-MEM algorithm phases that requires between great potential to provide large advantages in speed and 30%-50% of overall processing time, is offloaded onto the efficiency, in this paper, we demonstrate the effectiveness GPU. A thorough design space analysis is presented for an of GPU-based acceleration of BWA-MEM, the most widely optimized mapping of this phase onto the GPU. The re- used tool for the mapping stage of genomics pipelines. sulting systolic-array based implementation obtains a two- The following contributions are made: 1) an optimized fold overall application-level speedup, which is the maximum GPU-based implementation of the BWA-MEM Seed Exten- theoretically achievable speedup. Moreover, this speedup is sion phase, resulting in an overall application-level speedup sustained for systems with up to twenty-two logical cores. of up to 2x; 2) a thorough discussion of the design space ana- Based on the findings, a number of suggestions are made to lysis, providing key insight into the requirements of a highly improve GPU architecture, resulting in potentially greatly optimized implementation; and 3) recommendations to fur- increased performance for bioinformatics-class algorithms. ther improve the GPU architecture that would allow even higher performance for bioinformatics-class applications. The remainder of this paper is organized as follows. In 1. INTRODUCTION Section 2, related work is discussed. In Section 3, back- The introduction of Next Generation Sequencing (NGS) ground information is given on the BWA-MEM algorithm techniques has resulted in drastic, ongoing, cost reduction of and, in particular, on the Seed Extension phase. In Sec- genomic sequencing, which, in turn, has led to an enormous tion 4, the advantages and disadvantages of various imple- growth in the amount of genetic DNA data that is being se- mentation architectures are reviewed. In Section 5, the re- quenced. High-throughput sequencing facilities are coming sults are presented. Section 6 contains a discussion of the online around the world as facilities worldwide embrace NGS results and recommendations are made to improve the GPU [2]. The amount of data being generated is projected to architecture. Section 7 concludes the paper. rival, if not outright overtake, other key Big Data-fields, such as astronomy and streaming video services [13]. 2. RELATED WORK NGS machines output so-called short reads, short frag- Numerous GPU-accelerated implementations of short read ments of DNA of at most a few hundred base pairs (bp) mapping tools exist, notable examples include SOAPv3 [9] in length. This data requires extensive processing using a and CUSHAW [10]. Similar to BWA-MEM and most other genomics pipeline, which typically contain multiple stages state-of-the-art mapping tools, these consist of an Exact with a number of highly complex algorithms. In the case of Matching phase using the Burrows-Wheeler transform to a DNA sequencing pipeline, first, the millions of short reads find exactly matching subsequences, followed by an Inexact generated are mapped onto a reference genome. Then, these Matching phase. However, these implementations are lim- mapped reads are sorted and duplicates are marked or re- ited in the flexibility of their Inexact Matching algorithm, moved. Finally, the aligned data is compared at several allowing only for a small number of mismatches (CUSHAW), positions with known possibilities, in order to determine the or by disallowing gaps in the alignment (SOAPv3). most probable variant. Only then the data is ready for con- Using a variant of the Smith-Waterman (SW) algorithm sumption by the end-user, such as a clinician or researcher. [12] for its Inexact Matching, BWA-MEM does not impose such limitations. For example, gaps do not influence per- formance. The SW algorithm is a dynamic programming To appear in the International Symposium on Highly Efficient Acceler- technique able to find the optimal match between two sub- ators and Reconfigurable Technologies, July 2016, Hong Kong. sequences given a certain scoring scheme. Many accelera- 45 HEART2016, Hong Kong Figure 1: BWA-MEM processes reads using the Seed-and-Extend paradigm: for each read, likely mapping locations on the reference are found by searching for seeds, exactly matching subsequences between the read and the reference. These seeds are then extended in both directions using a Smith-Waterman-like approach allowing for inexact matches. The best scoring alignment is selected. ted implementations of this algorithm exist (e.g., [8], [11]). algorithm more challenging, as not only does it require the However, all these implementations perform one complete adaptation of multiple separate algorithms, but also care has sequence matching per compute thread, making such an im- to be taken to not shift the bottleneck to another part of the plementation unsuitable for direct application onto BWA- application, limiting the benefit of any potential speedup as MEM, as it requires batching and sorting of larger groups per Amdahl's law. This makes it quite difficult to obtain of work. Section 3.2 explains in more detail why such a larger performance gains. parallelization strategy is inapplicable for BWA-MEM. To the authors' knowledge, only a few accelerated imple- 3.1 The BWA-MEM Algorithm mentations of BWA-MEM exist: two FPGA implementa- The goal of the BWA-MEM algorithm is to find the best tions of BWA-MEM on the Convey supercomputing plat- mapping of a short read onto a reference genome [7]. To form: one offloading the Seed Extension phase onto four achieve this, it makes use of the Seed-and-Extend paradigm Xilinx Virtex-6 FPGAs [4] obtaining a 1.5x speedup, the (refer to Figure 1), a two-step method consisting of an Exact other accelerating multiple BWA-MEM phases [1] obtain- Matching phase and an Inexact Matching phase (for details, ing a 2.6x speedup; and a GPU-accelerated implementation see [1]). First, for each short read Seed Generation is per- of the Seed Extension phase [5], achieving a 1.6x speedup. formed: exactly matching subsequences of the read and re- This work improves upon [5], obtaining far better results: a ference called seeds are identified using a Burrows-Wheeler two-fold speedup for a system with up to twenty-two logi- Transform-based index. The BWT-method allows for effi- cal cores is obtained, compared to an at most 1.6x speedup cient string-lookup and forms the fundament of almost all for a system with up to four cores. Moreover, an NVIDIA contemporary state-of-the-art mapping tools. A single short GeForce GTX 970 is used, compared to using a setup with read can have many such seed locations identified. Genera- dual NVIDIA GeForce GTX TITAN X, equivalent to about ted seeds that are found to be in close proximity of each one-third of the GPU resources. Note that all these imple- other on the reference genome are grouped into chains. mentations are actual production-quality implementations. The Seed Generation phase is followed by a Seed Exten- sion phase. Here, seeds found earlier on are extended using an algorithm similar to the widely-used Smith-Waterman 3. BACKGROUND algorithm, using a scoring system that awards matches and There are a number of traits that bioinformatics-class penalizes mismatches, insertions and gaps. Typically, not algorithms share, making them interesting, but neverthe- all seeds are extended. Instead, on average only one seed less challenging candidates for acceleration efforts. The two per chain is extended. Out of all the extended seeds, the most important ones are outlined below: highest scoring match is chosen as final alignment. Extreme-Scale Data Size: The data size that many bioinformatics applications deal with are of an enormous 3.2 Seed Extension Phase magnitude, for example illustrated in the case of NGS se- BWA-MEM contains three main computational phases: quencing. A single human genome contains three billion Seed Generation, Seed Extension and Output Generation. base pairs. A base is one out of four possible nucleotides During Output Generation the best alignment is selected (A, C, G or T).