Breitwieser et al. Genome Biology (2018) 19:198 https://doi.org/10.1186/s13059-018-1568-0 SOFTWARE Open Access KrakenUniq: confident and fast metagenomics classification using unique k-mer counts F. P. Breitwieser1*, D. N. Baker1,2 and S. L. Salzberg1,2,3* Abstract False-positive identifications are a significant problem in metagenomics classification. We present KrakenUniq, a novel metagenomics classifier that combines the fast k-mer-based classification of Kraken with an efficient algorithm for assessing the coverage of unique k-mers found in each species in a dataset. On various test datasets, KrakenUniq gives better recall and precision than other methods and effectively classifies and distinguishes pathogens with low abundance from false positives in infectious disease samples. By using the probabilistic cardinality estimator HyperLogLog, KrakenUniq runs as fast as Kraken and requires little additional memory. KrakenUniq is freely available at https://github.com/fbreitwieser/krakenuniq. Keywords: Metagenomics, Microbiome, Metagenomics classification, Pathogen detection, Infectious disease diagnosis Background Such false-positive reads typically match only small Metagenomics classifiers attempt to assign a taxonomic portions of a genome, e.g., if a species’ genome con- identity to each read in a dataset. Because metagenomics tains a low-complexity region, and the only reads data often contain tens of millions of reads, classification matching that species fall in this region, then the is typically done using exact matching of short words of species was probably not present in the sample. length k (k-mers) rather than alignment, which would be Reads from microbes that are truly present should unacceptably slow. The results contain read classifica- distribute relatively uniformly across the genome ra- tions but not their aligned positions in the genomes (as ther than being concentrated in one or a few loca- reviewed by [1]). However, read counts can be deceiving. tions. Genome alignment can reveal this information. Sequence contamination of the samples—introduced However, alignment is resource intensive, requiring from laboratory kits or the environment during sample the construction of indexes for every genome and a extraction, handling, or sequencing—can yield high relatively slow alignment step to compare all reads numbers of spurious identifications [2, 3]. Having only against those indexes. Some metagenomics methods small amounts of input material can further compound do use coverage information to improve mapping or the problem of contamination. When using sequencing quantification accuracy, but these methods require for clinical diagnosis of infectious diseases, for example, results from much slower alignment methods as in- less than 0.1% of the DNA may derive from microbes of put [7]. Assembly-based methods also help to avoid interest [4, 5]. Additional spurious matches can result false positives, but these are useful only for highly from low-complexity regions of genomes and from con- abundant species [8]. tamination in the database genomes themselves [6]. Here, we present KrakenUniq, a novel method that combines very fast k-mer-based classification with a fast k-mer cardinality estimation. KrakenUniq is based on * Correspondence: [email protected]; [email protected] the Kraken metagenomics classifier [9], to which it adds 1Center for Computational Biology, McKusick-Nathans Institute of Genetic Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Breitwieser et al. Genome Biology (2018) 19:198 Page 2 of 10 a method for counting the number of unique k-mers of unique k-mers and the depth of k-mer coverage for identified for each taxon using the efficient cardinality each taxon that it observed in the input data (Fig. 1c). estimation algorithm HyperLogLog [10–12]. By counting how many of each genome’s unique k-mers are covered by reads, KrakenUniq can often discern false-positive Efficient k-mer cardinality estimation using the from true-positive matches. Furthermore, KrakenUniq HyperLogLog algorithm implements additional new features to improve metage- Cardinality is the number of elements in a set without nomics classification: (a) searches can be done against duplicates, e.g., the number of distinct words in a text. multiple databases hierarchically; (b) the taxonomy An exact count can be kept by storing the elements in a can be extended to include nodes for strains and plas- sorted list or linear probing hash table, but that requires mids, thus enabling their detection; and (c) the data- memory proportional to the number of unique elements. base build script allows the addition of > 100,000 When an accurate estimate of the cardinality is sufficient, viruses from the NCBI Viral Genome Resource [13]. however, the computation can be done efficiently with a KrakenUniq provides a superset of the information very small amount of fixed memory. The HyperLogLog al- provided by Kraken while running equally fast or gorithm (HLL) [10], which is well suited for k-mer count- slightly faster and while using very little additional ing [14], keeps a summary or sketch ofthedatathatis memory during classification. sufficient for precise estimation of the cardinality and re- quires only a small amount of constant space to estimate cardinalities up to billions. The method centers on the idea Results that long runs of leading zeros, which can be efficiently KrakenUniq was developed to provide efficient k-mer computed using machine instructions, are unlikely in ran- count information for all taxa identified in a metage- dom bitstrings. For example, about every fourth bitstring nomics experiment. The main workflow is as follows: in a random series should start with 012 (one 0 bit before As reads are processed, each k-mer is assigned a taxon the first 1 bit), and about every 32nd hash starts with from the database (Fig. 1a). KrakenUniq instantiates a 000012. Conversely, if we know the maximum number of HyperLogLog data sketch for each taxon and adds the leading zeros k of the members of a random set, we can k k-mers to it (Fig. 1b and Additional file 1: Section 1 on use 2 +1 as a crude estimate of its cardinality (more details the HyperLogLog algorithm). After classification of a in Additional file 1: Section 1 on the HLL algorithm). HLL p read, KrakenUniq traverses up the taxonomic tree and keeps m =2 1 byte counts of the maximum numbers of merges the estimators of each taxon with its parent. In leading zeros on the data (its data sketch), with p, the preci- its classification report, KrakenUniq includes the number sion parameter, typically between 10 and 18 (see Fig. 2). A B C Fig. 1 Overview of the KrakenUniq algorithm and output. a An input read is shown as a long gray rectangle, with k-mers shown as shorter rectangles below it. The taxon mappings for each k-mer are compared to the database, shown as larger rectangles on the right. For each taxon, a unique k-mer counter is instantiated, and the observed k-mers (K7, K8, and K9) are added to the counters. b Unique k-mer counting is implemented with the probabilistic estimation method HyperLogLog (HLL) using 16 KB of memory per counter, which limits the error in the cardinality estimate to 1% (see main text). c The output includes the number of reads, unique k-mers, duplicity (average time each k-mer has been seen), and coverage for each taxon observed in the input data Breitwieser et al. Genome Biology (2018) 19:198 Page 3 of 10 Fig. 2 Cardinality estimation using HyperLogLog for randomly sampled k-mers from microbial genomes. Left: standard deviations of the relative errors of the estimate with precision p ranging from 10 to 18. No systematic biases are apparent, and, as expected, the errors decrease with higher values of p. Up to cardinalities of about 2p/4, the relative error is near zero. At higher cardinalities, the error boundaries stay near constant. Right: the size of the registers, space requirement, and expected relative error for HyperLogLog cardinality estimates with different values of p. For example, with a precision p = 14, the expected relative error is 0.81%, and the counter only requires 16 KB of space, which is three orders of magnitude less than that of an exact counter (at a cardinality of one million). Up to cardinalities of 2p/4, KrakenUniq uses a sparse representation of the counter with a higher precision of 25 and an effective relative error rate of about 0.02% For cardinalities up to about m/4, we use the sparse repre- Results on 21 simulated and 10 biological test datasets sentation of the registers suggested by Heule et al. [11]that We assessed KrakenUniq’s performance on the 34 datasets has the much higher effective precision p′ of 25 by encod- compiled by McIntyre et al. [15](seeAdditionalfile2: ing each index and count in a vector of 4-byte values (see Table S3 for details on the datasets). We place greater em- Fig. 2). To add a k-mer to its taxon’s sketch, the k-mer phasis on the 11 biological datasets, which contain more (with k up to 31) is first mapped by a hash function to a realistic laboratory and environmental contamination.