Matrix Bloom Filter: an Efficient Probabilistic Data Structure for 2-Tuple Batch Lookup

Matrix Bloom Filter: an Efficient Probabilistic Data Structure for 2-Tuple Batch Lookup

Matrix Bloom Filter: An Efficient Probabilistic Data Structure for 2-tuple Batch Lookup Yue Fu∗z, Rong Du∗z, Haibo Huz, Man Ho Auy, Dagang Li∗ ∗School of ECE, Shenzhen Graduate School, Peking University, China yDepartment of Computing, The Hong Kong Polytechnic University, Hong Kong, China zDepartment of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hong Kong, China Email: [email protected], [email protected], [email protected] [email protected], [email protected] Abstract—With the growing scale of big data, probabilistic learning applications are multivariate. In this paper, we focus structures receive increasing popularity for efficient approxi- on a common multivariate data type, namely, 2-tuples, or mate storage and query processing. For example, Bloom filters equivalently, key-value pairs. Besides its wide applications in (BF) can achieve satisfactory performance for approximate membership existence query at the expense of false positives. database and data mining, 2-tuple is also a popular data type However, a standard Bloom filter can only handle univariate in other fields. For example, in a content distribution network data and single membership existence query, which is insuf- (CDN) the nodes (i.e., servers) are labelled as (A,B,C,...), ficient for OLAP and machine learning applications. In this and the contents are numbered by (1,2,3,...). A key problem paper, we focus on a common multivariate data type, namely, in CDN is to determine whether a node has a copy of a 2-tuples, or equivalently, key-value pairs. We design the matrix Bloom filter as a high-dimensional extension of the standard content. This is equivalent to a lookup of a 2-tuple (node Bloom filter. This new probabilistic data structure can not only name, content number) in the CDN metadata. As another insert and lookup a single 2-tuple efficiently, but also support example, most log files in operating systems and web servers these operations efficiently in batches — a key requirement record a timestamp together with the log event. As such, for OLAP and machine learning tasks. To further balance the each log entry can be considered as a 2-tuple in the form of insertion and query efficiency for different workload patterns, we propose two variants, namely, the maximum adaptive (event, timestamp). matrix BF and minimum storage matrix BF. Through both theoretical and empirical studies, we show the performance To support 2-tuple insertion and query, we are tempted of matrix Bloom filter is superior on datasets with common to adapt some prior work. Guo et al. proposed multi- statistical distributions; and even without them, it just degrades dimensional Bloom filter (MDBF) for insertion and query to a standard Bloom filter. of an l-dimensional element (x1; x2; :::; xl) [2]. MDBF Keywords: Bloom filters and hashing, batch processing, tuple query allocates l different standard Bloom filters of the same length, and insert each dimension into its corresponding I. INTRODUCTION Bloom filter. When querying an element, MDBF simply With the advent of Internet of things, all devices around us looks up each dimension in its corresponding Bloom filter. are generating data in day-to-day life. The explosion of data However, MDBF is not suitable for 2-tuple insertion and arXiv:1912.07153v1 [cs.DS] 16 Dec 2019 and wide-adoption of machine intelligence result in billions query. First and foremost, MDBF only performs membership of interactive or automated queries issued and executed per test on each dimension independently, without keeping the day on large web servers or databases. Due to the ever- correlation (e.g. co-existence) across dimensions. A key- increasing scale of data, probabilistic data structures that value pair, however, needs the correlation to be stored and store raw data and execute queries in an approximate manner queried. Second, MDBF also leads to huge performance becomes more and more popular. A classic one is the Bloom inefficiency because a false positive on any dimension al- filter [1], which was first proposed by Bloom in 1970. It is ways results in a false positive on the entire element. Last a bit array (initially set to 0) of length m, and is defined but not the least, MDBF cannot support both operations on k uniform and independent hash functions. To insert an efficiently in batches, which is required in many OLAP and element, BF sets the k positions from hash functions to 1. machine learning tasks. For instance, in CDN it is common To query whether an element has been inserted, BF simply to query on multiple contents such as “Does node A cache checks if all the k positions corresponding to this element contents 1, 4 and 6?”. Likewise, a web administrator may are set to 1. want to know if three IPs 58.61.50.73, 152.168.50.76, and The standard Bloom filter can only handle univariate 125.153.42.12 have accessed the server from 9:00 am - data. However, many data in modern OLAP and machine 10:00am. To support batch operations, MDBF has to treat duplicate keys or duplicate values as special elements and to perform 2-tuple partial existence tests, if there exists an insert them to different positions from where the first key or insertion/query pattern, such that insertions/queries on x1 value is inserted. This makes the batch insertion and query and x2 are conducted independently, while the dependency inefficient. between components remains. To the best of the author’s knowledge, there is no prior study on designing a probabilistic structure that could carry Apparently, 2-tuple partial existence tests are friendly to out batch lookups of 2-tuples. In this paper, we propose the batch queries on one component, since the queries are only matrix Bloom filter as an efficient solution to this problem. to be operated once on the repeating component. Here are In a nutshell, it is a high-dimensional extension of the some existing typical examples that can be concluded into standard Bloom filter that performs insertions and queries 2-tuple partial existence tests: on 2-tuples. Then we further propose two variants, namely, Example 1. Key-value search. Generically, hundreds of the minimum storage matrix and maximum adaptive matrix values can be corresponded to a key. Usually, we locate BF, that can balance the insertion and query efficiency for the key-value pairs through the key, and then search on the different workload patterns, and further exploit common values, which is essentially a batch 2-tuple partial existence statistical distributions of datasets. The contributions of this test. paper are summarized as follows: Example 2. Tmt-query. Peng et al. proposed a question in literature [3]: When an element comes in the form of (IP 1) Matrix Bloom filter. We present a unified framework address, time), how to answer such question “Is there any to batch process 2-tuple partial existence queries by packet comes from a given IP address in the range of time a novel data structure called matrix Bloom filter. It (t ; t )?” degrades to a standard Bloom filter for univariate data. 1 2 2) Minimum storage/maximum adaptive matrix. We Formally, the authors defined the notion of temporal propose two variants in the framework of matrix membership testing: Given an upper bound T on the time Bloom filter, i.e. minimum storage matrix and max- dimension, and a universe U where elements are drawn from, imum adaptive matrix, for datasets with specific sta- a temporal set A consist from (element, timestamp) pairs up tistical features. to time T (assuming discrete timestamps). Let A 2 A, and 3) Empirical studies. We experimentally verify the cor- A[s; e] refers to the subset of distinct elements in A[s; e]. rectness and effectiveness of matrix Bloom filter A temporal membership testing query (tmt-query) asks if against some baseline approaches on two real-world x 2 A[s; e] for any query element x chosen from U. datasets and one synthetic dataset. It is easy to see, tmt-query fixes in the IP address, and The rest of the paper is organized as follows. Section II then search on the range of a given discrete time interval, formally defines the problem and introduces some baseline which is also essentially a batch 2-tuple partial existence approaches. Section III proposes the framework of matrix test. Bloom filter. Section IV proposes the maximum adaptive • Baseline approach: Using a hashmap. A naive ap- matrix and the minimum storage matrix in this framework. proach to answer partial existence tests on key-value pairs Section V discusses the experimental design and results. is the hashmap. Given a set of key-value pairs holding the Section VI reviews the related work. Section VII draws the same key and varying values, it is straightforward to store conclusion. them in a hashmap structure. When queries are performed, the key is only required to be hashed a single time to find II. PROBLEM DEFINITION its bucket, which notably reduces the overall workload2. See A. 2-tuple partial existence tests Fig. 1 as an example of hashmap approach. A key-value pair As mentioned, a traditional Bloom filter does not perform finds its bucket according to the hash output of its key, and batch processing efficiently, as it treats a multi-dimensional values with the same key are mapped into the same buckets. input as an entirety and loses the capability to identify The items in a bucket are connected with a chain. The red common components that two different inputs may have. item in the figure is the case of hash collision, which implies On the other side, even though MDBF [2] is friendly for the hash output of key1 and key2 is the same.

View Full Text

Details

  • File Type
    pdf
  • Upload Time
    -
  • Content Languages
    English
  • Upload User
    Anonymous/Not logged-in
  • File Pages
    12 Page
  • File Size
    -

Download

Channel Download Status
Express Download Enable

Copyright

We respect the copyrights and intellectual property rights of all users. All uploaded documents are either original works of the uploader or authorized works of the rightful owners.

  • Not to be reproduced or distributed without explicit permission.
  • Not used for commercial purposes outside of approved use cases.
  • Not used to infringe on the rights of the original creators.
  • If you believe any content infringes your copyright, please contact us immediately.

Support

For help with questions, suggestions, or problems, please contact us