A Shifting Bloom Filter Framework for Set Queries Tong Yang* Alex X. Liu Muhammad Shahzad Peking University, China Michigan State University North Carolina State University [email protected] [email protected] [email protected] Yuankun Zhong Qiaobin Fu Zi Li Nanjing University, China Boston University, USA Nanjing University, China [email protected] [email protected] [email protected] Gaogang Xie Xiaoming Li ICT, CAS, China Peking University, China [email protected] [email protected] ABSTRACT allows elements to appear more than once. Network mea- Set queries are fundamental operations in computer system- surement applications, such as measuring flow sizes, often s and applications. This paper addresses the fundamental use multiplicity queries. problem of designing a probabilistic data structure that can This paper addresses the fundamental problem of design- quickly process set queries using a small amount of memory. ing a probabilistic data structure that can quickly process We propose a Shifting Bloom Filter (ShBF) framework for set queries, such as the above-mentioned membership, as- representing and querying sets. We demonstrate the effec- sociation, and multiplicity queries, using a small amount of tiveness of ShBF using three types of popular set queries: memory. Set query processing speed is critical for many sys- membership, association, and multiplicity queries. The key tems and applications, especially for networking applications novelty of ShBF is on encoding the auxiliary information as packets need to be processed at wire speed. Memory con- of a set element in a location offset. In contrast, prior BF sumption is also critical because small memory consumption based set data structures allocate additional memory to s- may allow the data structure to be stored in SRAM, which tore auxiliary information. We conducted experiments using is an order of magnitude faster than DRAM. real-world network traces, and results show that ShBF sig- Widely used set data structures are the standard Bloom nificantly advances the state-of-the-art on all three types of Filter (BF) [3] and the counting Bloom Filter (CBF) [11]. set queries. Let h1(.), ··· ,hk(.)bek independent hash functions with uniformly distributed outputs. Given a set S,BFcon- 1. INTRODUCTION structs an array B of m bits, where each bit is initial- 1.1 Motivations ized to 0, and for each element e∈S,BFsetsthek bits ··· Set queries, such as membership queries, association B[h1(e)%m], ,B[hk(e)%m]to1.Toprocessamember- queries,andmultiplicity queries, are fundamental operations ship query of whether element e is in S, BF returns true if all ∧k in computer systems and applications. Membership queries corresponding k bits are 1 (i.e., returns i=1B[hi(e)%m]). ∈ check whether an element is a member of a given set.Net- BF has no false negatives (FNs), i.e., it never says that e/S ∈ work applications, such as IP lookup, packet classification, when actually e S. However, BF has false positives (FP- ∈ ∈ and regular expression matching, often involve membership s), i.e.,itmaysaythate S when actually e/S with a queries. Association queries identify which set(s) among a certain probability. Note that BF does not support element pair of sets contain a given element. Network architectures deletion. CBF overcomes this shortcoming by replacing each such as distributed servers often use association queries. For bit in BF by a counter. Given a set of elements, CBF first example, when data is stored distributively on two servers constructs an array C of m counters, where each counter is and the popular content is replicated over both servers to initialized to 0. For each element e in S,foreach1 i k, achieve load balancing, for any incoming query, the gate- CBF increments C[hi(e)%m] by 1. To process a member- way needs to identify the server(s) that contain the data ship query of whether element e is in set S, CBF returns corresponding to that query. Multiplicity queries check how true if all corresponding k counters are at least 1 (i.e.,re- ∧k many times an element appears in a multi-set. A multi-set turns i=1(C[hi(e)%m] 1)). To delete an element e from S,foreach1 i k, CBF decrements C[hi(e)%m]by1. 1.2 Proposed Approach ∗ Tong Yang is also with State key Laboratory of Networking and Switch- In this paper, we propose a Shifting Bloom Filter (ShBF) ing Technology (Beijing University of Posts and Telecommunications). This framework for representing and querying sets. Let h1(.), ···, work is licensed under the Creative Commons Attribution-NonCommercial- hk(.)bek independent hash functions with uniformly dis- NoDerivatives 4.0 International License. To view a copy of this license, vis- tributed outputs. In the construction phase, ShBF first con- it http://creativecommons.org/licenses/by-nc-nd/4.0/. For any use beyond structs an array B of m bits, where each bit is initialized to 0. those covered by this license, obtain permission by emailing [email protected]. Proceedings of the VLDB Endowment, Vol. 9, No. 5 We observe that in general a set data structure needs to store Copyright 2016 VLDB Endowment 2150-8097/16/01. two types of information for each element e:(1)existence in- 408 formation, i.e.,whethere is in a set, and (2) auxiliary infor- is another hash function with uniformly distributed outputs mation, i.e., some additional information such as e’s counter and w is a function of machine word size w.Forthethird (i.e., multiplicity) or which set that e is in. For each ele- case, i.e., e ∈ S2 − S1, the offset function o(e)=o2(e)= ment e,weencodeitsexistenceinformationink hash values o1(e)+hk+2(e)%((w − 1)/2) + 1, where hk+2(.)isyetan- h1(e)%m, ··· ,hk(e)%m, and its auxiliary information in an other hash function with uniformly distributed outputs. In offset o(e). Instead of, or in addition to, setting the k bits at the construction phase, for each element e ∈ S1 ∪ S2,we locations h1(e)%m, ··· ,hk(e)%m to 1, we set the bits at lo- set the k bits B[h1(e)%m + o(e)], ··· ,B[hk(e)%m + o(e)] cations (h1(e)+o(e))%m, ··· , (hk(e)+o(e))%m to 1. For dif- to 1 using an appropriate value of o(e)asjustdescribed ferent set queries, the offset has different values. In the query for the three cases. In the query phase, given an ele- phase, to query an element e, we first calculate the following ment e ∈ S1 ∪ S2 ,foreach1 i k,wereadthe3 k locations: h1(e)%m, ··· ,hk(e)%m.Letc be the maximum bits B[hi(e)%m], B[hi(e)%m + o1(e)], and B[hi(e)%m + value of all offsets. For each 1 i k,wefirstreadthec o2(e)]. If all the k bits B[h1(e)%m], ··· ,B[hk(e)%m]are bits B[hi(e)%m],B[(hi(e)+1)%m], ··· ,B[(hi(e)+c−1)%m] 1, then e may belong to S1 − S2.Ifallthek bits and then calculate the existence and auxiliary information B[h1(e)%m + o1(e)], ··· ,B[hk(e)%m + o1(e)] are 1, then about e by analyzing where 1s appear in these c bits. To e may belong to S1 ∩ S2.Ifallthek bits B[h1(e)%m + minimize the number of memory accesses, we extend the o2(e)], ··· ,B[hk(e)%m + o2(e)] are 1, then e may belong to c 2 − 1 number of bits in ShBF to m + c; thus, we need k w num- S S . There are a few other possibilities that we will ber of memory accesses in the worst case, where w is the discuss later in Section 4.2, that ShBF takes into account word size. Figure 1 illustrates our ShBF framework. when answering the association queries. In comparison, the standard BF based association query scheme, namely iBF, constructs a BF for each set. In terms of accuracy, iBF is prone to false positives whenever it declares an elemen- t e ∈ S1 ∪ S2 in a query to be in S1 ∩ S2, whereas ShBF 1 1 1 achieves an FPR of zero. In terms of performance, ShBF is almost twice as fast as iBF because iBF needs 2k hash Figure 1: Shifting Bloom Filter framework. functions and 2k memory accesses per query, whereas ShBF We demonstrate the effectiveness of ShBF using three needs only k + 2 hash functions and k memory accesses per types of popular set queries: membership, association, and query. multiplicity queries. 1.2.3 Multiplicity Queries 1.2.1 Membership Queries For multiplicity queries, for each element e in a multi- Such queries only deal with the existence information of set S, the offset function o(.)=c(e) − 1wherec(e)ise’s each element, which is encoded in k random positions in counter (i.e., the number of occurrences of e in S). In the array B. To leverage our ShBF framework, we treat k/2 construction phase, for each element e,wesetthek bits positions as the existence information and the other k/2 B[h1(e)%m+c(e)−1], ··· ,B[hk(e)%m+c(e)−1] to 1. In the positions as the auxiliary information, assuming k is an query phase, for an element e,foreach1 i k,weread even number for simplicity. Specifically, the offset func- the c bits B[hi(e)%m],B[hi(e)%m+1], ··· ,B[hi(e)%m+c− tion o(.)=h k +1(.)%(w − 1) + 1, where h k +1(.)isanoth- 2 2 1], where c is the maximum number of occurrences that an er hash function with uniformly distributed outputs and w element can appear in S.Fortheseck bits, for each 1 j is a function of machine word size w.Intheconstruc- c,ifallthek bits B[h1(e)%m+j−1], ··· ,B[hk(e)%m+j−1] tion phase, for each element e ∈ S, we set both the are 1, then we output j as one possible value of c(e).