CSE 535 : Lecture 5 String Matching with Bloom Filters Washington University Fall 2003 http://www.arl.wustl.edu/arl/projects/fpx/cse535/ Copyright 2003, Sarang Dharmapurikar [Guest Lecture] CSE 535 : Fall 2003 1 Background on Bloom Filter • Data structure proposed by Burton Bloom • Randomized data structure – Strings are stored using multiple hash functions – It can be queried to check the presence of a string • Membership queries result in rare false positives but never false negatives • Originally used for UNIX spell check • Modern applications include : – Content Networks – Summary Caches – route trace-back – Network measurements – Intrusion Detection CSE 535 : Fall 2003 2 Hash Functions • Input : x •Output : H[x] • Properties – Each value of x maps to a value of H[x] – Typically: Size of (x) >> Size of (H[x]) – H[x] evenly distributed over values of x • Implementation – Hash Function • XOR of bits, Shifting, rotates .. X H H[X] CSE 535 : Fall 2003 3 Programming a Bloom Filter • Bloom filter computes k hash functions on input 1 1 H1 1 X H2 H3 H k 1 1 m-bit vector CSE 535 : Fall 2003 4 Programming a Bloom Filter 1 1 1 H1 1 Y H2 H 3 1 H k 1 1 1 m-bit vector CSE 535 : Fall 2003 5 Querying a Bloom Filter 1 1 1 H1 1 X H2 H 3 1 match H k 1 1 1 m-bit vector CSE 535 : Fall 2003 6 Querying a Bloom Filter 1 1 1 H1 1 W H2 H 3 1 Match H (false positive) k 1 1 1 m-bit vector CSE 535 : Fall 2003 7 Optimal Parameters of a Bloom filter • n : number of strings to be stored 1 • k : number of hash functions 1 • m : the size of the bit-array (memory) 1 H1 • The false positive probability H 1 Y 2 k H f = (½) 3 1 H4 H • The optimal value of hash functions, k, k 1 is k = ln2 × m/n = 0.693 × m/n 1 1 m-bit Array Key Point : False positive probability decreases exponentially with linear increase in the number of hash functions & memory CSE 535 : Fall 2003 8 Counting Bloom Filters • A message once programmed in the Bloom filter can not be deleted – Deletion of message requires clearing the corresponding bits – Since a bit can be set by multiple messages, clearing it will disturb other messages • Counting Bloom filters solve the problem – Array of counters instead of array of bits – Increment the corresponding counters when a message is added, decrement when deleted 1 2 0 0 A A 1 1 1 1 0 0 B 0 0 B 1 2 0 0 off-chip counter array CSE 535 : Fall 2003 9 Counting Bloom Filters • Maintain Bloom filters on the chip and corresponding counters off the chip 1 2 – Saves on the on-chip 0 0 resources to implement 1 1 counters 1 1 0 0 – Addition and deletion of 0 0 messages are rare 1 2 0 0 – Set the bit when On-chip bit array off-chip counter array corresponding counter changes 0 to 1, clear it when counter changes 1 to 0 CSE 535 : Fall 2003 10 Using Bloom filters for String Matching Hash Table False Positives Resolver BFW BF5 BF4 BF3 Entering byte bW --------- b5 b4 b3 b2 b1 Leaving byte CSE 535 : Fall 2003 11 Bloom filter for cs535 Hash Table BF16 Entering byte bW --------- b5 b4 b3 b2 b1 Leaving byte CSE 535 : Fall 2003 12 System Overview Off-chip • Receives the control packets, 64 Mega bytes decodes the commands in it and SDRAM accordingly either updates the A component that reads-writes data Bloom filter or updates the hash given by the user component in the table off-chip SDRAM SDRAM Controller • Implements the hash-table Sends Control packets to around SDRAM CPP and data packets to Bloom filter • communicates with the SDRAM Hash Table Interface controller through a request grant protocol When hash table instructs, it Process the packet Control sends a notification packet out headers Packet Processor Bloom Filter Input Output Controller Controller Protocol Wrappers CSE 535 : Fall 2003 13 Bloom Filters on the FPX Platform Bloom filters • Xilinx XCV2000E FPGA implemented on the Reconfigurable – Implements Application Device Reconfigurable Application Device (RAD) on the Field- programmable Port Extender (FPX) – Contains 160 Embedded RAMs • Each BlockRAM has dual (2) ports • Each BlockRAM stores 4096 bits – Enables MP2 to Field-programmable Port Extender (FPX) Platform implement large, fast, parallel Bloom filters CSE 535 : Fall 2003 14 Partial Bloom Filter 1 bit dinA weA doutA ‘0’ addrA Output H1(X) X Hash dinB (match/no match) Value 4096 bits doutB Calculator weB ‘0’ addrB H2(X) CSE 535 : Fall 2003 15 Partial Bloom Filter Address Valid PBF BRAM # Bit 1 bit dinA Request weA doutA Decoder addrA Output H1(X) X Hash dinB (match/no match) Value 4096 bits doutB Calculator weB ‘0’ addrB H2(X) CSE 535 : Fall 2003 16 Bloom Filter Control Interface H1 H2 PBF 1 H3 H4 PBF 2 H5 X Hash Value H6 PBF 3 Calculator Match H7 H8 PBF 4 H9 H10 PBF 5 CSE 535 : Fall 2003 17.