Probabilistic Data Structures Pedro Ribeiro DCC/FCUP 2020/2021 (heavily inspired/based on the lecture notes by Jeff Erickson @ University of Illinois at Urbana-Champaign) Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 1 / 36 What are probabilistic data structures? Data structures that use some randomized algorithm or takes advantage of some probabilistic characteristics internally There are mainly two types of randomized algorithms: I Las Vegas algorithm: always outputs the correct answer, but runtime is a random variable I Monte Carlo algorithm: always terminates in given time bound, and outputs the correct answer with at least some (high) probability Likewise, we have two types of probabilistic data structures: I Some always give exact results, but runtime is random variable F E.g.: Treaps, Skip Lists I Some have bounded runtime, but give approximate results F E.g.: Bloom Filters, Count-Min Sketches, HyperLogLog Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 2 / 36 Random Binary Search Trees What happens when we insert elements in randomized order in a binary search tree? I What is the expected depth of a node? The average efficiency of searching, inserting and removing from that random tree is related to the expected depth of a node I Depth might n, but what is the chance that we are that "unlucky"? (all permutations are equally likely) Let's have a first empirical look at this: Gnarley Trees - Visualization of Data Structures https://people.ksp.sk/~kuko/gnarley-trees/ 97 nodes, height = 12 = 1.71 ·opt, Avg. Depth = 7.09 Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 3 / 36 Random Binary Search Trees The depth seems to be always just a (small) constant away from the optimal and therefore logarithmic in terms of expected value Can we prove this? Example (Theorem - Expected Depth in Random BST ) If n values are inserted in random order on a binary search tree, the expected depth of any node will be O(log n) Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 4 / 36 Expected Depth of a Node in a Random BST Let xk denote the node the k-th smallest of n search keys We will use i " k to denote that xi is a (proper) ancestor of xk The depth of a node is simply the number of its ancestors, so: n X depth(xk ) = [i " k] i=1 [i " k] is just an indicator variable of i " k) this is called the Iverson bracket notation We can now express the expected depth of a node as: " n # n n X X h i X E[depth(xk )] = E [i " k] = E [i " k] = Pr[i " k] i=1 i=1 i=1 using linearity of expectation: E[X + Y ] = E[X ] + E[Y ] and indicator variables: E[X ] = Pr[X = 1] if X is an indicator variable Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 5 / 36 Expected Depth of a Node in a Random BST Let X (i; k) denote the subset of node between xi and xk I fxi ; xi+1; ··· ; xk g if i < k I fxk ; xk+1; ··· ; xi g if i > k If i 6= k, we have i " k if and only if xi was the first element being inserted in X (i; k) I xi must obviously be inserted before xk I if j is between i and k (i < j < k or i > j > k) and xj is inserted before xi , then xi and xj will end up in different subtrees Each node is in X (i; k) is equally likely to be the first one inserted, so: 8 1 if i < k [i 6= k] <> k−i+1 Pr[i " k] = = 0 if i = k jk − ij + 1 :> 1 i−k+1 if i > k Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 6 / 36 Expected Depth of a Node in a Random BST Plugging it in the expectation formula: n k−1 n X X 1 X 1 [depth(x )] = Pr[i " k] = + E k k − i + 1 i − k + 1 i=1 i=1 i=k+1 k n−k+1 X 1 X 1 = + j j j=2 j=2 = Hk − 1 + Hn−k+1 − 1 ≤ ln k + ln(n − k + 1) ≤ 2 ln n 2 O(log n) 1 1 [the sum Hn = 1 + 2 + ··· + n is known as the harmonic series and Hn ≤ ln(n) + 1] The expected depth of a node is logarithmic in relation to the nr of nodes Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 7 / 36 Using the ideas Random BSTs So, on a random BST with n keys, the nodes are expected to be at O(log n) depth However, we are not so lucky that the insertion order is random... Can we guarantee the same properties for any insertion order? "yes we can", and treaps are here for the rescue :) Aragon, C. R., and Seidel, R. (1989). Randomized search trees. In IEEE Symposium on Foundations of Computer Science (FOCS) (Vol. 30, pp. 540-545). Note: there are other ways of obtaining the same properties, such as: Mart´ınez,C., & Roura, S. (1998). Randomized binary search trees. Journal of the ACM (JACM), 45(2), 288-323. Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 8 / 36 Treaps - Concept A treap is a binary tree in which every node has both a search key and a priority, where the inorder sequence of search keys is sorted and each node's priority is smaller than the priorities of its children. In other words, it is simultaneously: I a binary search tree for the search keys I a (min-)heap for the priorities Let's see a first example using letters meaning search keys and numbers for the priorities: Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 9 / 36 Treaps are uniquely determined Let's assume that all keys and priorities are distinct The structure of the treap is completely determined by the search keys and priorities of its nodes I Since it's a heap: F the node v with highest priority must be the root I Since it's a BST: F any node u with key(u) < key(v) must be in the left subtree F any node w with key(w) > key(v) must be in the right subtree I Since the subtrees are treaps, by induction, their structures are completely determined (the base case is the empty treap). In other words, a treap is exactly the binary search tree that results of inserting the nodes in order of increasing priority into an initially empty tree (using the standard textbook insertion algorithm) Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 10 / 36 Treap Operations - Search Search is done as in a normal BST I Successful search has time proportional to the depth of the node I Unsuccessful search has time proportional to the depth of its predecessor or ancestor (time is proportional to the depth of the node) Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 11 / 36 Treap Operations - Insert Insertion is done in the following way I Let's call z to the new node I You do the insertion using the standard BST algorithm I This results in z being a leaf, at the bottom of the tree I At this point you have a BST, but priorities may no longer form a heap I To fix this, you do something similar to a standard heap: F As long as the parent of z has a smaller priority, you perform a rotation at z, decreasing the depth of z (and increasing the depth of the parent), while keeping the BST property F One rotation takes constant time Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 12 / 36 Treap Operations - Insert/Remove The overall time to insert z is proportional to the depth of z before the rotations (roughly equi. to 2× of an unsucessfull search for z): I we have to walk down the treap to insert z I and then walk back up the treap doing rotations This suggest an algorithm for removing an element z (as the inverse in time of insertion) I push z to the bottom of three using rotation I simply remove z, which is now a leaf I time is proportional to depth of z Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 13 / 36 Treap Operations - Split/Join Treaps are also very handy for splitting: imagine we want to split a treap T into two treaps T< and T> along some pivot π I T< contains all nodes with keys smaller than π I T> contains all nodes with keys bigger than π A simple way to do this is to insert a new node z with key(z) = π and priority(z) = −∞ I after the insertion z is the root I deleting the root the left and right subtrees are T < and T > Joining two treaps T < and T > is just splitting in reverse I create a dummy root, rotate it to the bottom and chop it off For both splitting or joining, time is again proportional to the depth Pedro Ribeiro (DCC/FCUP) Probabilistic Data Structures 2020/2021 14 / 36 Randomized Treaps A randomized treap is a treap in which the priorities are independently and uniformly distributed random variables. Whenever we insert a new search key into the treap, we generate a random real number between (say) 0 and 1 and use that number as the priority of the new node.