Practical Concurrent Traversals in Search Trees Dana Drachsler-Cohen∗ Martin Vechev Eran Yahav ETH Zurich, Switzerland ETH Zurich, Switzerland Technion, Israel [email protected] [email protected] [email protected]
Abstract efficiently remains a difficult11 task[ ]. This task is partic- Operations of concurrent objects often employ optimistic ularly challenging for search trees whose traversals may concurrency-control schemes that consist of a traversal fol- be performed concurrently with modifications that relocate lowed by a validation step. The validation checks if concur- nodes. Thus, a major challenge in designing a concurrent rent mutations interfered with the traversal to determine traversal operation is to ensure that target nodes are not if the operation should proceed or restart. A fundamental missed, even if they are relocated. This challenge is ampli- challenge is to discover a necessary and sufficient validation fied when a concurrent data structure is optimized forread check that has to be performed to guarantee correctness. operations and traversals are expected to complete without In this paper, we show a necessary and sufficient condi- costly synchronization primitives (e.g., locks, CAS). Recent tion for validating traversals in search trees. The condition work offers various concurrent data structures optimized for relies on a new concept of succinct path snapshots, which are lightweight traversals [1, 2, 4, 6–10, 12–14, 18, 19, 22, 25, 27]. derived from and embedded in the structure of the tree. We However, they are mostly specialized solutions, supporting leverage the condition to design a general lock-free mem- standard operations, that cannot be applied directly to other bership test suitable for any search tree. We then show how structures or easily extended with customized operations to integrate the validation condition in update operations (e.g., rotations). Thus, when customized tree operations are of (non-rebalancing) binary search trees, internal and exter- needed, programmers resort to general locking protocols. nal, and AVL trees. We experimentally show that our new Unfortunately, these protocols yield non-scalable solutions. algorithms outperform existing ones. We present a necessary and sufficient condition for path traversal validation for search trees (PaVT). PaVT is applica- CCS Concepts • Computing methodologies → Concur- ble to any search tree, including internal (all nodes have pay- rent algorithms; loads) and external trees (only leaves have payloads). Infor- Keywords Concurrency, Search Trees mally, the PaVT condition states that to determine whether a certain key is in the tree, it suffices to observe only a limited ACM Reference Format: number of nodes that are the succinct path snapshot (SPS) of Dana Drachsler-Cohen, Martin Vechev, and Eran Yahav. 2018. Prac- the path leading to this key. Moreover, concurrent updates tical Concurrent Traversals in Search Trees. In Proceedings of PPoPP may modify this path and the traversal may still complete ’18: 23nd ACM SIGPLAN Symposium on Principles and Practice of successfully if it has found the correct snapshot (Sec. 4). Parallel Programming, Vienna, Austria, February 24–28, 2018 (PPoPP ’18), 12 pages. We leverage PaVT to design a lock-free membership test https://doi.org/10.1145/3178487.3178503 (Sec. 5). To validate the PaVT condition efficiently, we extend nodes with their SPS, which precisely characterizes the path by which they are reached from the root of the tree. The 1 Introduction existence of the SPS allows us to validate a lock-free traversal Concurrent data structures are critical components of many (e.g., a lookup operation) – and to linearize that traversal systems [26]. However, implementing them correctly and with respect to concurrent mutations – by verifying, locally, that the SPS matches the path just taken by the traversal. ∗Work done while at Technion. We then show how to use PaVT in updates (Sec. 6). An update starts with a traversal and before validating, it blocks Permission to make digital or hard copies of all or part of this work for concurrent updates to the snapshot (with a local lock) to personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that enable updates to proceed safely, if validation succeeds. We copies bear this notice and the full citation on the first page. Copyrights show that this approach provides simple algorithms for in- for components of this work owned by others than the author(s) must sertion and removal in binary search trees (BSTs). be honored. Abstracting with credit is permitted. To copy otherwise, or Finally, we empirically evaluate the effectiveness of PaVT republish, to post on servers or to redistribute to lists, requires prior specific (Sec. 7). We observe that SPSs impose only modest space permission and/or a fee. Request permissions from [email protected]. PPoPP ’18, February 24–28, 2018, Vienna, Austria overhead and that maintaining them during tree mutations © 2018 Copyright held by the owner/author(s). Publication rights licensed imposes only modest time overhead. We also experimentally to Association for Computing Machinery. show that the PaVT-ed BST and PaVT-ed AVL tree outper- ACM ISBN 978-1-4503-4982-6/18/02...$15.00 form existing solutions. https://doi.org/10.1145/3178487.3178503 PPoPP ’18, February 24–28, 2018, Vienna, Austria Dana Drachsler-Cohen, Martin Vechev, and Eran Yahav
2 Preliminaries and Challenges 4 3 L R L R The PaVT condition guides how to efficiently traverse in search trees. Search trees consist of nodes. Each node n has 3 12 3 9 one or more keys, denoted by n.k1,n.k2,... (or simply n.k if there is one key) and m fields f1,..., fm pointing to other nodes. We assume m ≥ 2. In addition to the m fields, a node 9 9 has access to its parent via n.P. Membership tests in trees look for a key k and return a boolean indicating whether k is in the tree. They start with a traversal that begins at the (a) SPS of search(6) in (b) SPS of search(6) in root and proceeds to other nodes by repeatedly checking internal BST. external BST. conditions corresponding to the fields. The traversal termi- nates when reaching ⊥ (i.e., null) or when no condition is 1 3 L R met. At this point, the membership test determines (tests) M whether k is in the tree. In a sequential setting, k is in the 2 4 9 tree if and only if the traversal has not reached ⊥. Examples Fig. 1 shows examples for search trees. An in- ternal binary search tree (BST) consists of nodes storing a 7 8 single key k and having two fields, L and R. Traversals begin at n=root and proceed to the node pointed to by n.L, if n , ⊥ and k
in order to obtain snapshots in a lock free fashion. This later Algorithm 1: TraverseT (k) enables us to design a lock-free membership test. The suc- 1 Function nextField (n,k): cinct path snapshots can be seen as describing the paths that T 2 if c1(n,k) = 1 then return f1 are logically in the tree, even if due to concurrent mutations 3 ... the paths are temporarily broken. Finally, we show that the 4 if cl (n,k) = 1 then return fm PaVT condition is necessary: any validation condition that 5 return ⊥ does not check it may return incorrect results. 6 Function TraverseT (k): Preliminaries A traversal in a search tree T , TraverseT (k), 7 n ← root is the exhaustive application of an operation called 8 while n , ⊥ do nextFieldT starting from the root (Algorithm 1). The op- 9 f ← nextFieldT (n,k) eration nextField (n,k) takes a node in T and a key and T 10 if f = ⊥ then return true returns either (1) a field f indicating that if k is in T , it is 11 n ← n.f reachable from n.f or (2) ⊥ indicating that k is a key in n. 12 return false The nextFieldT (n,k) operation consists of a series of condition-field pairs [c → f ],..., [c → f ] and it returns 1 1 l l (1) For every (n ,c ) ∈ S, c (n ,k) = 1. the first field whose condition is met. A condition isafunc- i i i i (2) n ,...,n are logically in T (i.e., their marked is false). tion from a node n and a key k to a boolean value. We write i1 im (3) There is a path in T linking these nodes. c(n,k) = 1 if the condition c is met for n and k, and c(n,k) = 0 (4) This path is maximal: n .[nextField (n ,k)] = ⊥. otherwise. We say that c(n,k) (logically) implies a condition im T im (5) For every node n logically in T , either no node in S is c ′ checked on n′ and k ′ if whenever c(n,k) = 1, c ′(n′,k ′) = 1. reachable from it, or there exists a pair (n ,c ) ∈ S where The nextField operation has the property that for every i i T n is reachable from n via a field f , and c (n ,k) implies node n and key k, at most one condition is met, regardless i i i a condition c(n,k), such that [c → f ] is in nextField . of the evaluation order. We can thus refer to the condition T Then, there is no node logically in T with the key k. met in nextFieldT (n,k) (if exists). We assume that the con- ditions do not use the “or” operand. This does not affect We illustrate this by example on the BST in Fig. 1(a); a full generality since a condition of the form [(c1 or c2) → f ] proof is provided in Appendix A. Given k = 8 and the set can be represented by two distinct conditions: [c1 → f ] {(n4, > ), (n9, < )} (whose keys are 4 and 9), all conditions and [c2 → f ]. This also explains how different conditions hold. Assume in contradiction 8 is in the tree in n8. Clearly, may return the same field (but not vice-versa). A positive n8 is not n4 or n9. Also, n4 and n9 are not reachable from n8, example for this case is the nextField2D (of the 2-D tree), as this contradicts requirement (5). Lastly, n8 is not reachable which for some nodes checks the x coordinate and for some from n9, as this contradicts requirement (4). Since n4 is the nodes checks the y coordinate, although eventually either L root, this implies that n8 is not anywhere in the tree. For the or R is returned. We assume that for every node, there are general case, where no node is the root, see the proof. no two keys that meet different conditions but return the Minimal Set From this theorem, and in particular require- same field. Another property of nextFieldT is that by the ment (5), we can characterize when S is minimal: every con- construction of search trees, if a node n′ is reachable from dition appears at most once and conditions may be absent if ′ a node n via a field f , then nextFieldT (n,n .ki )= f for any they are implied by other conditions. This bounds the size of ′ key ki in n . In the following, we refer to this property as the the required S by the number of conditions in nextFieldT , search trees’ property. For example, in the BST in Fig. 1(a), which is typically linear in the number of fields. the nodes 12 and 9 are reachable from the node 4 via R and Corollary 4.2. Let T be a tree, k a key, and S a set of node- indeed nextField (4,12)= R and nextField (4,9)= R. T T condition pairs satisfying requirements (1)–(5) of Theorem 4.1. The PaVT Condition The PaVT condition determines when S is minimal if for every (n ,c ), (n ,c ) ∈ S such that j > i: it is safe to determine that a key is not in the tree. This is i i j j c (n ,k) does not imply c (n ,k). the challenging decision because determining whether a key j j i i is in the tree is done by looking for a node (logically in the From this characterization we can infer that S is unique: tree) that has this key. Our next theorem states the PaVT Lemma 4.3. Let T be a tree, k a key, and S1, S2 minimal condition. It states that given a set of node-condition pairs sets of node-condition pairs satisfying requirements (1)–(5) of that are maximal in their path w.r.t. the condition and given Theorem 4.1. Then, either S1 ⊆ S2 or S2 ⊆ S1. a key k that meets all these conditions, k is not in the tree. Proof is provided in Appendix A. Theorem 4.1 (The PaVT Condition). Given a tree T , a key k, Motivation for Succinct Path Snapshots The last corol- and a set of node-condition pairs S = {(ni1 ,ci1 ),..., (nim ,cim )}. lary and lemma uniquely define the minimal set of nodes, but If there exists a moment between the traversal’s invocation and do not explain how to construct it. Constructing it on-the-fly response where all the following hold: in a lock free fashion is possible, but may lead to reading an Practical Concurrent Traversals in Search Trees PPoPP ’18, February 24–28, 2018, Vienna, Austria inconsistent snapshot, which requires restarting the traver- S meets requirements (2)–(5) of the PaVT condition. This sal. Instead, we decouple requirement (1) of the PaVT con- simplifies the PaVT condition: dition (which pertains to the key the traversal searches for) Theorem 4.5 (The PaVT Condition via Succinct Path Snap- from requirements (2)–(5) (which pertain to the nodes on the shots). Let T be a tree, k a key, and S a succinct path snapshot traversal path), and strengthen the latter to logically capture such that for every (n ,c ) ∈ S, c (n ,k) = 1. Then, there is no the paths in the tree. We call the set of nodes satisfying the i i i i node logically in T with the key k. strengthened requirements the succinct path snapshots. The importance of strengthening the requirements is that con- Validating Succinct Path Snapshots is Necessary The current mutations can change nodes that link the snapshots’ validation of the SPS for an unsuccessful traversal (where the nodes as long as they do not affect the snapshot. For example, key k is not found) is not only sufficient but also necessary. in the BST in Fig. 1(a) and for the snapshot {(n4, > ), (n9, < )}, We show that if the tree permits replacements of nodes, we would like to allow concurrent updates to the nodes link- every validation check has to check that the snapshot meets ing them (e.g., n12), as long as n4 and n9 are on the same the conditions in Theorem 4.1. Intuitively, this follows since path and they comprise this path’s snapshot. For example, omitting any check of these conditions enables a concurrent we would like to allow concurrent updates that remove n12 thread to change the tree in that unobserved spot and add or add other nodes with keys k ∈ {10, 11,...}. We call the the key under search in a different path reachable from this set of keys that can be added to the path without affecting a spot. Thus, any weaker condition may result in returning snapshot S, the valid keys w.r.t. S. For example, 10, 11,... are incorrect traversal result (and as a result possibly incorrectly the valid keys w.r.t. {(n4, >), (n9, <)}. If a path snapshot is modifying the tree). Proof is provided in Appendix A. not affected by this kind of concurrent changes, we call itthe Lemma 4.6. Let T be a tree, k a key not in T , and S a succinct succinct path snapshot. We next provide formal definitions. path snapshot of k inT (logically capturing a path P inT ). Any Succinct Path Snapshots A path is a series of node-field validation condition that does not verify requirements (1)–(5) pairs (n0, f0),..., (nm, fm) such that: (i) n0 is the root, (ii) for of Theorem 4.1 for the nodes in S, cannot distinguish between every i ∈ [0,...,m − 1]: ni .fi = ni+1, and (iii) nm .fm = ⊥, T and another tree T ′ that contains k. i.e., no node follows nm .fm. We say that a condition c cor- responds to a field f , if when c is met in nextFieldT , f We note that if a tree does not permit addition/replacement is returned. For f = ⊥, the corresponding condition is of nodes in the middle of the tree, then some conditions of ¬c1 ∧ ... ∧ ¬cm (met when all other conditions are not met). Theorem 4.1 are implicitly guaranteed. The external tree A node-condition series (n0,c0),..., (nm,cm) corresponds to of [14] is such a tree in which nodes can only be added as a path (n0, f0),..., (nm, fm) if every ci corresponds to fi . leaves and only leaves can be removed. This guarantees: (i) Given a path P = (n0, f0),..., (nm, fm) and its snapshot the first node of the SPS (which is not checked at the endof ( ′ ) ( ′ ) the traversal) can be changed but only to one of its ancestors, S = n0,c0 ,..., nk ,ck , the valid keys w.r.t. S of a node n in P are all keys that n can have such that the nodes in S which guarantees conditions (1) and (3) and, combined with remain on the same path in the tree. the fact that the last node in the SPS is logically in the tree, also condition (2); (ii) the last node is a leaf, which guarantees Definition 4.4 (A Succinct Path Snapshot.). Let P = condition (4); and (iii) nodes cannot be added in the middle ( ) ( ) ( ′ ) ( ′ ) of the tree, which guarantees condition (5). n0, f0 ,..., nm, fm be a path and C = n0,c0 ,..., nk ,cm a node-condition series corresponding to P. S ⊆ C is a suc- cinct path snapshot (SPS) of P if: 5 Lock-free PaVT Membership Test ′ ′ (i) n0,...,nk are logically in T . In this section, we describe the PaVT membership test. (ii) There is a path in T linking these nodes which is max- PaVTTraverseT (Algorithm 2) leverages Theorem 4.5 to imple- ′ ⊥ imal: nk .fk = . ment a lock-free membership test. It begins similarly to Al- (iii) For every node n logically in T , either no node in S gorithm 1: it starts from the root (which is a sentinel node, as is reachable from it, or there exists a pair (ni ,ci ) ∈ S we shortly describe) and continues according to nextFieldT . where ni is reachable from n via some field f , such If it reaches the end of a path, it validates with the succinct that for every k, ci (ni ,k) implies c(n,k), that returns path snapshot, which is atomically read from the last node f for all valid keys w.r.t. S of n. traversed (n). Being able to read the snapshot atomically (iv) For every (ni ,ci ), (nj ,cj ) ∈ S such that j > i and for from a node, makes PaVTTraverseT lock-free. PaVTTraverseT every key k, cj (nj ,k) does not imply ci (ni ,k) for some is linearized either (i) when an unmarked node with the key valid key w.r.t. S of ni . k is found (in which case there is a moment between the invocation and response where k is logically in the tree) or The succinct path snapshots logically capture the paths in (ii) in the linearization point of the snapshot read at Line 8, the tree. In particular, the fact that a succinct path snapshot which guarantees that this snapshot logically captures the S was obtained implies that the path logically captured by suitable path for k in the tree. PPoPP ’18, February 24–28, 2018, Vienna, Austria Dana Drachsler-Cohen, Martin Vechev, and Eran Yahav
Algorithm 2: PaVT-TraverseT (k) Algorithm 3: UpdateSnaps(n)
1 n ← root ; f ← nextFieldT (n,k) 1 Function UpdateSnaps(n): 2 while n.f , ⊥ do 2 UpdateSnaps(n, n, n.P, {f1,..., fl }) ′ 3 n ← n.f 3 Function UpdateSnaps(n,n ,p, f Set): 4 f ← nextFieldT (n,k) 4 if f Set = ∅ then return ′ 5 if f = ⊥ then 5 f ←the field such that p.f = n ′ 6 if n.marked then restart 6 c ←the condition such that c(p,n .k1) = 1 7 return true 7 if k.c(n,k) = 0 then return ∀ p 8 Atomically change (p,c) to (n,c) in S 8 S ← snapshot(n, f ) f p ′ ′ ′ ′ 9 n ← ← \{ } 9 if (n ,c ) ∈ S.c (n ,k) = 0 then restart Sf Sf ; f Set f Set f ∃ 10 return false 10 UpdateSnaps(n, p, p.P, f Set)
To employ this traversal as part of an update (e.g., insertion Synchronization and Updates To keep the snapshots in or removal), we modify it to acquire the lock of n, which the nodes correct, we require operations that update paths serves also as the lock of the snapshot, just before validating to lock their snapshots. This is obtained by extending each and returning n. We call this variation PaVTTraverseT NLock. node with a lock and associating the snapshot’s lock to the In the remainder of this section, we explain how to maintain lock of its last node (i.e., the leaf). In general, this means that the succinct path snapshots (required at Algorithm 2, Line 8) updating a single path requires one lock. However, if the last to guarantee conditions (2), (3) and (5) of Theorem 4.1. This node of the snapshot is being removed and is replaced by means that a traversal only has to check for conditions (1) a node n, then it means that the snapshot changes its lock and (4) of Theorem 4.1. to n’s lock. Thus, n’s lock also has to be acquired before the Storing SPSs Although snapshots are (supposed to be) read update begins, to guarantee that throughout the operation, only from leaves, it simplifies their maintenance if every changes to the path are blocked. node contains all snapshots containing it. That is, snapshots After executing insertion, removal, or any other mutation, are stored in every node they consist of. More precisely, for we potentially need to fix the snapshots of the mutated nodes every node n and field f , we store a set of succinct path and their descendants. This is done by invoking UpdateSnaps Sn snapshots containing n.f , denoted by f . We begin with (Algorithm 3) on the mutated nodes. Updating the snapshots characterizing the sizes of the snapshot sets for different is done by traversing from the mutated node up the tree to trees. Proof is provided in Appendix A. fix the snapshots by overriding the old node-condition pairs. Traversing up the tree is possible due to the parent pointer Lemma 5.1. 1. In BSTs, internal or external, where the that each node has, which is denoted by n.P for a node n ≥ |Sn | ≤ conditions are < for L and > or for R, f 1 for (as mentioned in Section 2). When a node is removed, either every node n and field f . by replacing it with another node or setting its parent to 2. In internal m-ary trees, where the conditions are k < point to ⊥ instead of this node, the snapshot of the removed |Sn | ≤ n.k1, n.k1 < k < n.k2,..., n.km−1 > k, f 2 for every node is copied to the replacing-node/parent (which has to node n and field f . be locked as well, as it is being written to) and the replacing- |Sn | ≤ K−1 3. In K-D trees, f 2 for every node n and field f . node/parent looks for a replacement for the removed node 4. In tries, where nodes contain the complete prefix (i.e., the in its snapshots. We omit this from the code for simplicity. |Sn | ≤ Note that since the path snapshot is already locked, keys are not only on the edges), f 1 for every node n and field f . UpdateSnaps does not require further synchronization. We also note that for some trees, the snapshot updates can be We now describe how to maintain the snapshots to reflect optimized. For example, in BSTs, the updated snapshots can the logical paths in the tree. be computed directly from the snapshots of the mutated Sentinels To avoid edge cases, we make sure all snapshots nodes or their parents. are of the same size by assuming the tree has a sentinel Correctness The correctness of UpdateSnaps follows from node per condition. Namely, if [c1 → f1],..., [cl → fl ] are the fact that while a path is being updated, concurrent up- the condition-field series of nextFieldT , then there are l dates are blocked. Thus, the updates of the snapshot are S S S S sentinel nodes, n1 ,...,nl , such that ni .fi = ni+1. Since every guaranteed to reflect those of the path. Formally, this argu- key k has to be reachable from all sentinels, traversals may ment can be proven by induction. We omit this proof. S S begin from nl . Thus, we can set the tree root to be nl . Upon Linearization Although snapshots are not updated atomi- S cally with mutations to the tree, they always represent a con- creating the tree, the last sentinel node nl has one snapshot {{( S ) ( S )}} sistent (logical) path in a tree, which enables us to linearize for cl : n1 ,c1 ,..., nl ,cl . The other sentinels do not have a snapshot. them (assuming that the tree operations invoke UpdateSnaps Practical Concurrent Traversals in Search Trees PPoPP ’18, February 24–28, 2018, Vienna, Austria on all mutated nodes). To linearize snapshots, we determine PaVTTraverseNLock returns n (which is now locked). After a point where the snapshot is an SPS of a path in the tree. PaVTTraverseNLock completes, other nodes that are read or The simple case to linearize a snapshot is at the moment of written to are locked as well (with respect to the locking n reading its pointer (i.e., Sf ). However, since snapshots are order). Then, updates are executed. Finally, the snapshots not updated atomically with mutations to the tree (e.g., inser- are updated as described in the previous section, locks are tions), they may be linearized with respect to insertions or released, and the operation result is returned. removals (like the linearization point of an unsuccessful con- Optimizing the Snapshots As discussed in Section 3, in tains in [19]). Namely, if a snapshot contains a node which BSTs, snapshots are of size two (they are predecessor- has been removed, that snapshot is linearized just before the successor pairs). Thus, instead of maintaining the complete linearization point of the removal, and if a snapshot does not snapshot, nodes can store only the other nodes in their snap- contain a node whose insertion extends its path, the snap- shots and infer the condition of that node. For example, in shot is linearized just before the insertion. Intuitively, these the internal BST in Fig. 1(a), the node 9 can store for L the linearization points are correct because if a thread A reads snapshot: 4. When reading this snapshot, one can infer that a snapshot S, then S logically represents a path that existed the snapshot contains (9, <) and (4, >), since there are only in the tree after A has started this operation. Thus, even if two conditions, and the node corresponding to L is 9. in the meantime, a thread B changed the path represented We next provide the implementation details of insertion by S, the fact that S has not been updated yet implies that and removal, which extend the sequential operations with B has not completed its operation. Thus, we can linearize S the PaVT traversal (contains(k) is exactly Algorithm 2). To just before the linearization point of B’s operation. demonstrate the similarity of the PaVT-ed operations to the Complexity The complexity of UpdateSnaps is O(h) where sequential operations, we mark with rectangles statements h is height of the tree. This follows because UpdateSnaps added on top of the sequential implementation: the pink, updates at most a single snapshot at every node, and the wider rectangles are the PaVT operations, while the gray, new snapshot is copied to nodes by copying a pointer to it shorter rectangles are the lock acquisitions. (i.e., nodes of the snapshots are not copied individually). The Insertion The BST insert operation inserts a key k if k number of such updates depends on the update operation, is not in the tree. It begins with traversing from the tree but is typically a constant. In Section 6, we show how to root, turning left or right depending on whether the cur- optimize UpdateSnaps to reduce the complexity of updating rent node’s key is greater or smaller than k, and terminating a single snapshot to O(l) where l is the size of the snapshot. when k is found or the path has ended. If k is found in a node n, it returns false, otherwise it adds a new node af- 6 PaVT-ing Updates of BSTs and AVLs ter the last node read. The PaVT-BSTInsert(k) (Algorithm 4) is very similar to the BST insert and the differences are In this section, we demonstrate how to employ the PaVT tra- marked with rectangles. The differences involving PaVT are versal with a simple locking protocol to implement arbitrary PaVTTraverseNLock and UpdateSnaps, while the differences BSTs, internal and external, and an AVL tree. involving locks are: (i) acquisition of the lock of the last node Locking Protocol We employ the following locking protocol. traversed in PaVTTraverseNLock, and (ii) release of locks upon Each node has a lock and n1’s lock is acquired before n2’s if completion or inside PaVTTraverseNLock if validation fails. (i) n2 is reachable from n1 or (ii) n1 and n2 are reachable via fi Another difference compared to the sequential insertion is and fj , respectively, from their lowest common ancestor and that after locking n (that is, n’s snapshot suits k), we validate i < j. If a node n has to be locked against the locking order that n is the last node on the path; if not, we restart, since or it may be the case, an opportunistic attempt to acquire this means that n’s lock does not block concurrent updates n’s lock is taken (without blocking), and if it fails, locks to its snapshot. The linearization point of an insertion is at are released and the operation restarts. To guarantee that the moment the new node becomes reachable. restarts do not result in contaminated data, updates begin Removal The BST removal operation is PaVT-ed similarly, only after all required locks are acquired. This protocol is and its unique aspect is that it contains a nested traversal in deadlock-free and livelock-free because locks are always case the node to remove has two children, in which case a acquired in the same order by all threads, and the thread that nested traversal is required to look for the successor, starting acquires the locks of the lowest nodes (according to the tree) from the node’s right child (Line 22). is guaranteed to succeed in acquiring the locks. The BST remove operation removes a key k if k is in the PaVT-ing Operations We perform PaVT insertions and re- tree. It begins with traversing from the tree root, turning movals as follows. We invoke PaVTTraverseNLock, which is left or right depending on whether the current node’s key is identical to Algorithm 2 except that right before Lines 7 and greater or smaller than k, and terminating when k is found 8, it acquires the lock of the last node traversed, n. Then, if or the path has ended. If k is found in a node n, the removal the validation of Line 9 fails, the lock is released and the op- is one of the following: eration restarts. Otherwise, instead of returning true/false, PPoPP ’18, February 24–28, 2018, Vienna, Austria Dana Drachsler-Cohen, Martin Vechev, and Eran Yahav
Algorithm 4: PaVTBST-Insert(k) Algorithm 5: BST-PaVTRemove(k) 1 n ← PaVTTraverseNLock(k) 1 while true do 2 if n.k=k ∥ (k > n.k && n.R , ⊥) ∥ (k < n.k && n.L , ⊥) then 2 n ← PaVTTraverseNLock(k) 3 unlock(n) 3 if n.k , k then 4 return false 4 unlock(n) 5 return false 5 newNode ← new node(k) 6 newNode.P ← n 6 p ← n.P; l ← n.L; r ← n.R 7 (k > n.k? n.R : n.L) ← newNode 7 if !lock(p,l,r) then 8 UpdateSnaps(newNode) unlockAll(); continue 9 unlock(n) 8 if l = ⊥ || r = ⊥ then 10 return true 9 n.marked ← true 10 c ← l = ⊥? r : l • If n has at most one child, n’s parent is updated to 11 (p.L = n? p.L : p.R) ←c point to this child. 12 UpdateSnaps(p, c) • If n has two children and its right child r does not have 13 unlockAll() a left child (i.e., r is n’s successor), n’s parent is set to 14 return true r, and r’s left child is set to n’s left child. 15 if r.L = ⊥ then • If n has two children and its right child has a left child, 16 n.marked ← true n’s successor, s, is obtained by looking for the leftmost 17 r.L ← l; l.P ← r; r.P ← p node in the tree rooted with r. Then, s relocates to n’s 18 (p.L = n? p.L : p.R) ←r location to remove n. 19 UpdateSnaps(p, r) The PaVT-BSTRemove (Algorithm 5) is very similar to the 20 unlockAll() BST removal and the differences are marked with rectangles. 21 return true The differences involving PaVT are: (i) PaVTTraverseNLock 22 s ←PaVTTraverseNLock(r,k) // If failed is invoked to find n and to find s if n has two children, and locking s, release locks and restart (ii) UpdateSnaps. The differences involving locks are: if !lock(s.P,s.R) then • Locks are acquired on the last node traversed in 23 unlockAll(); break PaVTTraverseNLock. 24 • Additional locks are acquired on all nodes the removal n.marked ← true 25 sP ← s.R; sR ← s.R involves (i.e., n’s parent, n’s children, and s’s parent 26 s.R← r; r.P← s and right child). In case a lock is acquired against 27 s.L← l; l.P← s the locking order (e.g., n’s parent is locked after n is 28 s.P ← p locked), an optimistic attempt to lock it is taken, and if 29 (p.L = n? p.L : p.R) ←s it fails all locks are released and the operation restarts. 30 sP.L← s.R • Locks are released as described for the insertion. 31 if sR , ⊥ then sR.P= sP • n’s mark flag is set right after all locks are acquired. 32 UpdateSnaps(sR,sP,s,p) The linearization point is when n is marked as removed. 33 unlockAll() insert(k) Internal AVL The AVL extends the BST’s and 34 return true remove(k) with a rebalance operation that restores the tree after updates, if the updates have violated the AVL invari- ants. The rebalance operation traverses the tree bottom up inner node. Also, synchronization can be more efficient if from the modified nodes and may rotate nodes, i.e., relocate we limit the locks to the inner nodes. This is correct since adjacent nodes. Since the snapshots of m-ary trees consist of leaves cannot be updated without modifying their parent predecessor-successor pairs, AVL rotations do not affect the (insertion modifies the parent to point to the new inner node snapshots in internal BSTs. Thus, rotations are implemented and removal triggers the removal of the parent). Thus, leaves by traversing the mutated path upwards, acquiring the nodes’ are synchronized indirectly through their parents and a leaf locks, and rotating the tree if necessary. This guarantees that is indirectly marked as removed by marking its parent. at a quiescence state the tree is a valid AVL tree [5]. External BST External trees may be PaVT-ed similarly to an internal BST, but can be further optimized because inner 7 Evaluation nodes do not have payloads and leaves do not have children. We evaluated the PaVT-ed internal BST, external BST, and in- This enables us to condense the node structure by using ternal AVL. To evaluate PaVT, we compare to state-of-the-art the same field to denote a payload of a leaf or a child ofan algorithms that have online available Java implementations. Practical Concurrent Traversals in Search Trees PPoPP ’18, February 24–28, 2018, Vienna, Austria
90%C–9%I–1%R 70%C–20%I–10%R 0%C–50%I–50%R 6 10 · 2 ops/sec 6 10 Range: 5 10 · 2 ops/sec 6 10 Range: 6 10 · 2 ops/sec 6 10 Range: 5 10 · 2 ops/sec 6 10 Range:
Number of Threads Number of Threads Number of Threads
Figure 2. Comparison of binary search tree and AVL implementations.
We compare the (internal) PaVT-BST and External-PaVT- To fairly compare to BCCO-AVL, which in heavy update BST to: (i) LO-BST: a lock based, internal BST [12] and benchmarks easily revives logically removed nodes and (ii) EFRB-BST: a lock-free, external BST [14]. avoids rebalancing, we compare to a partially external AVL- We compare the (internal) PaVT-AVL to: PaVT. We note that non-blocking data structures provide • LO-AVL: a lock based, internal AVL [12]. stronger progress guarantees than lock-based, which re- • Chromatic RB Tree: a lock-free, external red-black tree, quire additional overhead in practice. Thus, comparing non- using the LLX/SCX primitives, optimized for updates: blocking algorithms to blocking algorithms may seem like a path is fixed when there are six violations [8]. comparing apples to oranges. However, since sometimes they • BCCO-AVL: a lock based, partially-external AVL [7], were shown to obtain better performance than blocking al- where internal nodes initially store data (keys) but gorithms [8], we compare to these structures as well. The may become routing nodes if their key is removed, by comparison of internal trees to external trees demonstrates logically marking them. These nodes may become data the tradeoff between contention and memory consumption. nodes again if their key is re-added to the tree. Experiments ran on an AMD Opteron Processor 6376 with 128GB RAM and 64 cores: 4 processors with 16 cores each PPoPP ’18, February 24–28, 2018, Vienna, Austria Dana Drachsler-Cohen, Martin Vechev, and Eran Yahav and with hyper-threading support. OS was Ubuntu 14.04 LTS and Java was SE Runtime (build 1.8.0_45-b14) with Java 90C,9I,1R 70C,20I,10R 0C,50I,50C HotSpot 64-Bit Server VM (build 25.45-b02, mixed mode). PaVT-BST 104,132 140,523 78,111 Our experiments follow the standard empirical evalua- External- 175,021 223,410 131,263 tion (e.g., [7, 8, 12]). We ran five-second trials and we report PaVT-BST the throughput (number of operations per second). In every EFRB-BST 148,118 193,623 111,236 trial, threads randomly chose the operation type accord- LO-BST 239,563 323,149 179,733 ing to a given workload distribution and then randomly PaVT-AVL 104,177 140,612 78,144 chose the key from a given range. The workloads were: Partially- (i) 90% contains, 9% insert, 1% remove, (ii) 70% contains, External- 118,454 145,631 97,101 20% insert, 10% remove, and (iii) 0% contains, 50% insert, PaVT-AVL 5 6 50% remove. The ranges were 2 · 10 and 2 · 10 . Before each BCCO-AVL 115,677 145,301 91,041 trial, the tree was prefilled to its expected size: 90% full in Chromatic 188,375 237,524 142,018 workload (i), 2/3 full in workload (ii), and 50% full in work- RB Tree load (iii). Every experiment was run 10 times and the arith- LO-AVL 249,999 337,455 187,544 metic average is reported. Every 10 trial batch was run in Table 1. Space consumption in kB for 64 threads, range 2·106. its own JVM instance and a warm-up phase was run before to avoid HotSpot effects. Threads were not bound to pro- cessors. The number of threads was varied over the range them results in poor scalability since they acquire locks on 1, 2, 4, 8, 16, 32, 64, 128, 132, 134, 144. every node along the traversal path. Universal constructions Experimental results (Fig. 2) indicate that the PaVT con- are either practical only for data structures with few con- dition is effective and improves performance over current tention points, such as stacks and queues (e.g., [16, 20, 21]), approaches: PaVT-BST on average outperforms the internal or provide only general guidelines (e.g., [10]). LO-BST on average by 17% and up to 48%, and PaVT-AVL out- Brown et al. [8] introduce the LLX/SCX primitives, which performs the LO-AVL on average by 22% and up to 60%. The are a multi-word generalization of the LL/SC primitives. improvement of PaVT-BST and PaVT-AVL over LO-BST and However, programmers have to integrate them manually. LO-AVL stems from the fact that the LO trees acquire more Further, LLX/SCX is a lock-free technique and thus updates locks (as for each predecessor-successor pair there is an addi- take place by first preparing a new modified subtree and then tional lock). External-PaVT-BST and EFRB-BST mostly have atomically changing the pointer to the old subtree to point similar results (on average EFRB-BST is better by 5%). We re- to the new one. Thus, when multiple nodes are modified mind that the EFRB-BST is lock-free and thus has additional (e.g., in a balancing operation), all, but their lowest common overhead compared to the lock-based External-PaVT-BST. ancestor, are reallocated, which affects performance. In12 [ ], The advantage of lock-free algorithms is usually observant a simple validation condition is proposed for BST, suitable under heavy contention (e.g., when the number of threads for data structures implementing totally ordered sets. The is bigger than the number of cores). The Partially-External- validation check validates a search for a key k by checking PaVT-AVL outperforms the partially external BCCO-AVL on k’s predecessor and successor in the tree, which is done effi- average by 13% and up to 65%. This can be attributed to the ciently by maintaining the nodes’ predecessor and successor. lock-free membership test that Partially-External-PaVT-AVL This was shown for internal trees, where the predecessor- has, in contrast to BCCO-AVL. successor pairs always share a path. However, in external Lastly, we provide results on the space consumption. Ta- trees, such approach would link the leaves, which do not ble 1 shows the results for all workloads, with 64 threads and share a path, and thus introduce redundant conflicts and con- key range 2·106 (results for the other scenarios were similar). tention. DomLock [23] presents a new locking protocol for Results show that the PaVT implementations provide the hierarchal data structures where nodes’ keys are contained lowest memory consumption, which indicates that explicitly in their parents’ keys (an example for such tree is one whose maintaining the SPSs in the nodes has low overhead. nodes have intervals for keys and keys of descendant nodes are contained intervals). However, since trees are not always 8 Related Work hierarchical, this protocol is inapplicable in our setting. Current solutions addressing the challenge of validating tra- Lastly, there are many practical implementations of trees versals range from standard locking protocols and univer- (e.g., [7]), with most of them supporting lock-free searches sal constructions, through validation conditions for sets, to (e.g., [1, 9, 10, 13]), and some offer complete lock-free im- highly-optimized solutions for specific tree algorithms. plementations (e.g., [6, 8, 18, 22, 25]), which typically cope General solutions such as the standard locking protocols better under heavy contention but less when updates are (two-phase locking [15], hand-over-hand locking [3], tree sparse and searches are dominant [17]. Unfortunately, these locking [24]) address this challenge, however employing support only standard operations and extending them to Practical Concurrent Traversals in Search Trees PPoPP ’18, February 24–28, 2018, Vienna, Austria other operations or trees is challenging. In contrast, we pro- [20] Danny Hendler, Itai Incze, Nir Shavit, and Moran Tzafrir. 2010. Scalable vide a general condition and a lock-free membership test Flat-Combining Based Synchronous Queues. In Distributed Computing. and demonstrate the simplicity of integrating our traversal LNCS, Vol. 6343. [21] Maurice Herlihy. 1991. Wait-free Synchronization. ACM Trans. Pro- and validation condition in update operations. gram. Lang. Syst. 13 (1991). [22] Shane V. Howley and Jeremy Jones. 2012. A non-blocking internal 9 Conclusion binary search tree. In SPAA ’12. We presented PaVT, a necessary and sufficient condition for [23] Saurabh Kalikar and Rupesh Nasre. 2016. DomLock: A New Multi- granularity Locking Technique for Hierarchies. In Proceedings of the validating concurrent traversals in search trees. PaVT relies 21st ACM SIGPLAN Symposium on Principles and Practice of Parallel on succinct path snapshots to validate traversal paths. Based Programming (PPoPP ’16). on this condition, we designed a lock-free membership test. [24] Zvi Kedem and Abraham Silberschatz. 1981. A characterization of We also showed how to leverage the traversal and validation database graphs admitting a simple locking protocol. Acta Informatica of the membership test to implement binary search trees, 16, 1 (1981). [25] A. Natarajan and N. Mittal. 2014. Fast Concurrent Lock-Free Binary internal and external, and AVL, and experimentally showed Search Trees. In PPoPP ’14. they outperform state-of-the-art trees. [26] Nir Shavit. 2011. Data structures in the multicore age. Commun. ACM 54, 3 (2011). References [27] V. Luchangco Y. Lev, M. Herlihy and N. Shavit. 2006. A Provably [1] Maya Arbel and Hagit Attiya. 2014. Concurrent updates with RCU: Correct Scalable Skiplist. In OPODIS ’06. search tree as an example. In PODC ’14. [2] Maya Arbel and Adam Morrison. 2015. Predicate RCU: an RCU for A Proofs scalable concurrent updates. In PPoPP ’15. [3] R. Bayer and M. Schkolnick. 1977. Concurrency of operations on In this section, we provide all the proofs. B-trees. Acta Informatica 9, 1 (1977). Proof (Sketch) of Theorem 4.1. Assume in contradiction that [4] Michael A. Bender, Jeremy T. Fineman, Seth Gilbert, and Bradley C. Kuszmaul. 2005. Concurrent cache-oblivious b-trees. In SPAA ’05. there is a node n logically in T with the key k. That node is [5] Luc Bougé, Joaquim Gabarró, Xavier Messeguer, and Nicolas Scha- not any of ni1 ,...,nim as otherwise it would not be true that for banel. 1998. Height-relaxed AVL rebalancing: A unified, fine-grained every (ni ,ci ) ∈ S, ci (ni ,k) = 1 (since nextFieldT (ni ,k) has approach to concurrent dictionaries. (1998). to return ⊥ in this case). Also, there is no n′ in S such that n′ [6] Anastasia Braginsky and Erez Petrank. 2012. A lock-free B+tree. In is reachable from n, since otherwise there is a pair (ni ,ci ) ∈ S SPAA ’12. ( ) ( ) [7] Nathan G. Bronson, Jared Casper, Hassan Chafi, and Kunle Olukotun. such that ni is reachable from n, ci ni ,k = 1, and ci ni ,k 2010. A practical concurrent binary search tree. In PPoPP ’10. logically implies a condition c(n,k) in nextFieldT (n,k). [8] Trevor Brown, Faith Ellen, and Eric Ruppert. 2014. A General Tech- Thus, nextFieldT (n,k), ⊥ and thus k is not in n. Also, n nique for Non-blocking Trees. In PPoPP ’14. cannot be reachable from all nodes in S, since the last node [9] Tyler Crain, Vincent Gramoli, and Michel Raynal. 2013. A Contention- is maximal: ni .[nextFieldT (ni ,k)]=⊥, indicating no node Friendly Binary Search Tree. In Euro-Par ’13. m m [10] Tudor David, Rachid Guerraoui, and Vasileios Trigonakis. 2015. Asyn- follows nim . Thus, n is not reachable from any of ni1 ,...,nim chronized Concurrency: The Secret to Scaling Concurrent Search Data and ni1 ,...,nim are not reachable from n. Consider the lowest ′ Structures. In ASPLOS ’15. common ancestor of ni1 and n, denoted by n , and assume n is [11] Simon Doherty, David L. Detlefs, Lindsay Groves, Christine H. Flood, ′ ′ reachable from n via f and ni1 via f . On the one hand, since Victor Luchangco, Paul A. Martin, Mark Moir, Nir Shavit, and Guy L. ′ ′ ni1 is reachable from n via f , there exists a pair (ni ,ci ) ∈ S Steele, Jr. 2004. DCAS is Not a Silver Bullet for Nonblocking Algorithm ′ such that ni is reachable from n and nextField(ni ,k) logi- Design. In SPAA ’04. ′ [12] Dana Drachsler, Martin Vechev, and Eran Yahav. 2014. Practical Con- cally implies some c(n,k) such that [c → f ] is in nextFieldT . ′ current Binary Search Trees via Logical Ordering. In PPoPP ’14. Thus, nextFieldT (n,k)=f . On the other hand, since k is in ′ [13] Dana Drachsler-Cohen and Erez Petrank. 2014. LCD: Local Combining n which is reachable from n via f , nextFieldT (n’,k)= f on Demand. In OPODIS ’14. by the search trees’ property. Since f , f ′, two different [14] Faith Ellen, Panagiota Fatourou, Eric Ruppert, and Franck van Breugel. nextField (n’,k) 2010. Non-blocking binary search trees. In PODC ’10. conditions are met in T – contradiction. [15] K. P. Eswaran, J. N. Gray, R. A. Lorie, and I. L. Traiger. 1976. The Proof (Sketch) of Lemma 4.3. Assume there are two minimal Notions of Consistency and Predicate Locks in a Database System. sets S1 , S2. Then, w.l.o.g. there exists (n,c) ∈ S1 \ S2. Since Commun. ACM 19, 11 (1976). ′ ′ ′ ′ [16] Panagiota Fatourou and Nikolaos D. Kallimanis. 2011. A Highly- (n,c) is not in S2, there exists (n ,c ) ∈ S2 such that c (n ,k) ′ ′ efficient Wait-free Universal Construction. In SPAA ’11. logically implies c(n,k). However, since (n,c) is in S1, (n ,c ) [17] Vincent Gramoli. 2015. More Than You Ever Wanted to Know About is not in S1, as otherwise S1 would not be minimal. Thus, Synchronization: Synchrobench, Measuring the Impact of the Syn- ′′ ′′ ′′ ′′ there exists (n ,c ) ∈ S1 such that c (n ,k) logically implies chronization on Concurrent Algorithms. In PPoPP 2015. ′( ′ ) ′′( ′′ ) ( ) [18] Timothy L. Harris. 2001. A Pragmatic Implementation of Non-blocking c n ,k . By transitivity, c n ,k logically implies c n,k , Linked-Lists. In DISC ’01. namely S1 is not minimal – contradiction. [19] Steve Heller, Maurice Herlihy, Victor Luchangco, Mark Moir, Bill Scherer, and Nir Shavit. 2005. A Lazy Concurrent List-based Set Algo- Proof (Sketch) of Lemma 4.6. Let v be a validation condition rithm. In OPODIS ’05. that does not check all requirements (1)–(5) of Theorem 4.1 PPoPP ’18, February 24–28, 2018, Vienna, Austria Dana Drachsler-Cohen, Martin Vechev, and Eran Yahav for the snapshot of k in T , Sk . We show for every check that without checking it, v cannot distinguish between T and a Proof (Sketch) of Lemma 5.1. 1. Assume in contradiction tree containing k. This implies that v is insufficient. |Sn | there is n and f such that f > 1 and assume w.l.o.g (1) If there is (ni ,ci ) ∈ S, for which it is not checked that that f = L. If |Sn | > 1, there are (at least) two paths ci (ni ,k) = 1. Since (ni ,ci ) ∈ S, for every j > i, cj (nj ,k) f ( , <) . ⊥ does not logically imply ci (ni ,k) for any valid key w.r.t. that contain n in their snapshots. Thus, n L , S of ni . In particular, the conjunction ci+1(ni+1,k) ∧ (as leaves are part of one path). Thus, there are two paths for which n1.R = n2.R = ⊥ and their snapshots ...∧cim (nim ,k) does not logically implyci (ni ,k) for any ( , <), ( , >) ( , <), ( , >) valid keys w.r.t. to S of ni . However, this conjunction are n n1 and n n2 . Consider nlca, the is satisfiable (e.g., by the nodes in S). Thus, there exists lowest common ancestor of n1 and n2. We split to cases: ′ • a valid key k for ni for which ci (ni ,k) = 0. Consider If n is reachable from nlca, then w.l.o.g. n1 is reach- a tree T ′ identical to T . Now let a thread A traverse T ′ able from n2. If n2 is reachable from n1.R, then and stop just before executing v. Then, another thread n1.R , ⊥ – contradiction. Otherwise, n2 is reachable ′ . < ( , ) B replaces ni with a node containing the key k . Now from n1 L. Thus, the condition n1 k is not logi- ′ ( , <) ( , >) ( , <), ( , >) B adds k to T . Since ci (ni ,k) = 0, k is added either to cally implied by n or n2 . Thus, n n2 ni or through a different field than the one returned by is not a valid snapshot – contradiction. • ci . Now A resumes and executes v. However, v cannot If nlca is reachable from n, then w.l.o.g. n1 is reach- distinguish between T and T ′. able from nlca via L. Thus, the condition < (nlca,k) ( , <) ( , >) (2) If there is (ni ,ci ) ∈ S, for which it is not checked that is not logically implied by n or n1 . Thus, n is logically T . Consider a tree T ′ identical to T but (n, <), (n1, >) is not a valid snapshot – contradiction. with a node nk containing k linked to the last node in 2. Succinct path snapshots of internal m-ary trees con- S (note that the search trees’ property is met). Now sist of nodes whose keys are each other predecessor- let a thread A traverse T ′ and stop just before reading successor in the tree. This follows from the require- nk . Then, another thread B replaces n with a node ment that the path snapshots contain node-condition containing the key k ′ (as described in the previous pairs that imply the former conditions on the path. bullet), connects to it nk (as described in the previous This means that snapshots are either a single node- ( , . < < . ) bullet), unlinks nk from its old position, and marks n condition pair of the form n n ki k n ki+1 or as logically removed. Now A resumes and executes v. two node-condition pairs, where at least one is < n.k1 However, v cannot distinguish between T and T ′. or < n.km−1. Since every key has at most one prede- (3) If it is not checked whether there is a path between cessor and one successor, the outer fields (< n.k1 and < . two nodes ni and ni+1 in the succinct path snapshot. n km−1) are part of exactly one snapshot (where Consider n, the parent of ni in T , linking to it via f they are the successor and predecessor, resp.), and the ′ . < < . (n.f = ni ). Let T be a tree identical to T but with a inner fieldsn ( ki k n ki+1) are part of at most two node nk containing k linked to the last node in S. Let snapshots (one where ki is the predecessor and one a thread A traverse T ′ and stop just before reading where ki+1 is the successor). ′ nk . Then, another thread B replaces ni with a node n 3. By a similar argument, the succinct path snapshots con- with k ′. The key k ′ is obtained similarly to bullet 1, but sist of predecessor-successor pairs in every dimension if there are multiple possible keys satisfying the con- (i.e., snapshots are of size 2K). Since dimensions are in- ′ ′ junction but not ci (ni ,k), k is selected to be such that dependent, if n is reachable from n , and n’s condition ′ ′ ′ connecting n .f = ni meets the search trees’ proper- depends on a dimension a whereas n ’s condition de- ′ ′ ′ ties, for f , f (if there is no such k , ni and ni+1 must pends on a dimension b, n can be part of the snapshots be linked in a path and thus this check is implicitly of both n.L and n.R. That is, a predecessor-successor verified). Then, B connects nk to ni and unlinks nk pair of dimension k1 can be part of every path depend- from its old position. Finally, A resumes and executes ing on the other dimensions: it can be part of a path v; however, v cannot distinguish between T and T ′. continuing to L or to R on dimension k2, then on a path (4) If the path is not maximal. Consider a tree that right continuing both to L or R from both these nodes, etc., up to the kK dimension. Since the number of different after nim .fim is linked to a node containing k. Note that this tree respects the search trees’ property, but v paths containing a given predecessor-successor is at K−1 cannot distinguish between T and this tree. most 2 , we get the bound. (5) If there is a node n logically in T , such that there 4. Follows because if every node contains the prefix, its is a node in S reachable from n and there is no pair condition logically implies all preceding nodes on its (ni ,ci ) ∈ S where ni is reachable from n via some field path. Thus, snapshots consist of a single node, which |Sn | ≤ f , and ci (ni ,k) logically implies a condition c(n,k), is the last node on the path and thus f 1. such that [c → f ] is in nextFieldT : similar to bullet 1.