1 Quadboost: A Scalable Concurrent Quadtree
Keren Zhou, Guangming Tan, Wei Zhou
Abstract—Building concurrent spatial trees is more complicated than binary search trees since a space hierarchy should be preserved during modifications. We present a non-blocking quadtree–quadboost–that supports concurrent insert, remove, move, and contain operations. To increase its concurrency, we propose a decoupling approach that separates physical adjustment from logical removal within the remove operation. In addition, we design a continuous find mechanism to reduce its search cost. The move operation combines the searches for different keys together and modifies different positions with atomicity. The experimental results show that quadboost scales well on a multi-core system with 32 hardware threads. More than that, it outperforms existing concurrent trees in retrieving two-dimensional keys with up to 109% improvement when the number of threads is large. The move operation proved to perform better than the best-known algorithm, with up to 47%.
Index Terms—Concurrent Data Structures, Quadtree, Continuous Find, Decoupling, LCA !
1 INTRODUCTION
ULTI-CORE processors are currently the universal we can adjust existing concurrent techniques from BSTs. M computing engine in computer systems. Therefore, it Therefore, we will refer to region quadtree as quadtree in is urgent to develop data structures that provide an efficient the following context. and scalable multi-thread execution. At present, concurrent In this paper, we design a non-blocking quadtree, re- data structures [1], such as stacks, linked-lists, and queues ferred to as quadboost, that supports concurrent insert, re- have been extensively investigated. As a fundamental build- move, contain, and move operations. Our key contributions ing block of many parallel programs, these concurrent data are as follows: structures provide significant performance benefits [2], [3]. Recently, research on concurrent trees is focused on • We propose the first non-blocking quadtree. It binary search trees (BSTs) [4], [5], [6], [7], [8], [9], [10], records traversal paths, compresses Empty nodes if which are at the heart of many tree-based algorithms. The necessary, adopts a decoupling technique to increase concurrent paradigms of BSTs were extended to design the concurrency, and devises a continuous find mech- concurrent spatial trees like R-Tree [11], [12]. However, there anism to reduce the cost of retries induced by CAS remains another unaddressed spatial tree–quadtree, which failures. is widely used in applications for multi-dimensional data. • We design a lowest common ancestor (LCA) based For instance, spatial databases, like PostGIS [13], adopt oc- move operation, which traverses a common path for tree, a three-dimensional variant of quadtree, to build spatial two different keys and modifies two distinct nodes indexes. Video games apply quadtree to handle collision with atomicity. detection [14]. In image processing [15], quadtree is used • We prove the correctness of quadboost algorithms to decompose pictures into separate regions. and evaluate them on a multi-core system. The ex- There are different categories of quadtree according to periments demonstrate that quadboost is highly effi- the type of data a node represents, where two major types cient for concurrent updating at different contention are region quadtree and point quadtree [16]. Point quadtree levels. arXiv:1607.03292v1 [cs.DS] 12 Jul 2016 stores actual keys in each node, yet it is hard to design The rest of this paper is organized as follows. In Section concurrent algorithms for point quadtree since an insert 2, we overview some basic operations. Section 3 describes operation might involve re-balance issues, and a remove a simple CAS quadtree to motivate this work. Section 4 operation needs to re-insert the whole subtree under a provides detailed algorithms for quadboost. We provide a removed node. The region quadtree divides a given region sketch of correctness proof in Section 5. Experimental results into several sub-regions, where internal nodes represent are discussed in Section 6. Section 7 summarizes related regions and leaf nodes store actual keys. Our work focuses works. Section 8 concludes the paper. on region quadtree for two reasons: (1) The shape of region quadtree is independent of insert/remove operations’ order, which allow us to avoid complex re-balance rules and devise 2 PRELIMINARY specific concurrent techniques for it. (2) Further, region quadtree could be regarded as a typical external tree that Quadtree can be considered as a dictionary for retriev- ing two-dimensional keys, where
The right picture is a mapping of the quadtree on a two- 1) insert(key, value) adds a node that contains key and dimensional region, where keys are located according to value into a quadtree. their coordinates, and regions are divided by their corre- 2) remove(key) deletes an existing node with key from a sponding width (w) and height (h). There are three types of quadtree. nodes in the quadtree, which represent different regions in 3) contain(key) checks whether key is in a quadtree. the right figure. Internal nodes are circles on the left figure, 4) move(oldKey, newKey) replaces an existing node with and each of them has four children which indicate four oldKey and value by a node with newKey and value equal sub-regions on different directions. The root node is that is not in a quadtree. an Internal node, and it is the largest region. Leaf nodes and Empty nodes are located at the terminal of the quadtree; they indicate the smallest regions on the right figure. Leaf nodes nw ne sw se nw ne sw se
are solid rectangles that store keys; they represent regions nw nw nw nw with the same numbers on the right figure. Empty nodes are 1 1 insert 7 dashed rectangles without any key. nw ne sw se nw ne sw se 2 3 4 3 4 ne sw ne sw ne sw nw ne 5 6 7 2 5 6 1 (a) Insert node 7 into the sample quadtree nw ne sw se nw ne sw se nw ne sw se nw 5 remove 5 1 2 6 nw nw nw ne sw se nw ne sw se 3 4 1 2 3 4 nw ne sw se nw ne sw se nw nw 5 6 sw se 2 3 4 1 ne sw nw ne sw se Fig. 1: A sample quadtree and its corresponding region 5 6 2 6 3 4 (b) Remove node 5 from the sample quadtree We describe the detailed structures of a quadtree in Figure 2. An Internal node maintains its four children and nw ne sw se nw ne sw se move 5 7 a two-dimensional routing structure–
1 c l a s s Node
2 c l a s s I n t e r n a l
There are four basic operations that rely on find: 1. Empty nodes are not drawn 3
3 CAS QUADTREE T1: remove 3
25 bool contain ( double keyX , double keyY ) { 26 Node p , l = root ; T1 success 27 // l: terminal node for retrieving
33 bool i n s e r t ( double keyX , double keyY, V value) { 34 while ( true ) { Fig. 5: Thread T1 intends to remove node 3 from quadtree. 35 Node p , l = root ; 36 find(p, l, keyX, keyY); After its removal, there remains a chain of Empty nodes 37 i f (inTree(l , keyX, keyY)) return false ; 38 Node newNode = createNode(l, p, keyX, keyY, value); / / c r e a t e s a sub−tree by split l until the candidate region of
42 bool remove ( double keyX , double keyY, V value) { T2 remove 1 success T1 insert 1 success 43 Node newNode = new Empty ( ) ; 44 while ( true ) { 1 1 45 Node p , l = root ; 46 find(p, l, keyX, keyY); 47 i f (!inTree(l , keyX, keyY)) return false ; Fig. 6: An example of an incorrect move operation. Thread 48 i f (helpReplace(p, l , newNode)) return true ; 49 } T1 intends to insert node 1 into a quadtree, and thread T2 50 } plans to move the value from node 1 to node 2. T2 first 51 void find (Node& p , Node& l , double keyX , double keyY ) { successfully removes node 1 and then attempts to insert 52 while ( l . c l a s s () == Internal) { 53 p = l ; // record the parent node node 2 into the quadtree. In the interval node 1 is added 54 getQuadrant(l, keyX, keyY); 55 } back by T1, but T2 is not aware of the action and reports 56 } success. 57 bool helpReplace(Internal p, Node oldChild, Node newChild) { 58 i f (p.nw == oldChild) return CAS(p.nw, oldChild , newChild) ; 59 else i f (p.ne == oldChild) return CAS(p.ne, oldChild , newChild); 60 else i f (p.sw == oldChild) return CAS(p.sw, oldChild , newChild) ; 61 else i f (p.se == oldChild) return CAS(p.se , oldChild , newChild); 62 return false ; adjustment. Figure 5 illustrates a detailed example showing 63 } that there remains a chain of Empty nodes. Hence, not only 64 bool inTree(Node node, double keyX , double keyY ) { 65 return node . c l a s s == Leaf && node.keyX == keyX && node.keyY == keyY; we have to traverse a long path to locate the terminal 66 } node for basic operations, but also a plenty of nodes are maintained in the memory. Fig. 4: CAS quadtree algorithm Second, we cannot implement the move operation in the simple algorithm. There might be two different nodes in a There are a plenty of concurrent tree algorithms, yet quadtree that were under modifying. If we directly erase the a formal concurrent quadtree has not been studied. Intu- node with oldKey and add the node with newKey, the move itively, we can devise a concurrent quadtree by modifying operation cannot be correctly linearized. Figure 6 shows the sequential algorithm described in Section 2 and adopt- such an incorrect move operation by combining an insert ing CAS instruction, which we name CAS quadtree. Figure 4 operation and a remove operation together. depicts CAS quadtree’s algorithms. Both the insert opera- Therefore, these drawbacks motivate us to develop a tion and the remove operation follow such a paradigm: it new concurrent algorithm to make the move operation starts by locating a terminal node; then, it checks whether correct and employ an efficient mechanism to compress the node satisfies some conditions. If conditions are not nodes. satisfied, it returns false. Otherwise, it tries to apply a single CAS to replace the terminal node by a new node. After a successful CAS, it returns true. Or else, it restarts locating a 4 QUADBOOST terminal node from the root node. For a insert operation, if 4.1 Rationale
r - remove flag, i - insert flag
T3: remove 3 T3: remove 3 T3: remove 3 T3: remove 1
T2: insert 2 T2: insert 2 T2: remove 1 T2: remove 1
T1: remove 1 T1: remove 1 r T1: remove 1 r T1: remove 1 r
r r r T3 flags 1’s T2 help remove 1 T3 help remove 1 1 3 gp and p 1 3 1 3 3 (a) Apply traditional removal T3: remove 3 T3: remove 3 T3: remove 3 T3: remove 3
T2: insert 2 T2: insert 2 T2: insert 2 T2: insert 2
T1: remove 1 T1: remove 1 T1: remove 1 i T1: remove 1 i
r r r 2 r T3 flags 1’s p T2 flag 2’s p T3 flag 3’s p 1 3 1 3 1 3 (b) Our decoupling approach
Fig. 7: At the beginning, three threads are performing different operations concurrently: (a) T1 removes key 1 in the lower level. (b) T2 inserts key 2 in the upper level. (c) T3 removes key 3 in the lower level. Consider the scenario that T1 precedes others threads, and then both 1’s parent and grandparent will be flagged. Hence, T2 and T3 will help T1 remove key 1 before restarting their operations. By applying our decoupling method, only the parent node should be flagged. Hence three threads could run without intervention.
grandparent of a terminal node. This mechanism mixes log- 67 c l a s s Node
Empty. 84 c l a s s Compress extends Operation { 85 Internal grandparent, parent; However, there is still a problem left after applying the 86 } above two methods. Recall the example in Figure 5. We 87 c l a s s Move extends Operation { 88 Internal iParent, rParent; flag the bottom Internal node to indicate that it should be 89 Node oldIChild, oldRChild, newIChild; 90 Operation oldIOp, oldROp; compressed, but after replacing the bottom Internal node 91 v o l a t i l e bool allFlag = false , i F i r s t = f a l s e ; with an Empty node, it results in four Empty nodes in the 92 } last level. How do we remove a series of nodes from a 93 c l a s s Clean extends Operation {} quadtree in a bottom-up way? We record the entire traversal path from the root to a terminal node in a stack. Since the Fig. 8: quadboost structures traversal path will be altered when a node is compressed, we only have to restart locating the terminal node from its parent if any flag operation other than the flag of Compress Move stores both oldKey’s and newKey’s terminal nodes, their fails. This is called the continuous find mechanism. parents, their parents’ prior ops, and a new node. Moreover, we use a bool variable–allFlag to indicate whether two 4.2 Structures and State Transitions parents have been attached to a Move op or not. Another As mentioned before, we add an Operation object to handle bool variable–iFirst is used to indicate the attaching order. concurrency issues. Figure 8 shows the data structures of For instance, if iFirst is true, iParent will be attached with a quadboost. Four sub-classes of Operation, including Substi- Move op before rParent. Clean means that there is no thread tute, Compress, Move, and Clean, describe all states in our modifying the node. In contrast to Figure 4, the Internal class algorithm. Substitute provides information on the insert adds an op field to hold its state and related information. operation and the remove operation that are designed to Note that a flag operation (CAS) is only applied on Internal replace an existing node by a new node. Hence, we shall let nodes. We atomically set Move ops in Leaf nodes to linearize other threads be aware of its parent, child, and a new node the move operation correctly. for substituting. Compress provides information for physical Each basic operation, except for the contain operation, adjustment. We erase the parent node, previously connected starts by changing an Internal node’s op from Clean to other to the grandparent, by swinging the link to an Empty node. states. The three basic operations, therefore, generate a cor- 5
responding state transition diagram which provides a high- 94 bool contain ( double keyX , double keyY ) { 95 Stack
operation to flag the parent if necessary. 144 void helpSubstitute(Substitute op) { 145 helpReplace(op.parent, op.oldChild, op.newNode); • Substitute. The thread uses a CAS to change the 146 helpFlag ( op . parent , op , new Clean ( ) ) ; // unflag the parent node to Clean existing node by a given node stored in the op. It 147 } 148 void help(Operation op) { then restores the parent’s state to Clean by unflag. 149 i f ( op . c l a s s == Compress) helpCompress(op) ; 150 else i f ( op . c l a s s == Substitute) helpSubstitute(op); • Move. The thread first determines the flag order 151 else i f ( op . c l a s s == Move) helpMove(op) ; by comparing oldKey’s parent with newKey’s parent. 152 } 153 bool moved(Node l ) { Suppose it flags newKey’s parent first, it will flag old- 154 return l . c l a s s == Leaf && l.op != null && !hasChild(l .op.iParent , Key oldKey l .op.oldIChild); ’s parent later. Then, the thread replaces ’s 155 } terminal with an Empty node and replaces newKey’s terminal with a new node. Finally, it unflags their 156 bool hasChild(Internal parent, Node oldChild) { 157 return parent.nw == oldChild || parent.ne == oldChild || parent.sw == parents’ op to Clean in the reverse order. oldChild || parent.se == oldChild; 158 } • Compress. The thread erases the node from the tree so that it cannot be detected. Different than other Fig. 10: quadboost insert and contain states, the node with a Compress op cannot be set to Clean by unflag.
node, we check whether the key of the node is in the 4.3 Concurrent Algorithms tree and whether the node is moved at line 108. We will Figure 10 reflects quadboost’s insert and contain operations. show the reason why we have to check whether a node Both of them start by a find process that locates a terminal is moved in Section 4.4. Then, we flag the parent of the node. The find operation (line 98) pushes Internal nodes into terminal node before replacing it. In the next step, we call a stack (line 95) and keeps recording the parent node’s op the helpSubstitute function at line 114, which first invokes (line 96). the helpReplace function at line 145 to replace the terminal The contain operation executes in a similar way as CAS node and then unflag the parent node at line 146. If the quadtree. It calls the find function to locate a terminal node flag operation fails, we update pOp at line 116 and help (line 98). The op at line 96 and the stack at line 95 are created it finish at line 118. More than that, we have to execute the for a modular presentation. We can omit them in a real continueFind function to restart from the nearest parent. The implementation. continueFind function (line 122) pops nodes from the path In the insert operation, we create a stack at the beginning until reaching a node whose op is not Compress. If the node’s to record the traversal path (line 98) and a pOp to record op is Compress, it helps the op finish. Or else, it breaks the the parent node’s op (line 96). After locating a terminal loop and performs the find operation from the last node 6
popped at line 132. r - remove path, i - insert path, c - common path The remove operation has a similar paradigm as the T : insert 2 c insert operation. It first locates a terminal node and checks 2 whether the node is in the tree or moved. After that, it c flags the parent node and replaces the terminal node with T : remove 1 c an Empty node. An extra step in the remove operation is 1 the compress function at line 171. In the algorithm, we c perform the compress function before the remove operation returns true. In this way, the linearization point of the r i lca
remove operation belongs to the execution of itself, and 1 extra adjustment induced by the compress function do affect the effectiveness. The compress function should examine Fig. 12: Paths of two operations–insert node 2 and remove three conditions before compressing the parent node. First, node 1 that share the LCA node because the remove operation must return true, we do not compress nor help if the state of the parent is not Clean (line 182). Second, we check if the grandparent node (gp) the findCommon function (line 212) that combines searches is the root (line 184) as we maintain two layers of dummy for two nodes together. The two searches share a common Internal nodes. At last, we check whether four children of a path that allows us to record them only once. We use two parent node are all Empty (line 186). stacks to record nodes at line 199. The shared path and the remove path are pushed into rPath; and the insert path is pushed into iPath. By doing this, we could avoid considering 159 bool remove ( double keyX , double keyY, V value) { 160 Stack
198 bool move( double oldKeyX , double oldKeyY , double newKeyX , double newKeyY) { 270 bool findCommon(Node& il , Node& rl , Internal& lca , Operation& rOp, Operation& 199 Stack rPath , iPath ; iOp, Stack& iPath , Stack& rPath, double oldKeyX , double oldKeyY , double 200 // rl: terminal node of the remove path newKeyX , double newKeyY) { 201 // il: terminal node of the insert path 271 while ( r l . c l a s s == Internal) { 202 // rp: parent node of the remove path 272 rPath . push ( r l ) ; 203 // ip: parent node of the insert path 273 rOp = r l . op ; 204 Node r l = root , i l , rp , ip ; 274 getQuadrant(rl, oldKeyX, oldKeyY); 205 // rOp: op object of rp 275 i f (!sameDirection(oldKeyX, oldKeyY, newKeyX, newKeyY, il , rl)) 206 // iOp: op object of ip 276 break // check whether two nodes are in the same direction 207 Operation rOp , iOp ; 277 } 208 // rFail: whether the remove path fail 278 l c a = rPath . top ( ) ; 209 // iFail: whether the insert path fail 279 find(rl, oldKeyX, oldKeyY, rOp, rPath); // find oldKey’s terminal 210 // cFail: whether the common path fail 280 i f (!inTree(rl , oldKeyX, oldKeyY) || moved( r l ) ) return false ; 211 bool r F a i l = false , i F a i l = false , c F a i l = f a l s e ; 281 i l = l c a ; 212 i f (!findCommon(il , rl , lca , rOp, iOp, iPath , rPath, oldKeyX, oldKeyY, 282 find ( i l , newKeyX , newKeyY , iOp , iPath ) ; // find newKey’s terminal newKeyX , newKeyY) ) return false ; 283 i f ( inTree ( i l , newKeyX , newKeyY) && ! moved( i l ) ) return false ; 213 ip = iPath . pop ( ) ; 284 } 214 rp = rPath . pop ( ) ; 215 while ( true ) { 285 bool continueFindCommon(Node& il , Node& rl , Internal& lca , Operation& rOp, 216 i f ( rOp . c l a s s != Clean) rFail = true ; Operation& iOp, Stack& iPath, Stack& rPath, Internal ip, Internal rp, 217 i f ( iOp . c l a s s != Clean) iFail = true ; double oldKeyX , double oldKeyY , double newKeyX , double newKeyY , bool 218 i f (iOp != rOp && ip == rp) cFail = true ; // the same parent node has iFail , bool rFail , bool cFail) { two different states 286 i f (rFail && !cFail) { // restart to find oldKey’s terminal in cast that 219 i f (!cFail && !iFail && !rFail) { the lca’s op remains the same 220 i f ( i l . c l a s s == Empty || i l == r l ) newNode = new Leaf (newKeyX , 287 help ( rOp ) ; newKeyY , r l . value ) ; 288 i f ( rOp . c l a s s != Compress) rl = rp; 221 else newNode = createNode ( i l , ip , newKeyX , newKeyY , r l . value ) ; 289 else { 222 //iFirst: flag which node first 290 while (! rPath.empty() ) { 223 bool iFirst = getSpatialOrder(ip, rp); 291 i f (rPath.size() <= indexOf(lca)) { // reach the lca node, 224 Operation op = new Move(ip, rp, il , rl , newNode, iOldOp, rOldOp, 292 c F a i l = true ; // break the loop the find the common path i F i r s t ) ; a ga i n 225 i f ( rp != ip ) { // two parents are different 293 break ; 226 bool hasFlag = iFirst ? helpFlag(ip, iOp, op) : helpFlag(rp, 294 } rOp , op ) ; 295 r l = rPath . pop ( ) ; 227 i f ( hasFlag ) { 296 rOp = r l . op ; 228 i f (helpMove(op) ) { 297 i f ( rOp . c l a s s == Compress) helpCompress(rOp) ; 229 compress ( rp , rPath ) ; 298 else break ; 230 return true ; 299 } 231 } else { // all flag operation fail 300 } 232 r F a i l = i F a i l = true ; 301 i f ( ! c F a i l ) { // the lca node has not been changed 233 } 302 find(rl, oldKeyX, oldKeyY, rOp, rPath); 234 } else { // one of two paths fail 303 i f (!inTree(rl , oldKeyX, oldKeyY) || moved( r l ) ) return false ; 235 i f ( i F i r s t ) {iOp = ip.op; iFail = true ;} 304 else { 236 else {rOp = rp.op; rFail = true ;} 305 rp = rPath . pop ( ) ; 237 } 306 return true ; 238 } else { // two parents are the same 307 } 239 i f (helpMove(op) ) return true ; 308 } 240 else {rOp = iOp = rp.op; cFail = true ;} 309 } 241 } 310 i f (iFail && !cFail) { 242 } 311 help ( iOp ) ; 243 i f (!continueFindCommon(iFail , rFail , cFail , il , rl , ip, rp, lca, rOp, 312 i f ( iOp . c l a s s != Compress) il = ip; iOp, oldKeyX, oldKeyY, newKeyX, newKeyY, iPath , rPath)) return 313 else { f a l s e ; // continuous find insert and remove paths 314 while (! iPath.empty() ) { 244 c F a i l = i F a i l = r F a i l = f a l s e ; 315 i l = iPath . pop ( ) ; 245 } 316 iOp = i l . op ; 246 } 317 i f ( iOp . c l a s s == Compress) helpCompress(iOp) ; 318 else break ; 247 bool helpMove (Move op ) { 319 } 248 i f (op. iFirst) helpFlag(op.rParent , op.oldROp, op); // flag the parent of 320 } the oldKey’s terminal first 321 i f ( iOp . c l a s s == Compress) cFail = true ; // check the last op which 249 else helpFlag(op.iParent , op.oldIOp, op); // flag the parent of the must be the op of the lca node newKey’s terminal first 322 i f ( ! c F a i l ) { 250 bool doCAS = op.iFirst ? op.rParent.op == op: op.iParent.op == op; // 323 find ( i l , newKeyX , newKeyY , iOp , iPath ) ; whether the flag operation succeed 324 i f ( inTree ( i l , newKeyX , newKeyY) && ! moved( i l ) ) return false ; 251 i f (doCAS) { // all flags have been done 325 else { 252 op . a l l F l a g = true ; 326 ip = iPath . pop ( ) ; 253 op . oldRChild . op = op ; 327 return true ; 254 i f (op.oldRChild == op.oldRChild) { // combine two CASes in two one 328 } 255 helpReplace(op.rParent, op.oldRChild, op.newIChild); 329 } 256 } else { 330 } 257 helpReplace(op.iParent, op.oldIChild, op.newIChild); 331 i f ( c F a i l ) { 258 helpReplace(op.rParent, op.oldRChild, new Empty ( ) ) ; 332 help ( iOp ) ; 259 } 333 help ( rOp ) ; 260 } 334 rPath.setIndex(indexOf(lca)); // pop out all nodes above the lca node 261 i f (op. iFirst) { // unflag in a reverse order 335 iPath . c l e a r ( ) ; // clean the input path 262 i f (op.allFlag) helpFlag(op.rParent , op, new Clean ( ) ) ; 336 while (! rPath.empty() ) { //first time must be not empty 263 i f (op.rParent != op.rParent) helpFlag(op.iParent , op, new Clean ( ) ) ; 337 r l = rPath . pop ( ) ; 264 } else { 338 rOp = r l . op ; 265 i f (op.allFlag) helpFlag(op.iParent , op, new Clean ( ) ) ; 339 i f (rOp. getClass () == Compress) helpCompress(rOp) ; 266 i f (op.rParent != op.rParent) helpFlag(op.rParent , op, new Clean ( ) ) ; 340 else break ; 267 } 341 } // pop up nodes in the common path 268 return op.allFlag; 342 i f (!findCommon(il , rl , lca , rOp, iOp, iPath , rPath, oldKeyX, oldKeyY, 269 } newKeyX , newKeyY) return false ; 343 else { // find two nodes 344 rp = rPath . pop ( ) ; 345 ip = iPath . pop ( ) ; Fig. 13: quadboost move 346 } 347 } 348 }
Fig. 14: quadboost findCommon and continueFindCommon
oldKey at line 302. If ip cannot be flagged, it pops nodes 4.5 One Parent Optimization from iPath until it is empty (line 315) or a node’s op is not Compress (line 317). It again continues to search for newKey In practice, we notice that pushing a whole stack during at line 323. If either rPath has popped the LCA node or iPath a traversal is highly expensive. When detecting a failed is empty, it clears iPath at line 335 and pops all nodes above flag operation, we only have to restart from the parent of the LCA node from rPath at line 334. In the end, it calls a terminal node because we do not change Internal nodes the findCommon function to locate terminal nodes again at unless the compress function erases them from a quadtree. line 342. We prove that if oldKey’s terminal and newKey’s Thus, to reduce the pushing cost, we could only record the terminal share an LCA node, the common path will never parent of the terminal node during a traversal. be changed unless the LCA is altered. Meanwhile, we have to change the continuous find 8
mechanism. For the insert operation and the remove op- Observation 6. The help function, the helpCompress function, eration, encountering a Compress op, we straightforwardly the helpMove function, and the helpSubstitute function are not restart from the root. For the move operation, if either called in a mutual way. (If method A calls method B, and method oldKey’s or newKey’s parent is under compression, we restart B also calls method A, we say A and B are called in a mutual from the LCA node. If the LCA node also has a Compress op, way.) we restart from the root. 5.1 Basic Invariants 5 PROOF SKETCH We use find(keys) to denote a set of find operations: find, continueF ind, findCommon, and In this section, we prove that quadboost is both lineariz- continueF indCommon. The find function and the able and non-blocking, and we propose a lengthy proof in continueFind function return a tuple hl, pOp, pathi. The appendix A. findCommon function and the continueFindCommon There are four kinds of basic operations in quadboost: the function return two such tuples. We specify functions insert operation, the remove operation, the move operation, outside the while loop in the insert operation, the remove and the contain operation. Other functions are called sub- operation, and the move operation are at iteration0, and routines, which are invoked by basic functions. The insert functions inside the while loop are at iterationi, 0 < i operation and the remove operation only modify one termi- ordered by their invocation sequence. nal node, whereas the move operation might operate two We suppose that find(keys) executes from a valid different terminals–one for inserting a node with newKey, snapshotT to derive the following conditions. Proofs for the other for removing a node with oldKey. We call them i conditions of other subroutines are included in appendix A. newKey’s terminal and oldKey’s terminal, and we call their parents newKey’s parent and oldKey’s parent, respectively. Lemma 1. The post-conditions of find(keys) returned at Ti, k k k We define snapshotTi as the state of our quadtree at some with tuples hl , pOp , path i, 0 ≤ k < |keys|. T time, i. 1) lk is a Leaf node or an Empty node. In our proofs, a CAS that changes a node’s op is a flag k 2) At some Ti1 < Ti, the top node in path has contained operation and a CAS that changes a node’s child is a replace pOpk. iflag rflag mflag operation. Specifically, we use , , , and 3) At some T < T , the top node in pathk has contained cflag i2 i to denote flag operations for the insert operation, the lk. remove operation, the move operation, and the compress k k 4) If pOp is read at Ti1, and l is read at Ti2, then Ti1 < operation separately. Likewise, we use ireplace, rreplace, Ti2 < Ti. mreplace, and creplace for replace operations. Moreover, k k 5) For each node n in the path , size(path ) ≥ 2, nt we specify a flag operation that attaches a Clean op on a is on the top of nt−1, and nt is on the direction d ∈ node as an unflag operation. {nw, ne, sw, se} of nt−1 at Ti1 < Ti. First, we present some observations from quadboost. Then, we propose lemmas to show that our subroutines sat- Based on these post-conditions, we show that each op isfy their pre-conditions and post-conditions because proofs created at Ti store their corresponding information. on basic operations depend on these conditions. Next, we Lemma 2. For op created at Ti: demonstrate that there are three categories of successful CAS transitions according to Figure 9. We also derive some 1) If op is Substitute, op.parent has contained invariants of these transitions. Using above post-conditions op.oldChild that is a Leaf node at Ti1 < Ti from the of subroutines and invariants, we could demonstrate that a results of find(keys) at the prior iteration. quadtree’s structure is maintained during concurrent mod- 2) If op is Compress, op.grandparent has contained ifications. In following proofs, we show that quadboost is op.parent that is an Internal node at Ti1 < Ti from linearizable because it can be ordered equivalently as a the results of find(keys) at the prior iteration. sequential one by its linearization points. In the last part, we 3) If op is Move, op.iP arent has contained op.oldIChild prove the non-blocking progress condition of quadboost. that is a Leaf node and op.oldIOp before Ti, and op.rP arent has contained op.oldRChild that is a Leaf Observation 1. The key field of a Leaf node is never changed. and op.oldROp before Ti from the results of find(keys) The op field of a Leaf node is initially null. The space information at the prior iteration. of an Internal node is never changed. Though we have not presented details of the createNode Observation 2. The root node is never changed. function, our implementation could guarantee that it has Observation 3. A flag operation attaches an op on a Internal following conditions. node. Lemma 3. For createNode(l, p, newKeyX, newKeyY, value) Observation 4. The allF lag and iF irst field in Move are that returns a newNode invoked at Ti, it has the post-condition: initially false, and they will never be set back after assigning to 1) The newNode returned is either a Leaf node with true. newKey and value, or a sub-tree that contains both Observation 5. If a Leaf node is moved, its op is set before l.key node and newKey node with the same parent. the replace operation on op.iP arent, which is before the replace Using prior conditions, we derive some invariants dur- operation on op.rP arent. ing concurrent executions and prove that there are three 9
kinds of successful CAS transitions. We put successful flag 1) unflagi is the first successful unflag operation on operations that attach ops on nodes at the beginning of each opi.rP arent. CAS transition. We say every successive replace operations 2) The first flag operation on opi.iP arent must fail, and that read op belongs to it and follows the flag operations. no later flag operation succeeds. Let flag0, flag1, ..., flagn be a sequence of successful 3) opi.iP arent and opi.rP arent are different. flag operations. flagi reads pOpi and attaches opi. replacei and unflagi read opi and follows it. Therefore, we say flagi, Claim 1. There are three kinds of successful transitions belong to replacei, and unflagi belong to the same op. In addition, an op: (1) flag → replace → unflag, (2) flag → unflag, if there are more than one replace operation belongs to (3) flag → replace. 0 1 the same opi, we denote them as replacei , replacei , ..., n Then, we prove that a quadtree maintains its properties replacei ordered by their successful sequence. Similarly, during concurrent modifications. if there is more than one flag operation that belonging to 0 1 opi on different nodes, we denote them as flagi , flagi , ..., Definition 1. Our quadtree has the following properties: n flagi . A similar notation is used for unflagi. The following lemmas prove the correct ordering of three 1) Two layers of dummy Internal nodes are never changed. different transitions. 2) An Internal node n has four children located in the direction d ∈ {nw, ne, sw, se} according to their Lemma 4. For a new node n: hx, y, w, hi, or hkeyX, keyY i. 1) It is created with a Clean op. For Internal nodes residing on four directions: 2) rflag, iflag, mflag or cflag succeeds only if n’s op • n.nw.x = n.x, n.nw.y = n.y; is Clean. • n.ne.x = n.x + w/2, n.ne.y = n.y; 3) unflag succeeds only if n’s op is Substitute or Move. • n.sw.x = n.x, n.sw.y = n.y + n.h/2; 4) Once n’s op is Compress, its op will never be changed. • n.se.x = n.x + w/2, n.se.y = n.y + h/2, Lemma 5. For an Internal node n, it never reuses an op that has 0 0 been set previously. All children have their w = n.w/2, h = n.h/2. k k For Leaf nodes residing on four directions: Lemma 6. replacei will not occur before flagi that belongs to the same op that has been done. • n.x ≤ n.nw.keyX < n.x + n.w/2, n.y ≤ n.nw.keyY < n.y + n.h/2; Lemma 7. The flag → replace → unflag transition occurs • n.x + n.w/2 ≤ n.ne.keyX < n.x + n.w, when rflag , iflag or mflag succeeds, and it has following i i i n.y ≤ n.ne.keyY < n.y + n.h/2; properties: • n.x ≤ n.sw.keyX < n.x + n.w/2, 1) replacei never occurs before flagi. n.y + n.h/2 ≤ n.sw.keyY < n.y + n.h; k 2) flagi , 0 ≤ k < |flagi| is the first successful flag • n.x + n.w/2 ≤ n.se.keyX < n.x + n.w, k k operation on opi.parent after Ti1 when pOpi is read. n.y + n.h/2 ≤ n.se.keyY < n.y + n.h. k 3) replacei , 0 ≤ k < |replacei| is the first success- k ful replace operation on opi.parent after Ti2 when To help clarify a quadtree’s properties during concur- k opi.oldChild is read. rent executions, we define active set and inactive set for k Internal Empty 4) replacei , 0 ≤ k < |replacei| is the first successful different kinds of nodes. For an node or an k replace operation on opi.parent that belongs to opi. node, if it is reachable from the root in snapshotTi , it is k Leaf 5) unflagi , 0 ≤ k < |unflagi| is the first successful active; otherwise, it is inactive. For a node, if it is k k snapshot unflag operation on opi.parent after flagi . reachable from the root in Ti and not moved, 6) There is no successful unflag operation that occurs before it is active; otherwise, it is inactive. We say a node n replacei. is moved in snapshotTi if the function moved(n) returns k k 7) The first replace operation on opi.parent that belongs true at Ti. We denote path(keys ), 0 ≤ k < |replacei| to opi must succeed. as a stack of nodes pushed by find(keys) in a snapshot. We define physical path(keysk) to be the path for keysk flag → replace Lemma 8. The transition occurs only when in snapshot , consisting of a sequence of Internal nodes cflag Ti i succeeds, and it has following properties: with a Leaf node or an Empty node at the end. We say k 1) creplacei never occurs before cflagi. a subpath of path(keys ) is an active path if all nodes k 2) cflagi is the first successful flag operation on from the root to node n ∈ path(keys ) are active. Hence k opi.parent after Ti1 when pOpi is read. a physical path(keys ) is active only if the end node is not 3) creplacei is the first successful replace operation on moved. op .grandparent after T when op .parent is read. i i2 i Lemma 10. Two layers of dummy nodes are never changed. 4) creplacei is the first successful replace operation on opi.grandparent that belongs to opi. Lemma 11. Children of a node with a Compress op will not be 5) There is no unflag operation after creplacei. changed. 6) The first replace operation on opi.grandparent that Lemma 12. Only an Internal node with all children Empty could belongs to opi must succeed. be attached with a Compress op. Lemma 9. For the flag → unflag transition, it only results from mflag (Suppose iF irst is false) when: Lemma 13. An Internal node whose op is not Compress is active. 10
Lemma 14. After the invocation of find(keys) which reads Lemma 18. If the insert operation, the remove operation, or lk, there is a snapshot in which the path from the root to lk is the move operation returns false, there is no successful replace physical path(keysk). happens during the execution. Lemma 15. After ireplace, rreplace, mreplace, and The next lemma points out the linearization points of creplace, a quadtree’s properties remain. the contain operation. Lemma 19. For the contain operation that returns true, there Proof. We shall prove that in any snapshot , a quadtree’s Ti is a corresponding snapshot that lk in physical path(keysk) properties remain. is active in snapshot . For the contain operation that re- First, Lemma 10 shows that two layers of dummy nodes Ti turns false, there is a corresponding snapshot that lk in remain in the tree. We have to consider other layers of nodes physical path(keysk) is inactive in snapshot . that are changed by replace operations. Ti Consider creplace that replaces an Internal node by an We list out the linearization points of other operations Empty node. Because the Internal node has been flagged on a as follows: Compress op before creplace (Lemma 8), all of its children are • insert(key). The linearization point of the insert oper- Empty and not changed (Lemma 11 and Lemma 12). Thus, ation that returns false is T after calling find(key) creplace does not affect the second claim of Definition 1. i at which l at the end of physical path(key) contains Consider ireplace, rreplace, or mreplace that replaces the key and is not moved. For the insert operation a terminal node by an Empty node, a Leaf node, or a sub- that returns true, the linearization point is at the first tree. By Lemma 7, before replacek, op.parentk is flagged successful replace operation (line 145). with op such that no successful replace operation could • remove(key). The linearization point of the remove op- happen on op.parentk. Therefore, if the new node is Empty, eration that returns false is Ti after calling find(key) it does not affect the tree property. If the new node is a at which l at the end of physical path(key) does Leaf node or a sub-tree, based on the post-conditions of the not contain the key or is moved. For the successful createNode function (Lemma 3), after replace operations the remove operation, we define the linearization point second claim of Definition 1 still holds. where the node with key is replaced by an Empty Claim 2. A quadtree maintains its properties in every snapshot. node (line 145). • move(oldKey, newKey). For the unsuccessful move operation, the linearization point depends on 5.2 Linearizability both newKey and oldKey. If rl at the end of physical path(oldKey) does not contain oldKey In this section, we define linearization points for basic op- or is moved, the linearization point is Ti1 after erations. As the compress function is included in the move calling find(keys). Or else, if il at the end of operation and the remove operation that return true, it does physical path(newKey) contains newKey and is not affect the linearization points of these operations. If an not moved, the linearization point is at Ti2 after algorithm is linearizable, its result be ordered equivalently calling find(keys). as a sequential history by the linearization points. Since For the successful move operation, the lineariza- find(keys) all modifications depend on , we first point tion point is the first successful replace operation. out its linearization point. For find(keys), we define its k (line 255 or line 257) linearization point at Ti such that l returned is on the k physical path(keys ) in snapshotTi . Claim 3. quadboost is linearizable. For the contain operation that returns true, we show k that there is a corresponding snapshot in which l in 5.3 Non-blocking physical path(keysk) is active. For the contain operation, Finally, we prove that quadboost is non-blocking, which the insert operation, the remove operation, or the move means that the system as a whole is making progress even operation that returns false, we show there is a corre- if some threads are starving. sponding snapshot in which lk in physical path(keysk) is inactive. For the insert, remove, and move operations Lemma 20. A node with a Compress op will not be pushed into that returns true, we define the linearization point to be path more than once. 0 its first successful replace operation—replacei . To make a k k Lemma 21. For path(keys ), if nt is active in snapshotTi , reasonable demonstration, we first show that replacei , 0 ≤ then n0, ..., nt−1 pushed before nt are active. k < |replacei| belongs to each operation that creates opi and illustrate that op is unique for each operation. Lemma 22. If n is the LCA node in snapshotTi on physical path for oldKey and newKey. Then at T ,T > Lemma 16. For find(keys) that returns tuples hlk, pOpk, i1 i1 k k Ti, n is still the LCA on active path for oldKey and newKey path i, there is a snapshotTi such that path returned with k k if it is active. l at the end is physical path(key ) in snapshotTi . Lemma 23. pathk returned by find(keys) consists of finite Lemma 17. If the insert operation, the remove operation, or the number of keys. move operation returns true, the first successful replace operation occurs before returning, and it belongs to the op created by the Lemma 24. There is a unique spatial order among nodes in a operation itself at the last iteration in the while loop. quadtree in every snapshot. 11
Lemma 25. There are a finite number of successful flag → 6 EVALUATION replace →, flag → replace, flag → unflag transitions. We ran experiments on a machine with 64GB main memory and two 2.6GHZ Intel(R) Xeon(R) 8-core E5-2670 proces- Lemma 26. If the help function returned at Ti, and find(keys) k sors with hyper-threading enabled, rendering 32 hardware at the prior iteration reads p .op at Ti1 < Ti, keys in snapshotT i threads in total. We used the RedHat Enterprise Server 6.3 and snapshotT are different. i1 with Linux core 2.6.32, and all experiments were ran under Claim 4. quadboost is non-blocking. Sun Java SE Runtime Environment (build 1.8.0 65). To avoid significant run-time garbage collection cost, we set the initial heap size to 6GB. Proof. We have to prove that no process will execute For each experiment, we ran eight 1-second cases, where loops infinitely without changing keys in a quadtree. the first 3 cases were performed to warm up JVM, and the First, we prove that path is terminable. Next, we k median of the last 5 cases was used as the real performance. prove that find(keys ) starts from an active node in Before the start of each case, we inserted half keys from the physical path keysk snapshot i ( ) in Ti between th iteration key set into a quadtree to guarantee that the insert and the i snapshot and +1th iteration. Finally, some Leaf nodes in Ti1 remove operation have equal success opportunity initially. at the returning of find(keys) at ith iteration are different We apply uniformly distributed key sets that contain snapshot find keys i from Ti2 at the returning of ( ) at + 1th two-dimensional points within a square. We use the range iteration. to denote the border of a square. Thus, points are located For the first part, we initially start from the root node. inside a range ∗ range square. In our experiments, we used Therefore, path is empty. Moreover, as Lemma 22 shows two different key sets: 102 keys to measure the performance k that path consists of finite number of keys, we establish under high contention, and 106 keys to measure the per- this part. formance under low contention. For simplicity, we let the For the second part, the continueFind function and the range of the first category experiment be 10, rendering continueFindCommon function pop all nodes with Compress 1 − 102 consecutive keys for one-dimensional structure. For op from path. For the insert operation and remove opera- the second category experiment, we let the range be 1000, tion, since Lemma 13 shows that an Internal nodes whose op generating 1 − 106 consecutive keys. is not Compress is active, and Lemma 21 shows that nodes above the active node are also active, there is a snapshot in TABLE 1: The concurrent quadtree algorithms with different which the top node of path is still in physical path(keysk). optimization strategies.
For the move operation, if either rF ail or iF ail is true, it is type insert remove move qc single CAS single CAS not support equivalent with the prior case. If cF ail is true, Lemma 22 il- flag CAS, flag CAS, flag CAS, continuous find, continuous find, physical path qb-s continuous find, lustrates that if the LCA node is active, it is in decoupling stack, decoupling stack, stack for both oldKey and newKey. Thus, there is also a snapshot recursive compression recursive compression flag CAS, flag CAS, flag CAS, qb-o continuous find, continuous find, that the start node is in physical path. continuous find decoupling compression decoupling compression The third part is proved by contradiction, assuming that a quadtree is stabilized at Ti, and all invocations after Ti are We evaluate quadboost algorithms via comparing with looping infinitely without changing Leaf nodes. the state-of-the-art concurrent trees (kary, ctrie, and patricia) For the insert operation and the remove operation, before for throughput (Section 6.1) and presenting the incremen- the invocation of find(keys) at the next iteration, they tal effects of the optimization strategies proposed in this must execute the help function at line 118 and line 175 work (Section 6.2). Table 1 lists the concurrent quadtree accordingly. In both cases, the help function changes the algorithms. qb-o (quadboost-one parent) is the one parent snapshot (Lemma 26). optimization based on qb-s (quadboost-stack) mentioned in For the move operation, different situations of the con- Section 4.4. qc is the CAS quadtree introduced in Section 3. tinueFindCommon function are considered. If rF ail or iF ail is true, two situations arise: (1) iOp or rOp is Clean 6.1 Throughput but mflag fails; or (2) iOp or rOp is not Clean. In both cases, To the best of our knowledge, a formal concurrent quadtree before the invocation of find(keysk), the help function is has not been published yet. Hence, we compare our performed at line 311 and line 287. Thus by Lemma 26, the quadtrees with three one-dimensional non-blocking trees: snapshot is changed between two iterations. Now consider • kary is a non-blocking k-way search tree, where k cF ail ip rp if is true. If and are the same, it could result represents the number of branches maintained by an rOp iOp from the difference between and . In this case, internal node. Like the non-blocking BST [4], keys rOp the snapshot might be changed between reading and are kept in leaf nodes. When k = 2, the structure iOp mflag . It could also result from the failure of . For the is similar as the non-blocking BST; when k = 4, above cases, the help function at line 332 would change the each internal node has four children, it has a simi- ip rp quadtree. If and are different, it results from either lar structure to quadtree. However, kary’s structure iP ath rP ath or that have popped the LCA node. We have depends on the modification order, and it does not iF ail rF ail proved the case in which either or is true and have a series of internal nodes representing the two- derives a contradiction. dimensional space hierarchy. From the above discussions, we prove that quadboost is • ctrie is a concurrent hash trie, where each node can non-blocking. store up to 2k children. We use k = 2 to make a 12
4-way hash trie that resembles quadtree. The hash s with the decoupling approach shown in Figure 7. qb- trie also incorporates a compression mechanism to s is worse than qb-o due to its extra cost of recording reduce unnecessary nodes. Different from quadtree, elements and compressing nodes recursively. Figure 15b it uses a control node (INODE as the paper indicates) exhibits results from when the key set was large. There is to coordinate concurrent updates. Hence, the search less collision among threads but deeper depths of trees than depth could be longer than quadtree. the smaller key set. In the scenario, qb-o and qb-s show a • patricia is a binary search tree, which adopts Ellen’s similar performance as qc because of less number of CAS BST techniques [4]. As the author points out, it can be failures caused by thread interventions. qb-o is only 12% used as quadtree by interleaving the bits of x and y. It worse than qc at 32 threads, but 47% better than ctrie, 35% also supports the move (replace) operation like quad- better than kary, and 39% better than patricia mainly for boost. Unlike our LCA-based operation, it searches its shorter traversal paths caused by its static representation two positions separately without a continuous find and the continuous find mechanism. As we discussed in mechanism. the next section, qb-s and qb-o save a significant number of nodes as shown in Figure 18. It implies that qb-s and Since the above structures only store one-dimensional qb-o occupy less memory and result in a shorter path for keys, we had to transform a two-dimensional key into a traversal. one-dimensional key for comparison. Though patricia could 7 ×10 ×107 store two-dimensional keys using the above method, ctrie 5 3 qb-s and kary cannot use it. Thus, we devised a general formula: 4 1 2 2 qb-o patricia 2 key = keyx ∗ range + keyy. Given a two-dimensional key- 3 2 key , and range, we transformed it into a one-dimensional 2 1 1 key-key . To refrain from trivial transformations by float- 1 Throughput(op/s) Throughput(op/s) ing numbers, we only considered integer numbers in this 0 0 1 2 4 8 16 32 1 2 4 8 16 32 section. #Threads #Threads 2 6 ×107 ×107 (a) 10 keys (b) 10 keys 5 4 qb-s 4 qc qb-o 3 Fig. 16: Comparison of the move operation’s throughput 3 ctrie kary 2 between quadboost and patricia in both small and large 2 patricia ranges (10% insert, 10% remove, 80% move). 1 1 Throughput(op/s) Throughput(op/s)
0 0 1 2 4 8 16 32 1 2 4 8 16 32 Figure 16b demonstrates that quadboost has an efficient #Threads #Threads move operation. Using the small key set, where the depth (a) 102 keys (b) 106 keys is not a significant impact, Figure 16a shows that qb-o is more efficient than patricia especially when contention Fig. 15: Throughput of different concurrent trees under both is high. For example, it performs better than patricia by high and low contention (50% insert, 50% remove). 47% at 32 threads because it adopts the continuous find mechanism to traverse less path and decouples physical Due to the lack of the move operation in CAS quadtree adjustment for higher concurrency. However, qb-s is similar (qc), we compare throughput with/without the move op- to patricia since it has to maintain a stack and recursively eration respectively. Figure 15 plots throughput without compress nodes in a quadtree. Figure 16b illustrates that any move operation for the concurrent algorithms. It is qb-s and qb-o have a similar throughput for the large key not surprising to observe that qc achieves the highest set. qb-o outperforms patricia by 31% at 32 threads. When throughput. To some extent, qc represents an upper bound the key set is large, the depth becomes a more significant of throughput because it maintains the hierarchy without factor due to less contention. Since each Internal node in physical removal, i.e., its remove operation only applies a a quadtree maintains four children while patricia maintains CAS on edges to change links, which leads to less contention two, the depth of patricia is deeper than quadboost. Further, than other practical concurrent algorithms. However, both the combination of the LCA node and the continuous find qb-s and qb-o can achieve comparable throughput when mechanism ensures that qb-o and qb-s do not need to the key set becomes larger. This phenomenon occurs due restart from the root even if flags on two different nodes to: (i) given the large key set, fewer thread interventions fail. results in a lesser number of CAS failures on nodes; and (ii) both algorithms compress nodes and use the continuous 6.2 Analysis find mechanism to reduce the length of traverse path. To determine how quadboost algorithms improve the per- As a comparison, ctrie, kary, and patricia show lower formance, we devised two algorithms that incrementally use performances with the increasing number of threads. For parts of techniques in qb-o: instance, in Figure 15a at 32 threads, qb-o outperforms ctrie by 49%, patricia by 79%, and kary by 109%. Note that qb- • qb-f flags the parent of a terminal node in the move o and qb-s incorporate the continuous find mechanism to operation, the insert operation, and the remove oper- reduce the length of traverse path. Further, both kary and ation. It restarts from the root without a continuous patricia flag the grandparent node in the remove operation, find mechanism. Further, it adopts the traditional which allows less concurrency than ctrie, qb-o and qb- remove mechanism mentioned in Figure 7a. 13
• qb-d decouples the physical adjustment in the re- bars to the left) and has three times the amount of nodes of move operation based on qb-f. qb-s. The result also indicates that qb-o contains a similar number of nodes to qb-s despite it only compresses one range here was set to 232 − 1, and both key and key x y layer of nodes. Figure 18b illustrates the effectiveness of could be floating numbers. We used an insert dominated compression in the face of tremendous contain operations. experiment and a remove dominated experiment to demon- qb-s outperforms qc by 30% at 9:1 insert:remove ratio because strate the effects of different techniques. We used a remove qb-s adjusts the quadtree’s structure by compression to dominated experiment to show the effect of decoupling, reduce the length of the search path. With the increment of where insert:remove ratio was 1:9. In the insert dominated the insert ratio, qb-s performs similar to qc due to the extra experiment, the insert:remove ratio was 9:1; hence, there cost of maintaining a stack and the recursive compression. were far more insert operations. Since fewer compress op- However, qb-o achieves good balance between qb-s and qc, erations were induced, the experiment showed the effect which compresses one layer of nodes without recording the of the continuous find mechanism. Figure 17a illustrates whole traverse path.
×107 ×107 3 6 qb-f 2.5 qb-d 7 RELATED WORKS qb-o 4 2 Because there are few formal works related to concurrent 1.5 2 quadtrees, we present a roadmap to show the development 1 Throughput(op/s) Throughput(op/s) of state-of-the-art concurrent trees in Figure 19. 0.5 0 1 2 4 8 16 32 1 2 4 8 16 32 #Threads #Threads Arbel (a) 10% insert, 90% remove (b) 90% insert, 10% remove Brown k-ary Brown general Fig. 17: Throughput comparison in insert dominated and 2 Ellen Howley Ellen amortized remove dominated cases (10 keys). Ramachandran Shafiei Natarajan lock that quadtrees with decoupling exhibit a higher throughput Crain Crain Ramachandran than qb-f, the basic flag concurrent quadtree. Besides, qb- speculation contention lock-free o which incorporates the continuous find is more efficient Bronson Besa Drachsler than qb-d. Specifically, at 32 threads, qb-o performs 15% better than qb-d and 51% better than qb-f. From Figure 17b, Prokopec Chatterjee we determine that qb-o outperforms qb-f by up to 35%. 2010 2011 2012 2013 2014 2015 Therefore, it demonstrates that the continuous find mecha- nism and the decoupling approach play a significant role in our algorithm. Fig. 19: Concurrent trees roadmap Ellen [4] provided the first non-blocking BST and proved ×106 ×107 3 5 qc qc it correct. This work is based on the cooperative method qb-s qb-s 4.5 2 qb-o qb-o described in Turek [17] and Barnes [18]. Brown [19] used a 4 similar approach for the concurrent k-ary tree. Shafiei [20]
#Nodes 1 also applied the method for the concurrent patricia trie. It 3.5 Throughput (op/s) also showed how to design a concurrent operation where 0 3 1:9 3:7 5:5 7:3 9:1 1:9 3:7 5:5 7:3 9:1 two pointers need to be changed. The above concurrent insert : remove insert : remove trees have an external structure, where only leaf nodes (a) Nodes (b) Throughput contain actual keys. Howley [5] designed the first internal BST by the cooperative method. Recently, Brown [21] pre- Fig. 18: Number of nodes left and throughput under differ- sented a generalized template for all concurrent down-trees. ent ratios of insert:remove from 9:1 to 1:9, with fixed 90% Ellen [22] exhibited how to incorporate a stack to reduce the contain, under 32 threads and 106 keys. original complexity from O(ch) to O(h + c). Our quadboost is a hybrid of these techniques. It uses a cooperative method Another advantage of quadboost results from the com- for concurrent coordination, changes two different positions pression technique, which reduces the search path for each with atomicity, and devises a continuous find mechanism to operation and the memory consumption. Figure 182 plots reduce restart cost. the number of nodes left and the throughput of each Different from the previously mentioned methods that quadtree at different insert:remove ratios. As qc only replaces apply flags on nodes, Natarajan [6] illustrated how to ap- a terminal node with an Empty node without compression, ply flags on edges for a non-blocking external BST. Ra- it results in the greatest number of nodes in the memory machandran [7] adopted CAS locks on edges to design a (Figure 18a). In contrast, qb-s and qb-o compress a quadtree concurrent internal BST and later extended the work to non- if necessary. When qc contains much more nodes than other blocking [8]. Their experiments showed that internal trees quadtrees, the remove operation dominates (the first groups are more scalable than external trees with a large key range. 2. Unlike previous experiments, we run eight 3-second cases in the On one hand, internal trees take up less memory; on the experiment to ensure stable amount of modifications other hand, the remove operation is more complicated than 14 that in external trees. Chatterjee [9] provided a threaded- [9] B. Chatterjee, N. Nguyen, and P. Tsigas, “Efficient lock-free binary BST with edge flags and claimed it has a low theoretical search trees,” in Proceedings of the 2014 ACM symposium on Princi- ples of distributed computing. ACM, 2014, pp. 322–331. complexity. Unlike the above trees that have to flag their [10] D. Drachsler, M. Vechev, and E. Yahav, “Practical concurrent binary edges before removal, our CAS quadtree uses a single CAS search trees via logical ordering,” in ACM SIGPLAN Notices, in both the insert operation and the remove operation. vol. 49, no. 8. ACM, 2014, pp. 343–356. The first balanced concurrent BST was proposed by [11] V. Ng and T. Kameda, “Concurrent accesses to r-trees,” in Advances in Spatial Databases. Springer, 1993, pp. 142–161. Bronson [23] in which they used an optimistic and relaxed [12] J. Chen, Y.-F. Huang, and Y.-H. Chin, “A study of concurrent balance method to build an AVL tree. Besa [24] employed operations on r-trees,” Information Sciences, vol. 98, no. 1, pp. 263– a similar method for a red-black tree. Crain [25] proposed 300, 1997. [13] R. Obe and L. Hsu, PostGIS in action. Manning Publications Co., a method that decouples physical adjustment from logical 2011. removal by a background thread. Drachsler [10] mentioned [14] A. C. Tassio Knop, “Qollide - Quadtrees and Collisions,” https:// an alternative technique called logical ordering, which uses graphics.ethz.ch/∼achapiro/gc.html, 2010, [Online; accessed 29- the key order of the BST to optimize the contain operation. August-2015]. [15] G. J. Sullivan and R. L. Baker, “Efficient quadtree coding of images All of these works were constructed on fine-grained locks and video,” Image Processing, IEEE Transactions on, vol. 3, no. 3, pp. and are deadlock free. Based on the special properties 327–331, 1994. of quadtree, we also decoupled physical adjustment from [16] D. P. Mehta and S. Sahni, Handbook of data structures and applica- tions. CRC Press, 2004. logical removal and achieved a high throughput. [17] J. Turek, D. Shasha, and S. Prakash, “Locking without blocking: There are numerous studies on concurrent trees in making lock based concurrent data structure algorithms nonblock- addition to ours and the ones mentioned in this work. ing,” in Proceedings of the eleventh ACM SIGACT-SIGMOD-SIGART Crain [26] designed a concurrent AVL tree based on STM. symposium on Principles of database systems. ACM, 1992, pp. 212– 222. Prokopec [27] used a control node, which is similar to our [18] G. Barnes, “A method for implementing lock-free shared-data Operation object to develop a concurrent trie. Arbel [28] structures,” in Proceedings of the fifth annual ACM symposium on provided a balanced BST with RCU and fine-grained locks. Parallel algorithms and architectures. ACM, 1993, pp. 261–270. [19] T. Brown and J. Helga, “Non-blocking k-ary search trees,” in Principles of Distributed Systems. Springer, 2011, pp. 207–221. 8 CONCLUSIONS [20] N. Shafiei, “Non-blocking patricia tries with replace operations,” in Distributed Computing Systems (ICDCS), 2013 IEEE 33rd Interna- In this paper, we present a set of concurrent quadtree tional Conference on. IEEE, 2013, pp. 216–225. algorithms–quadboost, which support concurrent insert, re- [21] T. Brown, F. Ellen, and E. Ruppert, “A general technique for non- blocking trees,” in ACM SIGPLAN Notices, vol. 49, no. 8. ACM, move, contain, and move operations. In the remove opera- 2014, pp. 329–342. tion, we decouple physical updates from the logical removal [22] F. Ellen, P. Fatourou, J. Helga, and E. Ruppert, “The amortized to improve concurrency. The continuous find mechanism complexity of non-blocking binary search trees,” in Proceedings analyses flags on nodes to decide whether to move down or of the 2014 ACM symposium on Principles of distributed computing. ACM, 2014, pp. 332–340. up. Further, our LCA-based move operation can modify two [23] N. G. Bronson, J. Casper, H. Chafi, and K. Olukotun, “A practical pointers with atomicity. The experimental results demon- concurrent binary search tree,” in ACM Sigplan Notices, vol. 45, strate that quadboost outperforms existing one-dimensional no. 5. ACM, 2010, pp. 257–268. [24] J. Besa and Y. Eterovic, “A concurrent red–black tree,” Journal of tree structures while maintaining a two-dimensional hierar- Parallel and Distributed Computing, vol. 73, no. 4, pp. 434–449, 2013. chy. The quadboost algorithms are scalable with a variety of [25] T. Crain, V. Gramoli, and M. Raynal, “A contention-friendly binary workloads and thread counts. search tree,” in Euro-Par 2013 Parallel Processing. Springer, 2013, pp. 229–240. [26] ——, “A speculation-friendly binary search tree,” Acm Sigplan REFERENCES Notices, vol. 47, no. 8, pp. 161–170, 2012. [27] A. Prokopec, N. G. Bronson, P. Bagwell, and M. Odersky, “Concur- [1] M. Moir and N. Shavit, “Concurrent data structures,” Handbook of rent tries with efficient non-blocking snapshots,” in Acm Sigplan Data Structures and Applications, pp. 47–14, 2007. Notices, vol. 47, no. 8. ACM, 2012, pp. 151–160. [2] N. Shavit, “Data structures in the multicore age,” Communications [28] M. Arbel and H. Attiya, “Concurrent updates with rcu: Search of the ACM, vol. 54, no. 3, pp. 76–84, 2011. tree as an example,” in Proceedings of the 2014 ACM symposium on [3] T. David, R. Guerraoui, and V. Trigonakis, “Asynchronized concur- Principles of distributed computing. ACM, 2014, pp. 196–205. rency: The secret to scaling concurrent search data structures,” in Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems. ACM, 2015, pp. 631–644. [4] F. Ellen, P. Fatourou, E. Ruppert, and F. van Breugel, “Non- blocking binary search trees,” in Proceedings of the 29th ACM SIGACT-SIGOPS symposium on Principles of distributed computing. ACM, 2010, pp. 131–140. [5] S. V. Howley and J. Jones, “A non-blocking internal binary search tree,” in Proceedings of the twenty-fourth annual ACM symposium on Parallelism in algorithms and architectures. ACM, 2012, pp. 161–171. [6] A. Natarajan and N. Mittal, “Fast concurrent lock-free binary search trees,” in ACM SIGPLAN Notices, vol. 49, no. 8. ACM, 2014, pp. 317–328. [7] A. Ramachandran and N. Mittal, “Castle: fast concurrent internal binary search tree using edge-based locking,” in Proceedings of the 20th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. ACM, 2015, pp. 281–282. [8] ——, “A fast lock-free internal binary search tree,” in Proceedings of the 2015 International Conference on Distributed Computing and Networking. ACM, 2015, p. 37. 15
APPENDIX Definition 2. The pre-conditions of findCommon(il, rl, lca, rOp, .1 Basic Invariants iOp, iPath, rPath, oldKeyX, oldKeyY, newKeyX, newKeyY) in- voked at Ti: We provide a detailed proof of quadboost in this section. We follow the same naming convention in the paper. There 1) If iP ath is not empty, il was on the direction d ∈ are four kinds of basic operations in quadboost, i.e. the insert {nw, ne, sw, se} of the top node in iP ath at Ti1 ≤ Ti. operation, the remove operation, the move operation, and 2) If rP ath is not empty, rl was on the direction d ∈ the contain operation. Other functions are called subroutines, {nw, ne, sw, se} of the top node in rP ath at Ti1 ≤ Ti. which are invoked by basic operations. The insert operation The post-conditions of findCommon(il, rl, lca, rOp, iOp, iPath, and the remove operation only operate on one terminal rPath, oldKeyX, oldKeyY, newKeyX, newKeyY) that returns two node, whereas the the move operation operates two differ- tuples hil, iOp, iP athi and hrl, rOp, rP athi at Ti: ent terminals–one for inserting a node with newKey, the other is for removing a node with oldKey. We call them 1) rl was a Leaf or an Empty node. newKey’s terminal and oldKey’s terminal, and we call their 2) il was a Leaf or an Empty node. parents newKey’s parent and oldKey’s parent accordingly. 3) At some Ti1 < Ti, the top node in rP ath has contained Moreover, we name a CAS that changes a node’s op a flag rOp. operation and a CAS that changes a node’s child a replace 4) At some Ti1 < Ti, the top node in iP ath has contained
operation. We define snapshotTi as the state of our quadtree iOp. at some time Ti. 5) At some Ti1 < Ti, the top node in rP ath has contained rl. .1.1 Subroutines 6) At some Ti1 < Ti, the top node in iP ath has contained To begin with, we have following observations from quad- il. boost. 7) If iOp was read at Ti1, and il was read at Ti2, then Ti1 < Ti2 < Ti. Observation 1. The key field of a Leaf node is never changed. 8) If rOp was read at Ti1, and rl was read at Ti2, then op The field of a Leaf node is initially null. Ti1 < Ti2 < Ti. Observation 2. The space information–hx, y, w, hi of an Inter- 9) For each node n in rP ath, size(rP ath) ≥ 2, nt is nal node is never changed. on the top of nt−1, and nt was on the direction d ∈ {nw, ne, sw, se} of nt−1 at Ti1 ≤ Ti. Observation 3. The root node is never changed. 10) For each node n in iP ath, size(iP ath) ≥ 2, nt is Based on Observation 3, we derive a Corollary as fol- on the top of nt−1, and nt was on the direction d ∈ lows: {nw, ne, sw, se} of nt−1 at Ti1 ≤ Ti. Corollary 1. Two nodes in quadtree must share a common search By observing the find function and the findCommon path starting from the root. function, we have following lemmas. Using these observations, we prove that each sub- Lemma 1. At the first time calling the find function, the contain routine satisfies their specific pre-conditions and post- operation, the insert operation, and the remove function start with conditions. Because all basic operations invoke the find l as an Internal node and an empty path. function at line 134 that returns hl, pOp, pathi and Proof. The contain operation at line 97, the insert operation the findCommon function at line 270 that returns at line 105 and the remove operation at line 161 start with hil, iOp, iP ath, rl, rOp, rP athi, we prove that the two func- the root node by l = root, an Internal node that is never tions satisfy their pre-conditions and post-conditions before- changed (Observation 3). snapshot hand. We suppose that they execute from a Ti to Besides, the contain operation at line 95, the insert oper- derive these conditions. ation at line 103, and the remove operation at line 160 start Definition 1. The pre-condition of find(l, pOp, path, keyX, keyY) with an empty stack to record nodes. invoked at Ti: Lemma 2. At the first time the move operation calls the findCom- 1) If path is not empty, l was on the direction d ∈ mon function, it starts with rl as an Internal node and rP ath and iP ath {nw, ne, sw, se} of the top node in path at Ti1 ≤ Ti. are empty. The post-conditions of find(l, pOp, path, keyX, keyY) that Proof. The move operation begins with the root node at rl = root returns a tuple hl, pOp, pathi at Ti: line by that is never changed (Observation 3) at line 204. Both rP ath and iP ath are initialized as empty 1) l is a Leaf node or an Empty node. stacks at line 199. 2) At some Ti1 < Ti, the top node in path has contained pOp. Lemma 3. All nodes pushed by the find function (line 134) and the continueFind function (line 122) are Internal nodes. 3) At some Ti1 < Ti, the top node in path has contained l. 4) If pOp was read at Ti1, and l was read at Ti2, then Proof. Before pushing into the stack, the find function at Ti1 < Ti2 < Ti. line 135 first check the class of a node. Also, the contin- 5) For each node n in path, size(path) ≥ 2, nt is on ueFind function calls the find function at line 132 to push the top of nt−1, and nt was on the direction d ∈ nodes into path. Thus, nodes other than Internal cannot be {nw, ne, sw, se} of nt−1 at Ti1leqTi. pushed. 16
Lemma 4. All nodes pushed by the findCommon function loop more than once to push Internal nodes into path. In (line 270) the continueFindCommon function (line 285) are In- the last iteration rl is pushed at line 272, and the parent of ternal. it has already been push at the prior iteration. Then, rl is read from at line 278. Hence, rl at line 279 is a child of the Proof. Before pushing into the stack, the findCommon func- top node in rP ath if it is not empty. iP ath is empty since it tion first checks the class of a node at line 271. Also, starts with an empty stack. the continueFindCommon function calls the findCommon Therefore, initially the pre-conditions are satisfied. function at line 342. Or it calls the find function at line 323 The post-conditions: and line 302 to push Internal nodes according to Lemma 3. The first condition always satisfies as the class of a node Thus, nodes other than Internal cannot be pushed. is checked at line 135. Lemma 5. For the loop in the find function at line 134 and the By Lemma 1, the contain operation, the insert operation, findCommon function at line 270, we suppose that path(rP ath) and the remove operation start with an empty stack. Thu is empty and refer l(rl) and pOp(rOp) to each field updated by Lemma 5 indicates that post-conditions are satisfied. the loop. If the loop executes at least once and breaks at Ti, l(rl), For the find function within the findCommon function, pOp(rOp), and path(rP ath) satisfy following conditions: by Lemma 5 it satisfies post-conditions when it breaks out. Hence, at line 282, post-conditions are satisfied because 1) l(rl) is a Leaf node or an Empty node. iP ath is empty. At line 279, if rl starts as a Leaf node, 2) At some Ti1 < Ti, the top node in path(rP ath) has the post-conditions are also satisfied. Otherwise, there are contained pOp(rOp). two parts of rP ath at the time the loop breaks out, where 3) At some Ti1 < Ti, the top node in path(rP ath) has we have proved each of them satisfy post-conditions by contained l(rl). Lemma 5. Moreover, by the pre-condition, rl has been a 4) If pOp(rOp) was read at Ti1, and l(rl) was read at Ti2, child of the top node in rP ath of the find function. Thus, all then Ti1 < Ti2 < Ti. nodes pushed later also satisfy the post-conditions. 5) For each node n in path(rP ath), size(path) ≥ 2, nt We have proved the base case. Then we assume that is on the top of n , and n was on the direction d ∈ t−1 t for invocations find0, find1, find2...findk, the first k in- {nw, ne, sw, se} of nt−1 at Ti1 ≤ Ti. vocations satisfy the pre-conditions and post-conditions. Proof. Apart from the terminate condition that the findCom- We should prove that findk+1 also satisfy the conditions. mon function judges whether two directions are the same We have to consider all places where the find function is or not, it performs in the same pattern as the find function. invoked. Both functions first push the previous Internal node l0 into The pre-condition: path, then get its op, and read a child pointer l finally. First we consider the contain operation, the insert oper- The first condition always holds because by Lemma 4 ation, and the move operation. and Lemma 3 Leaf nodes and Empty nodes will never be At line 98, line 106, line 163, the invocations follow the pushed into the path(rP ath). base case which we have proved satisfy the pre-condition. We then prove part 2-4. At the last iteration, pOp(rOp) At line 132, there are two scenarios l is read. l could and l(rl) are read from l0, which has already been pushed either be read from the path at line 126 or be assigned to its into the path(rP ath). Hence, part 2 and part 3 are correct. parent node at line 123. In the latter case, p was at the top of Besides, pOp(rOp) is read before l(rl) so that part 4 holds. path at Ti1 < Ti (line 109 and line 166) by the hypothesis. Otherwise l is assigned to a node in path. As there’s no other For the last part, we assume that at Ti1 the lemma holds, 0 push operations, n on the top of n must be its child at so at Ti2 > Ti1 l is read from l which has already been t t−1 pushed into the path(rP ath). As path(rP ath) is initially some Ti1 < Ti by the hypothesis. Then, we discuss the move operation. empty, and Ti is at the end of the last iteration, l which on the top of l0 has been a child of it. At line 279, the find function is wrapped by the find- Common function. Since Lemma 5 shows that the loop set Lemma 5 proves loop within the find function and the rl as a child of the top node in rP ath, we have to prove findCommon function satisfy post-conditions in Definition 1 that either it is an Internal node or an Leaf node that was a and Definition 2. Next parts are going to show that other child of the top node before entering the loop. At line 212, statements do not change these properties. We consider the findCommon function follows the base case. At line 342, the first invocation as the base case, and prove lemmas by Corollary 1 shows that the root node will always be a LCA induction. node. Hence, after setting rP ath to its lca index at line 334, rl Lemma 6. Every call to the find function satisfies its pre- it contains at least one node. By Lemma 3 and Lemma 4, rP ath condition and post-conditions. is an Internal node popped from at line 337. At line 282, the find function is also wrapped by the Proof. Now we prove the base case. We prove that initially findCommon function. We could prove that iP ath is empty. the lemma holds. At line 212, the findCommon function follows the base case. The pre-condition: At line 342, iP ath is set to empty before the invocation By Lemma 1, the contain operation, the insert operation, (line 335). and the remove operation start with an empty stack to At line 302, rl is either assigned to rp (line 288) or a node record nodes. popped from rP ath (line 295). If it is popped from rP ath, For the move operation, by Lemma 2 it starts with rl as our hypothesis ensures that it was a child of the top node an Internal node and empty paths. Therefore it executes the from rP ath. If it is read from rp, which is an Internal node 17
popped from rP ath at line 214, line 344, or line 305, it also The findCommon function wraps two find functions at satisfies the claim. line 282 and line 279. Because hrl, rOp, rP athi that passed At line 323, likewise, il is either assigned to ip (line 312) from the find function for rl and hil, iOp, iP athi that passed or a node popped from iP ath (line 315). If it is popped from from the find function for il hold the post-conditions, the iP ath, our hypothesis ensures that il was a child node of the lemma is true. top node from iP ath if it is not empty. If it is read from ip, After proving the pre-conditions and post-conditions of which is an Internal node popped from iP ath at line 213, the find function and the findCommon function, we shall line 345, or line 326, it also satisfies the claim. prove that the continuous find mechanism (i.e. the con- The post-conditions: tinueFind function and the continueFindCommon function) The first condition always holds. Because by Lemma 3, satisfies its pre-conditions and post-conditions. if l is an Internal node, it will be pushed into path. We then prove other conditions. Definition 3. The pre-conditions of continueFind(pOp, path, l, First we consider the contain operation, the insert oper- p): ation, and the move operation. 1) l was child of p. At line 98, line 106, line 163, the invocations follow the 2) p was child of the top node in path if it is not empty. base case which we have proved satisfy the post-conditions. At line 132, there are two scenarios l is read. l could The post-conditions are the same as the find function. either be read from the path at line 126 or be assigned to Definition 4. The pre-conditions of continueFindCommon(il, rl, its parent node at line 123. In both cases, l was an Internal lca, rOp, iOp, iPath, rPath, ip, rp, oldKeyX, oldKeyY, newKeyX, node in path. Therefore Lemma 5 indicates that all nodes newKeyY, iFail, rFail, cFail): followed by l satisfy the post-conditions. Then, we discuss the move operation. 1) At least one of iF ail, rF ail or cF ail is true. At line 279, the find function is wrapped by the findCom- 2) il was a child of ip. mon function. The pre-condition indicates rl was a child of 3) ip was a child of the top node in iP ath if it is not empty. rP ath if it is not empty. Hence, it suffices to show that the 4) rl was a child of rp. post-conditions are satisfied by Lemma 5. 5) rp was a child of the top node in rP ath if it is not empty. At line 282, the find function is also wrapped by the findCommon function. The pre-condition indicates iP ath The post-conditions are the same as the findCommon function. is an empty. In this way, by Lemma 5 we prove the post- We now prove that the continueFind function satisfies its conditions. pre-conditions and post-conditions. At line 302, likewise, the pre-condition suggests that rl was an Internal node in rP ath. By Lemma 5 we also prove Lemma 8. Every call to the continueFind function satisfies its the post-conditions. pre-condition and post-conditions. At line 323, the pre-condition suggests that il was an Proof. The post-conditions: Internal node in iP ath. Therefore, by Lemma 5 we prove At line 132 of the continueFind function, it executes the post-conditions. the find function. Because the find operation satisfies its post-conditions by Lemma 6, the continueFind function also After showing the pre-condition and post-conditions of satisfies the same conditions. the find function, we the prove that the findCommon func- The pre-condition: tion that returns hil, iOp, iP ath, rl, rOp, rP athi satisfies its The continueFind function is invoked at line 119 and pre-conditions and post-conditions. line 176. Lemma 7. Every call to the findCommon function satisfies its Initially, the continueFind function follows the find func- pre-conditions and post-conditions. tion at line 106 and line 163. For the first part, the post- condition of the find function shows that l was a child of Proof. The pre-condition: p which popped at line 109 or line 166(Lemma 5). For the We prove the lemma by induction. First we prove the second part, after popping p, it becomes a child of the top base case. node in path if it is not empty. At line 212, the findCommon function is initially in- Otherwise, it reads results from the continueFind invo- iP ath rP ath voked, and both and are empty (line 199). cation at the prior iteration. As the post-conditions of the findCommon , Then we assume that for invocations 0 continueFind function is the same as the find function, we findCommon , findCommon findCommon 1 2 ... k, the first prove the lemma. k invocations satisfy the pre-conditions. We prove that findCommonk+1 also satisfies the conditions. We now prove that the continueFindCommon function There are two places the findCommon function is in- satisfies its pre-conditions and post-conditions. voked. At line 212, the base case proves the post-conditions. Lemma 9. Every call to the continueFindCommon function At line 342, Corollary 1 shows that the root node will always satisfies its pre-conditions and post-conditions be a LCA node. Hence, after setting rP ath to its lca index at line 334, it contains at least one node. By Lemma 3 and Proof. The post-conditions: Lemma 4, rl is an Internal node popped from rP ath at There are three cases, the first part of the pre-condition line 337. Hence, we prove the pre-condition. shows that it must enter one of the loop. We consider each The post-conditions: case by the program execution order. 18
1) If rF ail is true and cF ail is false, it executes the For last two parts, since the compress function reads case starts from line 236. If cF ail is not set to true, result following find(keys), it suffices to prove the post- it executes the find function at line 302. Because the conditions of find(keys) satisfy the preconditions. By find function satisfies its post-conditions (Lemma 6), Lemma 8, Lemma 9, Lemma 6, and Lemma 7, the post- the lemma holds. If iF ail is true, it comes to the conditions of the find function, the findCommon function, second part of the proof. If cF ail is set to true, it the continueFind function, and the continueFindCommon comes to the third part of the proof. function establish the lemma. 2) If iF ail is true and cF ail is false, it executes the Corollary 2. Every call to the check function satisfies its pre- case starts from line 235. If cF ail is not set to true, conditions that p is an Internal node. it executes the find function at line 323. Because the find function satisfies its post-conditions, part 1 and Definition 6. For moved(node) invoked at Ti, it has following Lemma 6 indicate the Lemma holds. If cF ail is true, the pre-conditions that node is a Leaf node or an Empty node. it comes to the third part of the proof. Lemma 11. Every call to the moved function satisfies its pre- 3) If cF ail is true, it executes the case starts from condition. line 240. Because the findCommon function satisfies its post-conditions (Lemma 7), it establishes this part Proof. The moved function is called at line 108, line 165, of the lemma. line 283, line 280, line 324, line 303. At line 108, it reads l from line 106 or line 119. Hence, The pre-condition: according to Lemma 6, l is a Leaf node or an Empty node. The continueFindCommon function is invoked at At line 165, it reads l from line 163 or line 176. Hence, line 243. according to Lemma 6, l is a Leaf node or an Empty node. For the first condition, line 216, line 217, line 218, line 232, At line 283 and line 280, it reads il from line 282 and rl line 235, line 236, line 240 indicate that at least on of three from line 279. Therefore, according to Lemma 6, il and rl flags is set to true. are Leaf node or Empty nodes. For the next conditions, initially the continueFindCom- At line 324, it reads il from line 323. Therefore, according mon follows the findCommon function at line 212. As ip to Lemma 6, il is a Leaf node or an Empty node. and rp are popped at line 213 and line 214, il was a child of At line 303, it reads rl from line 302. Therefore, according ip and rl was a child of rp. Further, ip and rp was a child of to Lemma 6, rl is a Leaf node or an Empty node. the top node in iP ath and rP ath. All these claims are based op on Lemma 8. Otherwise, the continueFindCommon function Based on these post-conditions, we show that each T reads results from its invocation at the prior iteration. In created at i store their corresponding information. such a case, our post-conditions prove the pre-conditions. Lemma 12. For op created at Ti: 1) If op is a Substitute object, op.parent has contained op.oldChild, a Leaf node, at T < T from the result of We observe that the find function, the continueFind i1 i find(keys) at the previous iteration. function, the findCommon function, and the continueFind- 2) If op is a Compress object, op.grandparent has con- Common function return a similar pattern of results. The tained op.parent, an Internal node, at T < T from find function and the continueFind function return a tuple i1 i the result of find(keys) at the previous iteration. hl, pOp, pathi. The findCommon function and the contin- 3) If op is a Move object, op.iP arent has contained ueFindCommon function return two such tuples. Hence We op.oldIChild, a Leaf node, op.rP arent has contained use find(keys) to denote such a set of find operations for op.oldRChild, a Leaf node, op.iP arent has contained searching keys. Further, we number find(keys) by their op.oldIOp, and op.rP arent has contained op.oldROp invocation orders. Invocations at line 106, line 163, and before Ti from the result of find(keys) at the previous line 212 are at iteration0. Invocations at line 106, line 163, iteration. and line 212 are at iterationi, 0 < i. Proof. 1) A Substitute object is created at line 112 or Definition 5. For compress(path, p) invoked at T , it has i line 168 by assigning hp, l, newNodei. Since p read following pre-conditions: from the top node in path at line 109 or line 166, and 1) p is an Internal node. l is returned by the find function or findCommon 2) path is read from the result of find(keys) at the prior function, the post-conditions of them establish the iteration. claim according to Lemma 6, Lemma 7, Lemma 9, 3) If path is not empty, p was a child node of the top node and Lemma 8. in path at Ti1 < Ti. 2) A Compress object is created at line 185 by assigning hpath, pi. p is passed from the top of path at line 166 Lemma 10. Every call to the compress function satisfies its pre- or line 214, the post-conditions of find(keys) and conditions. the property of path establishes the claim (Lemma 6, Lemma 7, Lemma 9, Lemma 8, Lemma 3, and Proof. The compress function is called at line 171 or line 229. Lemma 4). At line 171, p is a node popped from path (line 166. At 3) A Move object is created at line 224 by assigning line 229, rp is an Internal node popped from rP ath (line 214, hip, rp, il, rl, newNode, iOldOp, rOldOpi. ip is as- line 305 or line 344. This proves the first part. signed to the top node of iP ath at line 213, rp is 19
assigned to the top node of rP ath at line 214, il was Lemma 14. Every call to the helpFlag function satisfies its pre- a child of the top node of iP ath, rl was a child of conditions. the top node of rP ath, and iOp and rOp are read Proof. The helpFlag function is invoked at line 113, line 169, from ip and rp by the post-conditions of the find- line 186, line 226, line 248-line 250, and line 261-line 267. Common function and the continueFindCommon At line 113 and line 169, the post-conditions of the function (Lemma 9 and Lemma 7. find function and the continueFind function (Lemma 6 and Lemma 8 imply that p is an Internal node popped at line 109 Then we prove the pre-conditions of other functions and has contained pOp. which use the result of above functions to modify quadtree. At line 186, the pre-condition of the compress function The following lemmas satisfy a basic pre-condition that (Lemma 10 shows that p and nodes from path are read from their arguments are the same type as indicated in algo- find(keys) at the prior iteration, and pOp is read at line 181. rithms. Thus, p has pOp at Ti1 < Ti after reading p. At line 226, ip and rp are popped from iP ath at line 213 Lemma 13. 1) Every call to the help function satisfies its and line 214 or read from the continueFindCommon func- pre-conditions. tion. By Lemma 7 and Lemma 9, either op.iP arent has 2) Every call to the helpSubstitute function satisfies its pre- contained op.iOp or op.rP arent has contained op.rOp. conditions. At line 248-line 250 and line 261-line 267, by the pre- 3) Every call to the helpCompress function satisfies its pre- condition of the helpMove function 13, op is a Move object. conditions. Therefore, according to Lemma 12, either op.iP arent has 4) Every call to the helpMove function satisfies its pre- contained op.iOp or op.rP arent has contained op.rOp. conditions. Lemma 15. Every call to the hasChild function satisfies its pre- Proof. 1) The help function is called at line 118, conditions that parent is an Internal node. line 175, line 311, line 287, line 333, and line 332. At line 118 and line 175, pOp might be obtained Proof. According to Lemma 11, node is a Leaf node or an from the result of the find function and the contin- Empty node. According to Lemma 14, flag operations only ueFind function. Hence, the post-conditions of them perform on Internal nodes. Therefore line is the only place ensures that pOp is an Operation object read from p. that set a Move op on a Leaf node. By Observation 1, op is Otherwise, pOp might be obtained from line 116, or initially null. After checking the condition, op is assigned line 173 by reading a node’s op. to a Leaf node where op.iP arent is an Internal node by At line 311 and line 287, rOp and iOp are passed Lemma 12 the move operation. The post-conditions of the con- Definition 8. For helpReplace(p, oldChild, newChild) in- tinueFindCommon function and the findCommon voked at T , it has the pre-conditions: function (Lemma 7 and Lemma 9), with line 235, i line 236, and line 240 guarantee that arguments’ 1) p, oldChild, and newChild are read from the same op. type are the same. 2) newChild is a node that has not been in quadtree. At line 333 and line 332, rOp and iOp could be 3) p is an Internal node that has contained oldChild at passed from the move function or read from ip and Ti1 < Ti. rp in the continueFindCommon function at lines Lemma 16. Every call to the helpReplace function satisfies its that have been mentioned. pre-conditions. 2) The helpSubstitute function is called at line 114, line 170, and line 150. At line 114 and line 170, op Proof. For the first condition: is created at line 112 or line 168 respectively. At The helpReplace function is invoked at line 145, line 193, line 150, it checks whether op is Substitute before line 254-259. calling the helpSubstitute function. By Lemma 13, at line 193, it reads a Compress object; at 3) The helpCompress function is called at line 185, line 145, it reads a Substitute object; at line 254-259. it reads line 317, line 297, and line 339. For each invocation, a Move object. it checks whether op is Compress beforehand. For the second condition: 4) The helpMove function is invoked at line 228, Since part 1 illustrates that p, oldChild, and newChild line 239, and line 151. At line 228 and line 239, op are read from the same op, if op is a Substitute object, is created at line 224. At line 151, it checks whether op.newNode is created at line 111 or line 161; if op is op is Move before the invocation. a Compress object, each time the helpCompress function creates a new Empty node at line 193; if op is a Move object, op.newIChild is created at line 220, and an empty node is Corollary 3. For help(op), op is read from an Internal node. created at line 145. Definition 7. For helpF lag(p, oldOp, newOp) invoked at Ti, For the third condition: it has the pre-conditions: Since part 1 illustrates that p, oldChild, and newChild are read from the same op, Lemma 12 shows that p has 1) p is an Internal node. contained oldChild. 2) For cflag, p has contained pOp at Ti1 < Ti after reading p; otherwise, p has contained pOp at Ti1 < Ti Definition 9. For createNode(l, p, newKeyX, newKeyY, value) from the result of find(keys) at the prior iteration. that returns a newNode invoked at Ti, it has the pre-conditions: 20
1) p is an Internal node. 2) p has contained l at T < T from the result of i1 i Next we illustrate that there’s no ABA problem on any find(keys) at the prior iteration. op. That is to say, in terms of an Internal node n, its opi at post-conditions: Ti has not appeared at Ti1 < Ti, opi1 =6 opi. We prove the The newNode returned is either a Leaf node with newKey following lemma: and value, or a sub-tree that contains both l.key node and n op newKey node with the same parent. Lemma 19. For an Internal node , it never reuse an that has been set previously. Lemma 17. Every call to the createNode function satisfies its pre-conditions and post-condition. Proof. If a node’s op is set to opi at Ti, it has never been ap- peared before. We prove the lemma by discussing different Proof. The createNode function is invoked at line 111 and types of flag operations. line 221. For iflag, a Substitute op is created at line 112 before its At line 111, the post-conditions of the find function and invocation. For rflag, a Substitute op is created at line 168 the continueFind function ensure that l has been a child of p before its invocation. For a cflag, a Compress op is created that popped from path at line 109 (Lemma 6 and Lemma 8). at line 185 before its invocation. For mflag, a Move op At line 221, the post-conditions of the findCommon is created at line 224 before its invocation. Because each function and the continueFindCommon function ensure that newOp is newly created, it has not been set before. For l has been a child of p popped at line 213. (Lemma 7, unflag, each time it creates a new Clean op to replace the Lemma 9). prior Move op or Substitute op by Lemma 18. Therefore, the Though we have not presented the details of the cre- new Clean op has never appeared before. ateNode function, our implementation guarantee that the post-condition must be satisfied. We have shown that for each successful flag operation, it never set an op that has been used before. Every Internal .1.2 Flag and replace operations node is initialized with Clean op by Lemma 18 and we We argue that each successful CAS operation occurs in a use a flag operation to change op at the beginning of each correct order. At outset, we argue that CAS behaviours for CAS transition. We say successive replace operations and each helpFlag function. A flag operation involves three ar- unflag operations that read the op belongs to it. According to guments: node, oldOp, newOp. A replace operation involves Figure 9, there are threes categories of successful CAS transi- three arguments: p, oldChild, newChild. From Figure 9, tions. We denote them as flag → unflag, flag → replace, we denote each flag operation accordingly. iflag occurs in and flag → replace → unflag. the insert operation, rflag occurs in the remove operation, Let flag0, flag1, ..., flagn be a sequence of successful and mflag occurs in the move operation. Moreover, we flag operations; let unflag0, unflag1, ..., unflagn be a specify a flag operation which attaches a Clean op on a sequence of successful unflag operations. flagi attaches node as an unflag operation. We call ireplace as the replace opi, and replacei and unflagi read opi and come after it. operation occurs in the insert operation, rreplace as the Therefore we say flagi, replacei, and unflagi belong to the replace operation occurs in the remove operation, mreplace same op. In addition, if there are more than one replace op- 0 as the replace operation occurs in the move operation, and erations belong to the same opi, we denote them as replacei , 1 k creplace as the replace operation occurs in the compress replacei , ..., replacei ordered by their successful sequence. operation. The next lemmas describe the behaviours of the Similarly, if there are more than one flag operations that 0 helpFlag function according to the state transition diagram. belong to opi on different nodes, we denote them as flagi , 1 k flagi , ..., flagi . A similar notation is used for unflagi. Observation 4. op is only attached on an Internal node. The following lemmas prove the correct ordering of three Lemma 18. For a new node n: different transitions. 1) It is created with a Clean op. Lemma 20. 1) ireplace will not be done before the success 2) rflag, iflag, mflag or cflag succeeds only if n’s op of iflag which belongs to the same op. is Clean. 2) rreplace will not be done before the success of rflag 3) unflag succeeds only if n’s op is Substitute or Move. which belongs to the same op. n op op 4) Once ’s is Compress, its will never be changed. Proof. For the insert operation and the remove operation, at Proof. 1) As shown at line 71, Internal nodes are as- line 145 the helpReplace function is wrapped by the help- signed a Clean op when they are created. Substitute function. The helpSubstitute function is called at 2) Before iflag, op is checked at line 110; before rflag, line 114, line 170, and line 150. At line 114, it is called after op is checked at line 167; before cflag, op is checked the success of iflag at line 113. At line 170, it is called after at line 182. For the move operation, before mflag the success of rflag at line 169. At line 150, by Corollary 3 on oldKey’s parent and newKey’s parent, they are and Lemma 18, there must have been iflag or rflag that checked at line 217 and line 216. change the node’s op from Clean to Substitute. 3) unflag is called at line 146 and line 261-line 267, Lemma 21. creplace will not be done before the success of cflag where the pre-conditions specify that op is Move or which belongs to the same op. Substitute according to Lemma 13. 4) By part 3, there’s no unflag operation happens on Proof. The helpCompress function which wraps creplace an Internal node with a Compress op. is called at line 185, line 317, line 297, and line 339, and 21
k k line 149. At line 185, the helpCompress function follows (Lemma 19). If flag happens after flagi , it contra- the successful helpFlag function at line 186. At line 317, dicts Lemma 18 that a flag operation starts Because k k k line 297, and line 339, a pOp is read from Internal nodes. At flag is done after reading opi.oldChild , replace line 149, Lemma 3 shows that op is read from an Internal is also after reading it. Therefore, if flagk succeeds, k node. Hence, by Lemma 18, successful cflag must have it contradicts part 3 that replacei is the first success- k changed the node’s op from Clean to Compress. ful replace operation after reading opi.oldChild . 5) Based on Lemma 14, if there is another unflag Lemma 22. mreplace will not be done before the success of k changes p.op, unflagi that use opi as the oldOp will mflag which belongs to the same op. k fail. This contradicts to the definition that unflagi Proof. The helpMove function which wraps two mreplaces is a successful unflag operation that reads opi. is called at line 228, line 239, and line 151. At line 228 and 6) We consider each operation separately. line 239, the Move op is created at line 267. Otherwise, at For the insert operation and the remove operation, line 151, by Corollary 3, op is read from an Internal node. unflag follows replace immediately at line 146. k k Hence, Lemma 18 demonstrates that a successful mflag Assuming that replacei occurs after unflagi , there must have changed the node’s op from Clean to Move. must be another replace reads opi and changes the link. It contradicts part 3 of the proof that replacek From the above lemmas, we have the following corol- i is the first successful replace operation that belongs lary: to opi. Corollary 4. A successful replace operation which belongs to an For the move operation, because we assume that op will not succeed before it’s flag operations have been done. replace operations exist, doCAS is set to true before performing replace operations. An unflag operation We discuss each CAS transition accordingly. We be- follows replace operations from line 261-267. Con- gin by discussing the flag → replace → unflag sider if a replacek, 0 ≤ k < |replace | happens after transition. We refer holdIOp, oldIChild, newIChildi and i i any unflag operation, then another replace which holdROp, oldRChild, newRChildi to different tuples of use the same op must have succeed as the program holdOp, oldChild, newChildi. order specifies that replace execute before unflag k Lemma 23. flag → replace → unflag occurs when rflagi, for a process. Hence, replacei fails and contradicts iflagi or mflagi succeeds, and it has following properties: the definition. Therefore all successful unflag operations occur af- 1) replace never occurs before flag . i i ter replace. 2) flagk, 0 ≤ k < |flag | is the first successful flag i i 7) According to Corollary 4, replacek follows flagk operation on op .parentk after T when pOp is read. i i i i1 i belongs to op . Suppose there exists replace af- 3) replacek, 0 ≤ k < |replace | is the first success- i i i ter reading op .parentk and before replacek. Then, ful replace operation on op .parentk after T when i i i i2 there must be flag that precedes replace by Corol- op .oldChildk is read. i lary 4. By part 1, flag is the first successful flag op- 4) replacek, 0 ≤ k < |replace | is the first successful i i i eration after reading op . By Lemma 14, op is read replace operation on op .parentk that belongs to op . i i i i before op .l. Hence, there does not exist any replace 5) unflagk, 0 ≤ k < |unflag | is the first successful i i i operation between T reading op .oldChildk and unflag operation on op .parentk after flagk. i1 i i i flagk. If flag happens between flagk and replacek, 6) There is no successful unflag operation occurs before i i i it will also fail because op .parentk.op is not Clean. replace . i i Then, the first replace operation that belongs to op 7) The first replace operation on op .parentk that belongs i i must succeed. to opi must succeed. Proof. 1) Corollary 4 proves the claims. 2) By Lemma 14, for each flag operation, p has con- We then discuss the ordering of the flag → replace tained pOp at some time Ti1. Suppose that there’s transition. another flagk that occurs after T but before flagk, i1 i Lemma 24. flag → replace occurs only when cflag succeeds, then flagk fails because Lemma 19 shows that op is i i it has following properties: not reused. Therefore, it contradicts to the definition flagk that i is a successful flag operation. 1) creplacei never occurs before cflagi. p 3) By Lemma 16, for each replace operation, has 2) cflagi is the first successful flag operation on oldChild T contained at some time i1. Suppose opi.parent after Ti1 when rOpi is read. replace T that there’s another that occurs after i1 3) creplacei is the first successful replace operation on replacek replacek but before i , then i fails because opi.grandparent after Ti1 when opi.parent is read. newChild Lemma 16 shows that is never a node 4) creplacei is the first successful replace operation on in the tree. Therefore, it contradicts to the definition op .grandparent that belongs to op . k i i that replacei is a successful flag operation. 5) There is no unflag operation after creplace . k i 4) Suppose there’s replace that belongs to opi and 6) The first replace operation that belongs to op must k k i occurs before creplacei . It must follow flag ac- succeed. cording to Corollary 4. If flagk happens before k k flagi , it fails because faili will fail by consequence Proof. 1) Corollary 4 proves the claims. 22
2) By Lemma 14, for each flag operation, p has con- 3) opi.iP arent and opi.rP arent are different. tained pOp at some time T . Suppose that there’s i1 Proof. 1) We prove it by contradiction. Suppose there’s another cflag that occurs after T but before cflag , i1 i another unflag operation that reads op and suc- then cflag fails because Lemma 19 shows that op is i i ceeds before unflag on op .rP arent. It must have not reused. Therefore, it contradicts to the definition i i changed op .rP arent.op to a new op, by Lemma 19 that cflag is a successful flag operation. i i unflag will fail. This contradicts the definition of 3) By Lemma 16, for each replace operation, p has i unflag . contained oldChild at some time T . Suppose that i i1 2) For the former part of the claim, we prove it by there’s another creplace that occurs after T but be- i1 contradiction. Assuming that op .iP arent is flagged fore replace , then creplace fails because oldChild i i i on op , and will be unflagged without performing remains the same but the link has been changed to i replace operations. However, unflag operations oc- another node. Therefore, it contradicts to the defini- cur after the judgement at line 254. The first process tion that creplace is a successful flag operation. i must set doCAS to true. In the next step, it executes 4) Suppose there’s a replace operation that belongs line 254-259. According to Corollary 5, the first to op and occurs before creplace . It must follow i i replacet must succeed. Therefore it contradicts the cflag according to Corollary 4. If cflag happens i definition that there’s no replace operation before an before cflag , it fails because cfail will fail by i i unflag operation. For the latter part, by Lemma 12 consequence (Lemma 19. If cflag happens after op .iP arent has contained op .oldIOp. Because the cflag , it contradicts Lemma 18 that a flag opera- i i i first flag operation on op .iP arent fails, latter flag tion starts with a Clean op. Therefore, cflag is the i operations read the op .oldIOp also fail. same as cflag . Because cflag is done after reading i i 3) Assuming op .iP arent and op .rP arent are the op .parent, creplace is also after reading it. There- i i i same. By part 2, flag operations on op .iP arent fore, if cflag succeeds, it contradicts part 3 that i must fail, hence doCAS will never be true. creplace is the first successful replace operation i op .allF lag is always false by by Observation 5. In after reading op .parent. i i this way, line 261-267 prevents any unflag operation. 5) By Lemma 18, once a node’s op is Compress, it will This derives a contradiction. never be changed. 6) Suppose the first replace operation that belongs to opi fails, there must exist some successful replace af- .1.3 Quadtree properties ter reading opi.parent and before creplacei. Further, In the Section, we use above lemmas to show that quadtree’s there is flag that precedes replace by Corollary 4. If properties are maintained during concurrent modifications. flag happens between creplacei and cflagi, flag will fail by Lemma 18. If flag happens between Definition 10. Our quadtree has these properties: reading opi and cflagi, it contradicts part 1 that 1) Two layers of dummy Internal nodes are never changed. cflagi is the first successful flag operation after 2) An Internal node n has four children, which locate in the reading opi. Because opi is read after opi.parent, we direction d ∈ {nw, ne, sw, se} respectively according to also have to consider if flag happens before reading their hx, y, w, hi, or hkeyX, keyY i. opi. Consider if cflag happens, flagi will fail be- For Internal nodes reside on four directions: cause opi.parent.op will not be set back to Clean ac- cording to the fourth part. Consider if mflag, iflag • n.nw.x = n.x, n.nw.y = n.y; or rflag happens, it will not change the link from • n.ne.x = n.x + w/2, n.ne.y = n.y; opi.grandparent to opi.p according to Lemma 23. • n.sw.x = n.x, n.sw.y = n.y + n.h/2; • n.se.x = n.x + w/2, n.se.y = n.y + h/2, w0 n.w/ h0 n.h/ Corollary 5. The first replace operation belongs to opi must and all children have their = 2, = 2. succeed. For Leaf nodes reside on four directions: Next we discuss the flag → unflag transition, where • n.x ≤ n.nw.keyX < n.x + n.w/2, n.y ≤ replace operation occurs between a successful flag operation n.nw.keyY < n.y + n.h/2; and an unflag operation that belong to the same op. We • n.x+n.w/2 ≤ n.ne.keyX < n.x+n.w, n.y ≤ suppose that op.iF irst is true. The claim for op.iF irst = n.ne.keyY < n.y + n.h/2; false is symmetric. • n.x ≤ n.sw.keyX < n.x + n.w/2, n.y + n.h/2 ≤ n.sw.keyY < n.y + n.h; Observation 5. allF lag and iF irst in a Move op are initially • n.x + n.w/2 ≤ n.se.keyX < n.x + n.w, n.y + false, and they will never be set false after assigning to true. n.h/2 ≤ n.se.keyY < n.y + n.h. Lemma 25. For flag → unflag circle, it only results from a Lemma 26. Two layers of dummy nodes are never changed. Move object opi such that: Proof. We prove the lemma by induction. Only replace op- 1) unflagi is the first successful unflag operation on erations will affect the structure of quadtree as we define in opi.rP arent. Definition 10. 2) The first flag operation on opi.iP arent must fail, and Initially the property holds as we initialize quadtree with no later flag operation will succeed. two layers of dummy nodes and a layer of Empty nodes. 23
0 Suppose that after replacei the lemma holds, we shall Proof. Assume that there are two nodes n and n reside k prove that after replacei+1 the lemma is still true. For in the active path(keys ) with Compress op. We prove the ireplace, rrepalce, and mreplace, the pre-condition 16 lemma by contradiction. Because n and n0 are active with ensures that three nodes are from the same op. Thus, Compress op, all children of n and n0 are Empty (Lemma 28. as Lemma 12 further ensures that they only swing Leaf Since n and n0 are Internal nodes in the same path, it derives nodes and Empty node, dummy node are not changed. For a contradiction. creplace, since at replacei the property is true, the root is For the second part of the proof, if n resides in other connected to a layer of dummy nodes. Lemma 12 shows that places in path, its children should be Empty by Lemma 28. op.oldChild is an Internal node, and line 186 shows that the Therefore, only if n is on the end of path, conditions are root node’s child pointer will never be changed. Therefore, satisfied. replacei+1 will not occur on the root node. k Lemma 31. For path(keys ), if nt is active in snapshotTi , Lemma 27. Children of a node with Compress op will not be then n0, ..., nt−1 above nt are active. changed. Proof. By Lemma 29, nodes with op other than Compress are Proof. Based on Lemma 18, the node’s op will never reachable from the root. Lemma 30 indicates that a node op be changed after being set to Compress. Lemma 24 and with a Compress is on the end of the path. Therefore, op Lemma 23 indicate that for a node flagged with Compress op, other nodes do not have a Compress , which establishes only itself will be unlinked. This establishes the Lemma. the lemma. Lemma 32. If in the snapshot , n is the LCA on active path Lemma 28. Only an Internal node with all children Empty could Ti for oldKey and newKey. Then at T ,T > T , n is still be attached with a Compress object. i1 i1 i the LCA on active path for both oldKey and newKey if it Proof. Corollary 2 ensures that the pre-conditions of the is active. check function is satisfied. It checks whether all children Proof. Nodes from root to the LCA node shares a common are Empty before calling the helpFlag function at line 186. path (Observation 1. Because the LCA node is active, nodes This establishes the Lemma. above the LCA node are active. Hence, the subpath from the root to the LCA node is active by Lemma 31. Lemma 29. An Internal node whose op is not Compress is reachable from the root. Lemma 33. Nodes a with a Compress op will not be pushed into path more than once. Proof. We have to consider all kinds of replace operations.
For ireplace, rrepalce, and mreplace, by Lemma 16 and Proof. Supposing that a node n is active in snapshotTi , and Lemma 12 they use a new node to replace a Leaf node or an it becomes inactive at Ti1 > Ti. Besides, suppose that op of Empty node. Thus, Internal nodes are still reachable from the n is Compress and pushed at Ti2,Ti < Ti2 < Ti1. We prove root. that after Ti2, n will never be pushed again. For creplace, by Lemma 16 and Lemma 12, it replaces Lemma 24 shows that a node with a Compress op Internal nodes. Lemma 24 shows that it is preceded by cflag. will never be unflagged. Hence, based on Lemma 6 and Lemma 28 implies that all children are Empty nodes. There- Lemma 6), we have to prove that nodes in path with a fore, an Internal node with a Compress op and its children Compress op will not be pushed into again. For the insert are not reachable from the root only after the success of operation and the remove operation, pOp is checked at creplace. line 123 and line 127 before the next find operation. For the move operation, rOp is checked at line 312, line 316, line 339, Next we define active set and inactive set for different and iOp is checked at line 296 and line 288 before calling kinds of nodes. We say a node is moved if the moved function the findCommon function and the find function. Hence, at turns true. For an Internal node or an Empty node, if it is Ti2 < Ti1, op must be checked before find(keys) so that reachable from the root in snapshotTi , it is active; otherwise, nodes with a Compress op will not be pushed more than it is inactive. For a Leaf node, if it is reachable from the once. root in snapshotTi and not moved, it is active; otherwise, k Lemma 30 shows that for active path, active nodes with it is inactive. We denote path(keys ), 0 ≤ k < |replacei| as a stack of nodes pushed by find(keys) in a snapshot. a Compress op will only reside at the end. If other nodes We define physical path(keysk) to be the path for keysk pushed before n will become inactive, we can pop them beforehand until reaching the last node an op other than in snapshotTi , consisting of a sequence of Internal nodes with a Leaf node or an Empty node at the end, which Compress. is the actual path in the snapshot. We say a subpath of Lemma 34. After the invocation of find(keys) which reads k path(keys ) is an active path if all nodes from the root lt, there is a snapshot, where the path from the root to lt is k to node n ∈ path(keys ) are active due to their pushing physical path(keyst). order. Hence a physical path with a Leaf node is active only if path(keysk) is an active path and the Leaf node is active. Proof. We prove the lemma by induction. In the base case, where path starts from the root node, Lemma 30. There’s at most a node with Compress op in the the claim is true. path keysk subpath of ( ) that is active, and it resides in the end of We consider the pushing sequence as n0, n1, ..., nk. Sup- the path if exists. pose that for first k nodes, the path from the root to nk is 24
t the physical path. We shall prove that for nk+1, the lemma linearization point at some snapshotTi so that l returned k is true. If there is no replace operation before reading nk+1, is on the pysical path for key in snapshotTi . it is obvious that we can linearized it as the same snapshot Lemma 36. For find(keys) that returns tuples before pushing n . k hlk, pOpk, pathki, there’s a snapshot after its invocation Next, we assume that replace occurs on n before read- k and before reading lk, such that path(keysk) returned with lk at ing nk+1, and results in snapshot. There are two cases: the end is a physical path in snapshotTi . 1) Consider if replace occurs after reading nk+1. We lk could have snapshotk+1 = snapshotk because Proof. Since is a Leaf node or an Empty node, by Lemma 34 snapshot lk l l nk+1 is connected to an active node in a snapshot there is a Ti that as the last node from i−1, and 0 n path keysk path keysk so that nk+1 is also reachable from the root. to i−1 in the ( ) are active. Therefore ( ) lk physical path keysk snapshot 2) Consider if replace occurs before reading nk+1. with is ( ) in Ti . We define snapshot find keysk If the replace operation changes nk, then nk is Ti as ( )’s linearization point. flagged with a Compress op by Lemma 24. Then we In the next, we define linearization points for basic oper- have snapshot = snapshot because Lemma 27 k+1 k ations. For the insert operation, the remove operation, and demonstrates that a node flagged with a Com- the move operation that returns true, we define it to be the press has no successive replace operations on its first replacek, 0 ≤ k < |replace | belongs to op . To make child pointers. Otherwise, if the replace operation i i i a reasonable demonstration, we first show that the first changes n , n is not changed by Lemma 23. k+1 k replacek belongs the op created by each operation itself, Hence, we have snapshot = snapshot, where i i k+1 and then illustrate that op is unique for each operation. nk+1 is reachable from the root in the snapshot just after a replace operation. Lemma 37. If the insert operation, the remove operation, or the k move operation that returns true, the first replacei occurs before the returning, and it belongs to opi created by the operation itself. ireplace rreplace mreplace Lemma 35. After , , , and k creplace, quadtree’s properties remain. Proof. First we prove that replacei belongs to opi created by the operation itself.
Proof. We shall prove that in any snapshotTi , quadtree’s For the insert operation, it returns true after successful properties remain. iflag and ireplace using op created at line 112. First, Lemma 10 shows that two layers of dummy nodes For the remove operation, it returns true after successful remain in the tree. We have to consider other layers of nodes rflag and rreplace using op created at line 168. which are changed by replace operations. For the move operation, it returns true after successful Consider creplace that replaces an Internal node by an mflag and mreplace using op created at line 224. Empty node. Because the Internal node has been flagged on Corollary 5 indicates that the first replace operation must a Compress op before creplace (Lemma 24), all of its children succeed, therefore all successful replace operations happen are Empty and not changed (Lemma 28 and Lemma 27). before returning. Thus, creplace does not affect the second claim of Defini- tion 10. Then, we define the linearization points of the insert Consider ireplace, rreplace, or mreplace that replaces a operation, the remove operation, and the move operation terminal node by an Empty node, a Leaf node, or a sub-tree. that returns true are before returning. We should also point By Lemma 23, before replacek, op.parentk is flagged with out that there’s only one op lead to the linearization point op such that no successful replace operation could happen for each operation. on op.parentk. Therefore, if the new node is Empty, it does Lemma 38. For the insert operation, the remove operation, and not affect the tree property. If the new node is a Leaf node or the move operation that returns true, op is created at the last a sub-tree, based on the post-conditions of the createNode iteration in the while loop. function (Lemma 17), after replace operations the second claim of Definition 10 still holds. Proof. Based on Lemma 23, all successful insert, remove and move operations follow the flag → replace → unflag tran- .2 Linearizability sition. flag is the first successful flag operation after reading op .parent, so there is no other flag operation further. This In this Section, we define linearization points for basic op- i establishes the claim. erations. As the compress operation is included in the move operation and the remove operation that returns true, it does Lemma 39. If the insert operation, the remove operation, and not affect the linearization points of them. If an algorithm is the move operation that return false, successful replace happens linearizable, it could be ordered equivalently as a sequential before returning. one by their linearization points. Since all the modifications depend on find(keys), we first point out its linearization Proof. As Lemma 38 points out that op is created at the point. last iteration. Moreover, before returning true, mflag, iflag To prove that our linearization points are correct, we and rflag must succeed according to the program order. As shall demonstrate that for some time Ti, the key set in the first replace operation that belongs to opi must succeed
snapshotTi is the same as the results of modifications lin- (Lemma 5, replace happens before returning. earized before Ti. For find(keys), we should define the 25
Lemma 40. If the insert operation, the remove operation, and the Proof. Lemma 39 shows that there is successful move operation that return false, there’s no successful replace replace occurs before the insert operation, the re- ever happened. move operation, and the move operation that re- turns true. Proof. Consider the execution of the insert operation, it If a replace operation succeeds, op.parent has been returns false at line 108 where op created are not flagged flagged with op other than Compress before replace on nodes. Consider the remove operation, it returns false at (Lemma 24 and Lemma 23). Lemma 29 implies that line 165. Also, op created are not flagged on nodes. For the op.parent is reachable from the root. Therefore, it move operation, it returns false by calling the findCommon is active. Besides, Corollary 31 shows that the path function or the continueFindCommon function so that the from the root to op.parent is an active. Therefore a helpMove function must fail or it is not called. If the help- snapshot exists such that the path from the root to Move function is not called, there is no replace operations. op.oldChild is a physical path(key). If the helpMove function returns false, by the ordering of First we prove that key∈ / s . By Lemma 41, in the mflag and unflag (Lemma 23 and Lemma 25), op.iP arent i−1 snapshot l is inactive. Otherwise, replace operations and op.rP arent are different. Moreover, the flag operation will not execute. Hence, key∈ / s . on the latter parent will fail such that doCAS is false and no i−1 Second, by the post-conditions of the createNode replace operation will perform. function 17, it creates a node or a sub-tree that In the next step, we show that how the algorithm could contains newKey. According to Corollary 35, after be ordered as a sequential execution. First we linearize the ireplace the node contains newKey is inserted and insert operation, the remove operation and the move oper- quadtree’s properties maintained. Hence, key ∈ si. ation that returns true, where replace results in changing 1)2) First we prove that key ∈ si. By Lemma 41, in the the key set. Let si be a set of Leaf nodes that are active snapshot l is active. Otherwise, replace operations in snapshotTi after performing operations o0, o1, ..., on, will not execute. Hence, key∈ / si−1. ordered by their linearization points replace0, repalce1, ..., Second, the remove operation creates an Empty replacen sequentially. node. According to Corollary 35, after rreplace the node contains oldKey is removed so that key ∈ s . Observation 6. If a Leaf node is moved, its op is set before i 3) First we prove that oldKey ∈ s , newKey∈ / s . the replace operation on op.iP arent, which is before the replace i−1 i−1 By Lemma 41, rl is active at snapshot , il is inac- operation on op.rP arent. Ti tive at snapshotTi1 ,Ti < Ti1. We have to demon-
Lemma 41. For the contain operation that returns true, there strate that rl is still active at snapshotTi1 so that we lk snapshot is a corresponding snapshot that is active in Ti . For can use snapshotTi1 as si−1. the contain operation that returns false, there is a corresponding We prove it by contradiction. Assuming that at lk snapshot snapshot that is inactive in Ti . snapshotTi1 rl is inactive, hence there’s some re- place operation happens before T and after T . Proof. For the first part, we prove that in a snapshot, the end i1 i However, based on Lemma 23, replace is the first node lk contains keysk and is not moved. Lemma 36 shows i successful replace operation that after reading rl. that there is a snapshot consists of physical path(keysk) k k Therefore, it derives a contradiction. Hence we after calling, thus l that contains keys is in snapshotT i prove that in snapshot rl is still active. before judging whether the node is moved. We only have Ti1 to discuss mreplace in the next cases because other replace operations will not affect lk.op. If lk.op is null at verifying, mreplace must have not been done according to Obser- Finally we order linearization points for the insert op- eration, the remove operation and the move operation that vation 6. In this case, we could linearize it at snapshotTi . If lk.op is not null, but mreplace occurs after verifying returns false. We also order the contain operation. We con- replace replace whether l.op.iP arent has l.op.oldIChild, then it is also be- sider they are linearized between i−1 and i if fore the second mreplace on l.op.rP arent (Observation 6). exists.
We could also linearize it snapshotTi . Lemma 43. 1) For the insert operation returns false, For the second part, Lemma 36 shows that there is a key ∈ si−1. k snapshot consists physical path(keys ). By Lemma 35, in 2) For the remove operation returns false, key∈ / si−1. every snapshot our quadtree’s property maintains. Hence, 3) For the move operation returns false, either oldKey∈ / k k if l does not contain keys , we could linearize it at si−1 or newKey ∈ si−1. k k snapshotTi . Or else, if l is in physical path, but l is snapshot moved at verifying, we could linearize at Ti1 after Proof. The first two parts are equivalent as the case in mreplace. Lemma 41. Lemma 42. 1) For the insert operation, it returns true Lemma 44. Our quadboost is a linearizable implementation. when key∈ / si−1, key ∈ si. 2) For the remove operation, it returns true and key ∈ si−1, key∈ / si. Proof. By lemma 42 and Lemma 43, our algorithm returns 3) For the move operation, it returns true and oldKey ∈ the same result as they are finished in the linearized order. si−1, newKey∈ / si−1, oldKey∈ / si, and newKey ∈ Therefore we prove our algorithm is linearizable and our si. linearization points are correct. 26
.3 Progress Condition that the flag → replace transition happens only We say an algorithm is non-blocking if the system as a whole in the compress function which is called by the is making progress even if some threads are starving. We successful remove operation or the move operation. prove our quadboost is non-blocking by following set of Hence, there are a finite number of flag → replace lemmas. We assume that there are finite number of basic transitions, as Corollary 6 shows that the compress operations invocations. function is terminable. 3) The flag → unflag transition executes only in Observation 7. There are finite number of basic operations. the helpMove function where op.iP arent cannot Lemma 45. path(keyk) returned by find(keys) consists of be flagged according to Lemma 25 (if op.iF irst is finite number of keys. true). The case that op.iF irst is false can be proved symmetrically. Proof. Observation 7 shows that the number of successful From first two parts of the lemma, there are fi- insert operation and move operation is limited. Lemma 40 nite number of replace operations that change illustrates that for basic operations return false, there is no quadtree’s structure. If the flag → replace → successful replace operation. Hence, only successful insert transition happens simultaneously, it will reset a and move operations add nodes into quadtree. The post- node’s op to Clean. If the flag → replace → unflag conditions of the createNode function (Lemma 17) and the transition happens in the meantime. Lemma 33 in- effects of replace (Lemma 35) demonstrate that they will dicates that a node with Compress op will not be add finite number of nodes into quadtree. We then prove pushed twice. Consider if the helpMove function that pathk is terminable. By Lemma 34, find(keys) returns detects op.iP arent.op as Compress, then op.iP arent path(keyk) which is a subpath of physical path(keysk) is not reachable in the next time. Hence, quadtree in snapshotTi which is terminable. Thus, there are finite is stabilized and only when flag → unflag tran- number of nodes in pathk. sitions execute infinitely and never set doCAS as true. Corollary 6. The compress function must terminate. According to Lemma 46, we order spatial order on k Proof. By Lemma 45, path is terminable. Since other func- both op.iP arent and op.rP arent as ip0 > ip1... > tions called by the compress function do not consist loops, ipn and rp0 > rp1... > rpn. We assume that there’s the compress function must terminate. a ring such that movei which flags rpi but fails on 0 ipi, and movei which flags ipi but fails on rpi. By The we shall demonstrate that there are finite number of assumption we order rpi and ipi as rpi > ipi by three different CAS transitions. 0 movei’s order, and ipi > rpi by movei’s order. But Lemma 46. There is an unique spatial order among nodes in by Lemma 46, all Internal nodes in a snapshot can be ordered uniquely. Therefore it derives a contra- quadtree in snapshptTi diction. Proof. As illustrated by the algorithm at line 223, the getSpa- Then, we prove that there’s no ring among all tialOrder function compares ip with rp by the order: x → movei in the same snapshot, so at least one of y → w. In our quadtree, we only consider square partitions movei could set doCAS to true, resulting in the on two-dimensional space. Hence, w is always equal to h. flag → replace → unflag transition. Let’s consider We prove the lemma by contradiction. Assuming there are the dependency among all movei. If movei which Internal hx, y, wi 0 two different nodes with the same , they fails on pi has been flagged by movei, then there hx, yi 0 represent the same square starting with left coroner is a directed edge from movei to movei by order. with width w. By Lemma 35, in every snapshot quadtree’s In this way, the head node in the graph must have properties remain. There cannot be two squares with the no out edge as proved. It will set doCAS to true, same left corner and w. Hence, in snapshptTi , a Internal all move operations that are directly connected to node consists of a unique hx, y, wi tuple that can be ordered it will help it finish. After erase the former head correctly. node from the graph, there will be other nodes that Lemma 47. 1) There are finite number of successful have no out edge. Finally, all dependency edges are flag → replace → unflag transitions. removed. Therefore, there are a finite number of flag → unflag 2) There are finite number of successful flag → replace transitions. transitions. 3) There are finite number of successful flag → unflag Observation 8. The help function, the helpCompress function, transitions. the helpMove function, and the helpSubstitute function are not Proof. 1) By Lemma 39, replace operations only occur called in a mutual way. (If method A calls method B, and method in basic operations that return true. By Lemma 38, B calls method A reciprocally, we say method A and B are called there’s a unique op created for operations that re- in a mutual way.)
turn true. Hence, for the move operation, the insert Lemma 48. If help function returned at Ti, and find(keys) 0 operation and the remove operation, there are finite at the prior iteration reads p .op at Ti1 < Ti, Leaf nodes in number of flag → replace → circles. snapshotTi and snapshotTi1 are different. 2) By Lemma 39, replace operations only occur in basic operations that return true. Lemma 24 shows Proof. We prove the lemma by contradiction, there are two 27
possible scenarios. First, the help function might be inter- also result from the failure of mflag. For above cases, the minable. But based on Observation 8, the help function must help function at line 332 would change quadtree. If ip and terminate. Second, the help function might execute without rp are different, it results from that either iP ath or rP ath changing quadtree. According to Lemma 47, there are finite has popped the LCA node. We have proved the case in the number of CAS transitions. So if some help function does former paragraph. Thus, we derives a contradiction. not change quadtree, the structure might be changed before From above discussions, we prove that quadboost is Ti1. For the move function, if ip and rp are the same but non-blocking. their ops are different, the help function might not change
the structure. Nevertheless, snapshotTi and snapshotTi1 are different. Or else, there are some interval that all transitions are flag → unflag. However, Lemma 47 also shows that all transitions form a acyclic graph that there is no inter- val that all transitions are flag → unflag. Therefore, a flag → replace → unflag transition or a flag → unflag transition is executed to change the snapshot. Lemma 49. Our quadboost algorithms are non-blocking. Proof. We have to prove that no process will execute loops infinitely without changing keys in quadtree. First, we prove that path is terminable. Next, we prove that find(keysk) starts from an active node in physical path(keysk) in
snapshotTi between ith iteration and i + 1th iteration.
Finally, Leaf nodes in snapshotTi1 at the returning of
find(keys) at ith iteration is different from snapshotTi2 that at the returning of find(keys) at the i + 1th iteration. For the first part, initially we start from the root node. Therefore, path is empty. Moreover, as Lemma 45 shows that pathk consists of finite number of keys, we establish this part. For the second part, the continueFind function and the continueFindCommon function pop all nodes with a Compress op from path. For the insert operation and the remove operation, since Lemma 29 shows that an Internal nodes whose op is not Compress is active , and Lemma 31 shows that nodes above the active node are also active, there is a snapshot in which the top node of path is still in physical path(keysk). For the move operation, if either rF ail or iF ail is true, it is equivalent with the prior case. If cF ail is true, Lemma 32 illustrates that if the LCA node is active, it is in physical path for both oldKey and newKey. Thus, there is also a snapshot in which the start node is in physical path. We prove the third part by contradiction, assuming that quadtree is stabilized at Ti, and all invocations after Ti are looping infinitely without changing Leaf nodes. For the insert and remove operations, before the invoca- tion of find(keys) at the next iteration, they must execute the help function at line 118 and line 175 accordingly. In both cases, the help function changes the snapshot (Lemma 48). For the move operation, consider different situations of the continueFindCommon function. If rF ail or iF ail is true, two situations arise: (1) iOp or rOp is Clean but mflag fails. (2) iOp or rOp is not Clean. Consider the first case that mflag fails, iOp and rOp are updated respectively at line 236 and line 235. If ip and rp are different, before its invocation of find(keysk), the help function is performed at line 311 and line 287. Thus by Lemma 48, the snapshot is changed between two iterations. Now consider if cF ail is true. If ip and rp are the same, it could result from the difference between rOp and iOp. In this case, the snapshot might be changed between reading rOp and iOp. It could