The Art of Multiprocessor Programming Copyright 2007 Elsevier Inc. All rights reserved Maurice Herlihy Nir Shavit October 3, 2007 DRAFT COPY 2 DRAFT COPY Contents 1 Introduction 13 1.1SharedObjectsandSynchronization.............. 16 1.2AFable.............................. 19 1.2.1 PropertiesofMutualExclusion............. 21 1.2.2 TheMoral......................... 22 1.3TheProducer-ConsumerProblem................ 23 1.4TheReaders/WritersProblem.................. 26 1.5TheHarshRealitiesofParallelization............. 27 1.6Missive.............................. 29 1.7ChapterNotes.......................... 30 1.8Exercises............................. 30 Principles 35 2 Mutual Exclusion 37 2.1Time................................ 37 2.2CriticalSections......................... 38 2.3Two-ThreadSolutions...................... 41 2.3.1 The LockOne Class.................... 41 2.3.2 The LockTwo Class.................... 43 2.3.3 ThePetersonLock.................... 44 2.4TheFilterLock..........................DRAFT COPY 46 2.5Fairness.............................. 49 2.6Lamport’sBakeryAlgorithm.................. 49 2.7 Bounded Timestamps . ................ 51 2.8 Lower Bounds on Number of Locations . ...... 56 2.9ChapterNotes.......................... 60 2.10Exercises............................. 60 3 4 CONTENTS 3 Concurrent Objects 67 3.1ConcurrencyandCorrectness.................. 67 3.2SequentialObjects........................ 71 3.3QuiescentConsistency...................... 72 3.3.1 Remarks.......................... 74 3.4SequentialConsistency...................... 75 3.4.1 Remarks.......................... 76 3.5 Linearizability . ......................... 79 3.5.1 LinearizationPoints................... 79 3.5.2 Remarks.......................... 79 3.6FormalDefinitions........................ 80 3.6.1 Linearizability . .................. 81 3.6.2 Linearizability is Compositional . ........ 82 3.6.3 TheNon-BlockingProperty............... 83 3.7ProgressConditions....................... 84 3.7.1 DependentProgressConditions............. 85 3.8TheJavaMemoryModel.................... 86 3.8.1 LocksandSynchronizedBlocks............. 88 3.8.2 VolatileFields...................... 88 3.8.3 FinalFields........................ 89 3.9Remarks.............................. 90 3.10ChapterNotes.......................... 91 3.11Exercises............................. 91 4 Foundations of Shared Memory 97 4.1TheSpaceofRegisters...................... 98 4.2RegisterConstructions......................104 4.2.1 MRSWSafeRegisters..................105 4.2.2 ARegularBooleanMRSWRegister..........105 4.2.3 A regular M-valuedMRSWregister...........106 4.2.4 AnAtomicSRSWRegister...............109 4.2.5 AnAtomicMRSWRegister...............112 DRAFT4.2.6 AnAtomicMRMWRegister..............115 COPY 4.3AtomicSnapshots........................117 4.3.1 AnObstruction-freeSnapshot..............117 4.3.2 AWait-FreeSnapshot..................119 4.3.3 CorrectnessArguments.................119 4.4ChapterNotes..........................121 4.5Exercises.............................122 CONTENTS 5 5 The Relative Power of Primitive Synchronization Opera- tions 133 5.1ConsensusNumbers.......................134 5.1.1 StatesandValence....................135 5.2AtomicRegisters.........................138 5.3ConsensusProtocols.......................140 5.4FIFOQueues...........................141 5.5MultipleAssignmentObjects..................145 5.6Read-Modify-WriteOperations.................148 5.7Common2RMWOperations..................150 5.8 The compareAndSet() Operation................152 5.9ChapterNotes..........................154 5.10Exercises.............................155 6 Universality of Consensus 163 6.1Introduction............................163 6.2Universality............................164 6.3ALock-freeUniversalConstruction...............165 6.4AWait-freeUniversalConstruction...............169 6.5ChapterNotes..........................176 6.6Exercises.............................176 Practice 179 7 Spin Locks and Contention 181 7.1WelcometotheRealWorld...................181 7.2Test-and-SetLocks........................185 7.3TAS-BasedSpinLocksRevisited................188 7.4ExponentialBackoff.......................190 7.5QueueLocks...........................192 7.5.1 Array-BasedLocks....................192 7.5.2 TheCLHQueueLock..................193 7.5.3 TheMCSQueueLock..................197 7.6AQueueLockwithTimeouts..................200DRAFT COPY 7.7ACompositeLock........................203 7.7.1 AFast-PathCompositeLock..............210 7.8HierarchicalLocks........................212 7.8.1 AHierarchicalBackoffLock...............213 7.8.2 AHierarchicalCLHQueueLock............214 7.9OneLockToRuleThemAll..................220 6 CONTENTS 7.10ChapterNotes..........................220 7.11Exercises.............................221 8 Monitors and Blocking Synchronization 223 8.1Introduction............................223 8.2MonitorLocksandConditions.................224 8.2.1 Conditions........................225 8.2.2 TheLost-WakeupProblem...............228 8.3Readers-WritersLocks......................231 8.3.1 SimpleReaders-WritersLock..............231 8.3.2 FairReaders-WritersLock................232 8.4AReentrantLock.........................234 8.5Semaphores............................237 8.6ChapterNotes..........................239 8.7Exercises.............................240 9 Linked Lists: the Role of Locking 245 9.1Introduction............................245 9.2List-basedSets..........................246 9.3ConcurrentReasoning......................248 9.4Coarse-GrainedSynchronization................250 9.5Fine-GrainedSynchronization..................252 9.6OptimisticSynchronization...................257 9.7LazySynchronization......................261 9.8ALock-FreeList.........................267 9.9Discussion.............................272 9.10ChapterNotes..........................274 9.11Exercises.............................274 10 Concurrent Queues and the ABA Problem 277 10.1Introduction............................277 10.2Queues...............................279 10.3 A Bounded Partial Queue . ..................279 DRAFT10.4 An Unbounded Total Queue . ..................285COPY 10.5 An Unbounded Lock-Free Queue . ........286 10.6MemoryreclamationandtheABAproblem..........290 10.6.1 A Na¨ıveSynchronousQueue..............294 10.7DualDataStructures......................296 10.8ChapterNotes..........................299 10.9Exercises.............................299 CONTENTS 7 11 Concurrent Stacks and Elimination 303 11.1Introduction............................303 11.2 An Unbounded Lock-free Stack . ................303 11.3Elimination............................304 11.4TheEliminationBackoffStack.................305 11.4.1ALock-freeExchanger..................308 11.4.2TheEliminationArray..................311 11.5ChapterNotes..........................314 11.6Exercises.............................315 12 Counting, Sorting, and Distributed Coordination 321 12.1Introduction............................321 12.2SharedCounting.........................321 12.3SoftwareCombining.......................322 12.3.1Overview.........................323 12.3.2AnExtendedExample..................330 12.3.3PerformanceandRobustness..............333 12.4Quiescently-ConsistentPoolsandCounters..........333 12.5CountingNetworks........................334 12.5.1Networksthatcount...................334 12.5.2TheBitonicCountingNetwork.............337 12.5.3PerformanceandPipelining...............345 12.6DiffractingTrees.........................348 12.7ParallelSorting..........................353 12.8SortingNetworks.........................354 12.8.1DesigningaSortingNetwork..............354 12.9SampleSorting..........................357 12.10DistributedCoordination....................360 12.11ChapterNotes..........................361 12.12Exercises.............................362 13 Concurrent Hashing and Natural Parallelism 367 13.1Introduction............................367 13.2Closed-AddressHashSets....................368DRAFT COPY 13.2.1ACoarse-GrainedHashSet...............371 13.2.2AStripedHashSet....................373 13.2.3ARefinableHashSet..................376 13.3ALock-freeHashSet.......................379 13.3.1RecursiveSplit-ordering.................379 13.3.2 The BucketListclass...................383 8 CONTENTS 13.3.3 The LockFreeHashSet<T> class.............385 13.4AnOpen-AddressedHashSet..................388 13.4.1CuckooHashing.....................389 13.4.2ConcurrentCuckooHashing...............390 13.4.3StripedConcurrentCuckooHashing..........395 13.4.4 A Refinable Concurrent Cuckoo Hash Set . 395 13.5ChapterNotes..........................396 13.6Exercises.............................396 14 Skiplists and Balanced Search 405 14.1Introduction............................405 14.2SequentialSkiplists........................406 14.3ALock-BasedConcurrentSkiplist...............407 14.3.1ABird’sEyeView....................407 14.3.2TheAlgorithm......................410 14.4ALock-FreeConcurrentSkiplist................417 14.4.1ABird’sEyeView....................417 14.4.2TheAlgorithminDetail.................420 14.5ConcurrentSkiplists.......................426 14.6ChapterNotes..........................428 14.7Exercises.............................429 15 Priority Queues 433 15.1Introduction............................433 15.1.1ConcurrentPriorityQueues...............434 15.2 An Array-Based Bounded Priority Queue . ........434 15.3 A Tree-Based Bounded Priority Queue . ........435 15.4 An Unbounded Heap-Based Priority Queue