DOCSLIB.ORG

Shared-Memory Synchronization

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Shared-Memory Synchronization Shared-Memory Synchronization MIchael L. Scott University of Rochester SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE #1 &MC Morgan& cLaypool publishers ABSTRACT Ever since the advent of time sharing in the 1960s, designers of concurrent and parallel systems have needed to synchronize the activities of threads of control that share data structures in memory. In recent years, the study of synchronization has gained new urgency with the proliferation of multicore processors, on which even relatively simple user-level programs must frequently run in parallel. This monograph offers a comprehensive survey of shared-memory synchronization, with an emphasis on “systems-level” issues. It includes sufficient coverage of architectural details to understand correctness and performance on modern multicore machines, and sufficient coverage of higher-level issues to understand how synchronization is embedded in modern programming languages. The primary intended audience is “systems programmers”—the authors of operating systems, library packages, language run-time systems, and server and utility programs. Much of the discussion should also be of interest to application programmers who want to make good use of the synchronization mechanisms available to them, and to computer architects who want to understand the ramifications of their design decisions on systems- level code. KEYWORDS Atomicity, barriers, busy-waiting, conditions, locality, locking, memory mod- els, monitors, multiprocessor architecture, nonblocking algorithms, scheduling, semaphores, synchronization, transactional memory. iii To Kelly, my wife and partner of more than 30 years. iv Contents 1 Introduction .................................................................1 1.1 Atomicity.................................................................3 1.2 Condition Synchronization . 5 1.3 Spinning vs. Blocking . 6 1.4 Safety and Liveness . 7 1.5 The Rest of this Monograph . 8 2 Architectural Background ...................................................9 2.1 Cores and Caches: Basic Shared-Memory Architecture . 9 2.1.1 Temporal and Spatial Locality . 11 2.1.2 Cache Coherence. .12 2.1.3 Processor (Core) Locality . 13 2.2 Cache Consistency . 14 2.2.1 Sources of Inconsistency. .14 2.2.2 Memory Fence Instructions . 16 2.2.3 Example Architectures . 17 2.3 Atomic Primitives . 18 2.3.1 The ABA Problem . 21 2.3.2 Other Synchronization Hardware . 23 3 Some Useful Theory ........................................................24 3.1 Safety....................................................................24 3.1.1 Deadlock Freedom . 25 3.1.2 Atomicity . 26 3.2 Liveness..................................................................33 3.2.1 Nonblocking Progress . 34 CONTENTS v 3.2.2 Fairness. .36 3.3 The Consensus Hierarchy. .37 3.4 Memory Models . 38 3.4.1 Formal Framework . 39 3.4.2 Data Races . 41 3.4.3 Real-World Models. .42 4 Practical Spin Locks ........................................................44 4.1 Classical load-store-only Algorithms . 44 4.2 Centralized Algorithms. .47 4.2.1 Test and set Locks.................................................48 4.2.2 The Ticket Lock . 50 4.3 Queued Spin Locks. .51 4.3.1 The MCS Lock . 52 4.3.2 The CLH Lock. .56 4.3.3 Which Spin Lock Should I Use? . 60 4.4 Special-case Optimizations . 60 4.4.1 Nested Locks . 60 4.4.2 Locality-conscious Locking . 61 4.4.3 Double-checked Locking. .63 4.4.4 Asymmetric Locking . 63 5 Spin-based Conditions and Barriers.........................................67 5.1 Flags. .67 5.2 Barrier Algorithms . 68 5.2.1 The Sense-Reversing Centralized Barrier . 69 5.2.2 Software Combining. .69 5.2.3 The Dissemination Barrier . 71 5.2.4 Tournament Barriers . 72 5.2.5 Static Tree Barriers . 74 5.2.6 Which Barrier Should I Use? . 74 vi CONTENTS 5.3 Barrier Extensions . 76 5.3.1 Fuzzy Barriers . 76 5.3.2 Adaptive Barriers . 79 5.3.3 Barrier-like Constructs . 81 6 Read-mostly Atomicity .....................................................83 6.1 Reader-writer Locks . 83 6.1.1 Centralized Algorithms . 84 6.1.2 Queued Reader-writer Locks . 85 6.2 Sequence Locks . 90 6.3 Read-Copy Update . 93 7 Synchronization and Scheduling ............................................98 7.1 Scheduling . 98 7.2 Semaphores . 98 7.3 Monitors . 98 7.4 Other Language Mechanisms . 98 7.4.1 Conditional Critical Regions . 98 7.4.2 Futures . 98 7.4.3 Series-Parallel Execution . 98 7.5 Kernel-Only Mechanisms . 98 7.6 Performance Considerations . 98 7.6.1 Spin-Then-Wait. .99 7.6.2.
Load more

Recommended publications
  • ACM SIGACT News Distributed Computing Column 28
    ACM SIGACT News Distributed Computing Column 28 Idit Keidar Dept. of Electrical Engineering, Technion Haifa, 32000, Israel [email protected] Sergio Rajsbaum, who edited this column for seven years and established it as a relevant and popular venue, is stepping down. This issue is my first step in the big shoes he vacated. I would like to take this opportunity to thank Sergio for providing us with seven years’ worth of interesting columns. In producing these columns, Sergio has enjoyed the support of the community at-large and obtained material from many authors, who greatly contributed to the column’s success. I hope to enjoy a similar level of support; I warmly welcome your feedback and suggestions for material to include in this column! The main two conferences in the area of principles of distributed computing, PODC and DISC, took place this summer. This issue is centered around these conferences, and more broadly, distributed computing research as reflected therein, ranging from reviews of this year’s instantiations, through influential papers in past instantiations, to examining PODC’s place within the realm of computer science. I begin with a short review of PODC’07, and highlight some “hot” trends that have taken root in PODC, as reflected in this year’s program. Some of the forthcoming columns will be dedicated to these up-and- coming research topics. This is followed by a review of this year’s DISC, by Edward (Eddie) Bortnikov. For some perspective on long-running trends in the field, I next include the announcement of this year’s Edsger W.
    [Show full text]
  • Fast and Correct Load-Link/Store-Conditional Instruction Handling in DBT Systems
    Fast and Correct Load-Link/Store-Conditional Instruction Handling in DBT Systems Abstract Atomic Read Memory Dynamic Binary Translation (DBT) requires the implemen- Operation tation of load-link/store-conditional (LL/SC) primitives for guest systems that rely on this form of synchronization. When Equals targeting e.g. x86 host systems, LL/SC guest instructions are YES NO typically emulated using atomic Compare-and-Swap (CAS) expected value? instructions on the host. Whilst this direct mapping is efficient, this approach is problematic due to subtle differences between Write new value LL/SC and CAS semantics. In this paper, we demonstrate that to memory this is a real problem, and we provide code examples that fail to execute correctly on QEMU and a commercial DBT system, Exchanged = true Exchanged = false which both use the CAS approach to LL/SC emulation. We then develop two novel and provably correct LL/SC emula- tion schemes: (1) A purely software based scheme, which uses the DBT system’s page translation cache for correctly se- Figure 1: The compare-and-swap instruction atomically reads lecting between fast, but unsynchronized, and slow, but fully from a memory address, and compares the current value synchronized memory accesses, and (2) a hardware acceler- with an expected value. If these values are equal, then a new ated scheme that leverages hardware transactional memory value is written to memory. The instruction indicates (usually (HTM) provided by the host. We have implemented these two through a return register or flag) whether or not the value was schemes in the Synopsys DesignWare® ARC® nSIM DBT successfully written.
    [Show full text]
  • Edsger Dijkstra: the Man Who Carried Computer Science on His Shoulders
    INFERENCE / Vol. 5, No. 3 Edsger Dijkstra The Man Who Carried Computer Science on His Shoulders Krzysztof Apt s it turned out, the train I had taken from dsger dijkstra was born in Rotterdam in 1930. Nijmegen to Eindhoven arrived late. To make He described his father, at one time the president matters worse, I was then unable to find the right of the Dutch Chemical Society, as “an excellent Aoffice in the university building. When I eventually arrived Echemist,” and his mother as “a brilliant mathematician for my appointment, I was more than half an hour behind who had no job.”1 In 1948, Dijkstra achieved remarkable schedule. The professor completely ignored my profuse results when he completed secondary school at the famous apologies and proceeded to take a full hour for the meet- Erasmiaans Gymnasium in Rotterdam. His school diploma ing. It was the first time I met Edsger Wybe Dijkstra. shows that he earned the highest possible grade in no less At the time of our meeting in 1975, Dijkstra was 45 than six out of thirteen subjects. He then enrolled at the years old. The most prestigious award in computer sci- University of Leiden to study physics. ence, the ACM Turing Award, had been conferred on In September 1951, Dijkstra’s father suggested he attend him three years earlier. Almost twenty years his junior, I a three-week course on programming in Cambridge. It knew very little about the field—I had only learned what turned out to be an idea with far-reaching consequences. a flowchart was a couple of weeks earlier.
    [Show full text]
  • Arxiv:1901.04709V1 [Cs.GT]
    Self-Stabilization Through the Lens of Game Theory Krzysztof R. Apt1,2 and Ehsan Shoja3 1 CWI, Amsterdam, The Netherlands 2 MIMUW, University of Warsaw, Warsaw, Poland 3 Sharif University of Technology, Tehran, Iran Abstract. In 1974 E.W. Dijkstra introduced the seminal concept of self- stabilization that turned out to be one of the main approaches to fault- tolerant computing. We show here how his three solutions can be for- malized and reasoned about using the concepts of game theory. We also determine the precise number of steps needed to reach self-stabilization in his first solution. 1 Introduction In 1974 Edsger W. Dijkstra introduced in a two-page article [10] the notion of self-stabilization. The paper was completely ignored until 1983, when Leslie Lamport stressed its importance in his invited talk at the ACM Symposium on Principles of Distributed Computing (PODC), published a year later as [21]. Things have changed since then. According to Google Scholar Dijkstra’s paper has been by now cited more than 2300 times. It became one of the main ap- proaches to fault tolerant computing. An early survey was published in 1993 as [26], while the research on the subject until 2000 was summarized in the book [13]. In 2002 Dijkstra’s paper won the PODC influential paper award (renamed in 2003 to Dijkstra Prize). The literature on the subject initiated by it continues to grow. There are annual Self-Stabilizing Systems Workshops, the 18th edition of which took part in 2016. The idea proposed by Dijkstra is very simple. Consider a distributed system arXiv:1901.04709v1 [cs.GT] 15 Jan 2019 viewed as a network of machines.
    [Show full text]
  • Distributed Algorithms for Wireless Networks
    A THEORETICAL VIEW OF DISTRIBUTED SYSTEMS Nancy Lynch, MIT EECS, CSAIL National Science Foundation, April 1, 2021 Theory for Distributed Systems • We have worked on theory for distributed systems, trying to understand (mathematically) their capabilities and limitations. • This has included: • Defining abstract, mathematical models for problems solved by distributed systems, and for the algorithms used to solve them. • Developing new algorithms. • Producing rigorous proofs, of correctness, performance, fault-tolerance. • Proving impossibility results and lower bounds, expressing inherent limitations of distributed systems for solving problems. • Developing foundations for modeling, analyzing distributed systems. • Kinds of systems: • Distributed data-management systems. • Wired, wireless communication systems. • Biological systems: Insect colonies, developmental biology, brains. This talk: 1. Algorithms for Traditional Distributed Systems 2. Impossibility Results 3. Foundations 4. Algorithms for New Distributed Systems 1. Algorithms for Traditional Distributed Systems • Mutual exclusion in shared-memory systems, resource allocation: Fischer, Burns,…late 70s and early 80s. • Dolev, Lynch, Pinter, Stark, Weihl. Reaching approximate agreement in the presence of faults. JACM,1986. • Lundelius, Lynch. A new fault-tolerant algorithm for clock synchronization. Information and Computation,1988. • Dwork, Lynch, Stockmeyer. Consensus in the presence of partial synchrony. JACM,1988. Dijkstra Prize, 2007. 1A. Dwork, Lynch, Stockmeyer [DLS] “This paper introduces a number of practically motivated partial synchrony models that lie between the completely synchronous and the completely asynchronous models and in which consensus is solvable. It gave practitioners the right tool for building fault-tolerant systems and contributed to the understanding that safety can be maintained at all times, despite the impossibility of consensus, and progress is facilitated during periods of stability.
    [Show full text]
  • 2020 SIGACT REPORT SIGACT EC – Eric Allender, Shuchi Chawla, Nicole Immorlica, Samir Khuller (Chair), Bobby Kleinberg September 14Th, 2020
    2020 SIGACT REPORT SIGACT EC – Eric Allender, Shuchi Chawla, Nicole Immorlica, Samir Khuller (chair), Bobby Kleinberg September 14th, 2020 SIGACT Mission Statement: The primary mission of ACM SIGACT (Association for Computing Machinery Special Interest Group on Algorithms and Computation Theory) is to foster and promote the discovery and dissemination of high quality research in the domain of theoretical computer science. The field of theoretical computer science is the rigorous study of all computational phenomena - natural, artificial or man-made. This includes the diverse areas of algorithms, data structures, complexity theory, distributed computation, parallel computation, VLSI, machine learning, computational biology, computational geometry, information theory, cryptography, quantum computation, computational number theory and algebra, program semantics and verification, automata theory, and the study of randomness. Work in this field is often distinguished by its emphasis on mathematical technique and rigor. 1. Awards ▪ 2020 Gödel Prize: This was awarded to Robin A. Moser and Gábor Tardos for their paper “A constructive proof of the general Lovász Local Lemma”, Journal of the ACM, Vol 57 (2), 2010. The Lovász Local Lemma (LLL) is a fundamental tool of the probabilistic method. It enables one to show the existence of certain objects even though they occur with exponentially small probability. The original proof was not algorithmic, and subsequent algorithmic versions had significant losses in parameters. This paper provides a simple, powerful algorithmic paradigm that converts almost all known applications of the LLL into randomized algorithms matching the bounds of the existence proof. The paper further gives a derandomized algorithm, a parallel algorithm, and an extension to the “lopsided” LLL.
    [Show full text]
  • LNCS 4308, Pp
    On Distributed Verification Amos Korman and Shay Kutten Faculty of IE&M, Technion, Haifa 32000, Israel Abstract. This paper describes the invited talk given at the 8th Inter- national Conference on Distributed Computing and Networking (ICDCN 2006), at the Indian Institute of Technology Guwahati, India. This talk was intended to give a partial survey and to motivate further studies of distributed verification. To serve the purpose of motivating, we allow ourselves to speculate freely on the potential impact of such research. In the context of sequential computing, it is common to assume that the task of verifying a property of an object may be much easier than computing it (consider, for example, solving an NP-Complete problem versus verifying a witness). Extrapolating from the impact the separation of these two notions (computing and verifying) had in the context of se- quential computing, the separation may prove to have a profound impact on the field of distributed computing as well. In addition, in the context of distributed computing, the motivation for the separation seems even stronger than in the centralized sequential case. In this paper we explain some motivations for specific definitions, sur- vey some very related notions and their motivations in the literature, survey some examples for problems and solutions, and mention some ad- ditional general results such as general algorithmic methods and general lower bounds. Since this paper is mostly intended to “give a taste” rather than be a comprehensive survey, we apologize to authors of additional related papers that we did not mention or detailed. 1 Introduction This paper addresses the problem of locally verifying global properties.
    [Show full text]
  • Stabilization
    Chapter 13 Stabilization A large branch of research in distributed computing deals with fault-tolerance. Being able to tolerate a considerable fraction of failing or even maliciously be- having (\Byzantine") nodes while trying to reach consensus (on e.g. the output of a function) among the nodes that work properly is crucial for building reli- able systems. However, consensus protocols require that a majority of the nodes remains non-faulty all the time. Can we design a distributed system that survives transient (short-lived) failures, even if all nodes are temporarily failing? In other words, can we build a distributed system that repairs itself ? 13.1 Self-Stabilization Definition 13.1 (Self-Stabilization). A distributed system is self-stabilizing if, starting from an arbitrary state, it is guaranteed to converge to a legitimate state. If the system is in a legitimate state, it is guaranteed to remain there, provided that no further faults happen. A state is legitimate if the state satisfies the specifications of the distributed system. Remarks: • What kind of transient failures can we tolerate? An adversary can crash nodes, or make nodes behave Byzantine. Indeed, temporarily an adversary can do harm in even worse ways, e.g. by corrupting the volatile memory of a node (without the node noticing { not unlike the movie Memento), or by corrupting messages on the fly (without anybody noticing). However, as all failures are transient, eventually all nodes must work correctly again, that is, crashed nodes get res- urrected, Byzantine nodes stop being malicious, messages are being delivered reliably, and the memory of the nodes is secure.
    [Show full text]
  • Mastering Concurrent Computing Through Sequential Thinking
    review articles DOI:10.1145/3363823 we do not have good tools to build ef- A 50-year history of concurrency. ficient, scalable, and reliable concur- rent systems. BY SERGIO RAJSBAUM AND MICHEL RAYNAL Concurrency was once a specialized discipline for experts, but today the chal- lenge is for the entire information tech- nology community because of two dis- ruptive phenomena: the development of Mastering networking communications, and the end of the ability to increase processors speed at an exponential rate. Increases in performance come through concur- Concurrent rency, as in multicore architectures. Concurrency is also critical to achieve fault-tolerant, distributed services, as in global databases, cloud computing, and Computing blockchain applications. Concurrent computing through sequen- tial thinking. Right from the start in the 1960s, the main way of dealing with con- through currency has been by reduction to se- quential reasoning. Transforming problems in the concurrent domain into simpler problems in the sequential Sequential domain, yields benefits for specifying, implementing, and verifying concur- rent programs. It is a two-sided strategy, together with a bridge connecting the Thinking two sides. First, a sequential specificationof an object (or service) that can be ac- key insights ˽ A main way of dealing with the enormous challenges of building concurrent systems is by reduction to sequential I must appeal to the patience of the wondering readers, thinking. Over more than 50 years, more sophisticated techniques have been suffering as I am from the sequential nature of human developed to build complex systems in communication. this way. 12 ˽ The strategy starts by designing —E.W.
    [Show full text]
  • A Separation Logic for Refining Concurrent Objects
    A Separation Logic for Refining Concurrent Objects Aaron Turon ∗ Mitchell Wand Northeastern University {turon, wand}@ccs.neu.edu Abstract do { tmp = *C; } until CAS(C, tmp, tmp+1); Fine-grained concurrent data structures are crucial for gaining per- implements inc using an optimistic approach: it takes a snapshot formance from multiprocessing, but their design is a subtle art. Re- of the counter without acquiring a lock, computes the new value cent literature has made large strides in verifying these data struc- of the counter, and uses compare-and-set (CAS) to safely install the tures, using either atomicity refinement or separation logic with new value. The key is that CAS compares *C with the value of tmp, rely-guarantee reasoning. In this paper we show how the owner- atomically updating *C with tmp + 1 and returning true if they ship discipline of separation logic can be used to enable atomicity are the same, and just returning false otherwise. refinement, and we develop a new rely-guarantee method that is lo- Even for this simple data structure, the fine-grained imple- calized to the definition of a data structure. We present the first se- mentation significantly outperforms the lock-based implementa- mantics of separation logic that is sensitive to atomicity, and show tion [17]. Likewise, even for this simple example, we would prefer how to control this sensitivity through ownership. The result is a to think of the counter in a more abstract way when reasoning about logic that enables compositional reasoning about atomicity and in- its clients, giving it the following specification: terference, even for programs that use fine-grained synchronization and dynamic memory allocation.
    [Show full text]
  • Lock-Free Programming
    Lock-Free Programming Geoff Langdale L31_Lockfree 1 Desynchronization ● This is an interesting topic ● This will (may?) become even more relevant with near ubiquitous multi-processing ● Still: please don’t rewrite any Project 3s! L31_Lockfree 2 Synchronization ● We received notification via the web form that one group has passed the P3/P4 test suite. Congratulations! ● We will be releasing a version of the fork-wait bomb which doesn't make as many assumptions about task id's. – Please look for it today and let us know right away if it causes any trouble for you. ● Personal and group disk quotas have been grown in order to reduce the number of people running out over the weekend – if you try hard enough you'll still be able to do it. L31_Lockfree 3 Outline ● Problems with locking ● Definition of Lock-free programming ● Examples of Lock-free programming ● Linux OS uses of Lock-free data structures ● Miscellanea (higher-level constructs, ‘wait-freedom’) ● Conclusion L31_Lockfree 4 Problems with Locking ● This list is more or less contentious, not equally relevant to all locking situations: – Deadlock – Priority Inversion – Convoying – “Async-signal-safety” – Kill-tolerant availability – Pre-emption tolerance – Overall performance L31_Lockfree 5 Problems with Locking 2 ● Deadlock – Processes that cannot proceed because they are waiting for resources that are held by processes that are waiting for… ● Priority inversion – Low-priority processes hold a lock required by a higher- priority process – Priority inheritance a possible solution L31_Lockfree
    [Show full text]
  • Scalable NUMA-Aware Blocking Synchronization Primitives
    Scalable NUMA-aware Blocking Synchronization Primitives Sanidhya Kashyap, Changwoo Min, Taesoo Kim The rise of big NUMA machines 'i The rise of big NUMA machines 'i The rise of big NUMA machines 'i Importance of NUMA awareness NUMA node 1 NUMA node 2 NUMA oblivious W1 W2 W3 W4 W6 W5 W6 W5 W3 W1 W4 L W2 File Importance of NUMA awareness NUMA node 1 NUMA node 2 NUMA oblivious W1 W2 W3 W4 W6 W5 W6 W5 W3 W1 W4 NUMA aware/hierarchical L W2 W1 W6 W2 W3 W4 W5 File Importance of NUMA awareness NUMA node 1 NUMA node 2 NUMA oblivious W1 W2 W3 W4 W6 W5 W6 W5 W3 W1 W4 NUMA aware/hierarchical L W2 W1 W6 W2 W3 W4 W5 File Idea: Make synchronization primitives NUMA aware! Lock's research efforts and their use Lock's research efforts Linux kernel lock Dekker's algorithm (1962) adoption / modification Semaphore (1965) Lamport's bakery algorithm (1974) Backoff lock (1989) Ticket lock (1991) MCS lock (1991) HBO lock (2003) Hierarchical lock – HCLH (2006) Flat combining NUMA lock (2011) Remote Core locking (2012) Cohort lock (2012) RW cohort lock (2013) Malthusian lock (2014) HMCS lock (2015) AHMCS lock(2016) Lock's research efforts and their use Lock's research efforts Linux kernel lock Dekker's algorithm (1962) adoption / modification Semaphore (1965) Lamport's bakery algorithm (1974) Backoff lock (1989) Ticket lock (1991) MCS lock (1991) HBO lock (2003) Hierarchical lock – HCLH (2006) Flat combining NUMA lock (2011) Remote Core locking (2012) NUMA- Cohort lock (2012) aware RW cohort lock (2013) locks Malthusian lock (2014) HMCS lock (2015) AHMCS lock(2016)
    [Show full text]