Checking Concurrent Data Structures Under the C/C++11 Memory Model ifact rt * * Comple A t te n * A te s W i E * s e n l C l o P D C o P * * Peizhao Ou and Brian Demsky c u o e m s E P u e e n P R t v e o d t * y * s E a a l d u e a University of California, Irvine, USA t fpeizhaoo,[email protected] Abstract 1.1 Concurrent Data Structure Specifications Concurrent data structures often provide better performance Researchers have developed several techniques for specify- on multi-core processors but are significantly more difficult ing correctness properties of concurrent data structures writ- to design and test than their sequential counterparts. The ten for the sequential consistency memory model. While C/C++11 standard introduced a weak memory model with these techniques cannot handle the behaviors exhibited by support for low-level atomic operations such as compare data structure implementations under relaxed memory mod- and swap (CAS). While low-level atomic operations can els, they provide insight into intuitive approaches for speci- significantly improve the performance of concurrent data fying concurrent data structure behavior. structures, they introduce non-intuitive behaviors that can A common approach for specifying the correctness of increase the difficulty of developing code. concurrent data structures is in terms of sequential execu- In this paper, we develop a correctness model for con- tions of either the concurrent data structure or a simplified current data structures that make use of atomic operations. sequential version. The problem then becomes how do we Based on this correctness model, we present CDSSPEC, a map a concurrent execution to a sequential execution? A specification checker for concurrent data structures under the common criterion is linearizability — linearizability states C/C++11 memory model. We have evaluated CDSSPEC on that a concurrent operation can be viewed as taking effect at 10 concurrent data structures, among which CDSSPEC de- some time between its invocation and its response [32]. tected 3 known bugs and 93% of the injected bugs. An equivalent sequential data structure is a sequential version of a concurrent data structure that can be used to Categories and Subject Descriptors D.1.3 [Programming express correctness properties by relating executions of the Techniques]: Concurrent Programing; D.2.4 [Software En- concurrent data structure to executions of the equivalent se- gineering]: Software/Program Verification quential data structure. The equivalent sequential data struc- ture is often simpler, and in many cases, one can simply use Keywords Relaxed Memory Models; Concurrent Data existing well-tested implementations from libraries. Structure Specifications A history is a total order of the method invocations and 1. Introduction responses in an execution. A sequential history is one where Concurrent data structures can improve scalability by sup- all invocations are immediately followed by the correspond- porting multiple simultaneous operations, reducing cache ing responses. A concurrent object is linearizable if for all coherence traffic, and reducing the time taken by individual executions: (1) the invocations and responses can be re- data structure operations. Researchers have developed many ordered to yield a sequential history under the constraint that concurrent data structures with these goals [24, 35, 38]. Con- an invocation cannot be reordered before the preceding re- current data structures often use sophisticated techniques sponses and (2) the concurrent execution yields the same be- including low-level atomic instructions (e.g., compare and havior as this sequential history. swap), careful reasoning about the order of memory opera- A weaker variation of linearization is sequential consis- tions, and fine-grained locking. Developing correct concur- tency1. Sequential consistency only requires that there exists rent data structures requires subtle reasoning about the data a sequential history that is consistent with the program order structure and the memory model, and there are numerous ex- (intra-thread order). This order does not need to be consis- amples of incorrect concurrent data structures [40, 41]. tent with the order that operations were actually issued in. Unfortunately, efficient implementations of many com- mon data structures, e.g., RCU [24], MS Queue [38], etc., for the C/C++ memory model are neither linearizable [29] Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice nor sequentially consistent! and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. 1 PPoPP ’17, February 04 - 08, 2017, Austin, TX, USA Note that the term sequential consistency in the literature is applied to both Copyright © 2017 held by owner/author(s). Publication rights licensed to ACM. ACM 978-1-4503-4493-7/17/02. $15.00 the consistency model that data structures expose to clients as well as the DOI: http://dx.doi.org/10.1145/3018743.3018749 guarantees that the memory system provides for loads and stores. 1.2 New Challenges from the C/C++ Memory Model queue and dequeue operations. Figure 1 shows a problematic The C/C++ memory model [1, 2] brings the following execution of such a queue: (1) thread A initializes the fields two challenges that prevent the application of previous ap- of a new object X; (2) thread A enqueues the reference to X; proaches for specifying the correctness of concurrent data (3) thread B dequeues the reference to X; (4) thread B reads structures: the fields of X through the dequeued reference. In (4), thread Relaxed Executions Break Existing Data Structure Con- B could fail to see the initializing writes from (1). This sur- sistency Models: C/C++ data structures often expose prising behavior could occur if the compiler or CPU reorders clients to weaker (non-SC) behaviors to gain performance. the initializing writes to be executed after the enqueue oper- A common guarantee is to provide happens-before syn- ation, i.e., reordering (1) after (2). chronization between operations that perform updates and Thread A Thread B the operations that read those updates. These data struc- X->field1=1; // (1) Obj *r1=q->deq(); // (3) tures often do not guarantee that different threads ob- ... if (r1!=NULL) serve updates in the same order. For example, even when q->enq(X); // (2) int r2=r1->field1; // (4) one uses the relatively strong release and acquire Figure 1. The problematic execution of a concurrent queue (i.e., std::memory_order_release and std::memory_ in which the load in (4) does not see the initialization in (1) order_acquire) memory orderings in C++, it is possible (i.e., r2!=1) when the dequeue operation in (3) dequeues for two different threads to observe two stores happening from the enqueue operation in (2) in different orders. Thus, many data structures legitimately admit executions for which there are no sequential histories 1.3 Specification Language and Tool Support that preserve program order. This paper presents CDSSPEC, a specification checking tool Like many other relaxed memory models, the C/C++ that is designed to be used in conjunction with model check- memory model does not define a total order over all mem- ing tools. We have implemented it as a plugin for the CD- ory operations. Instead of a total order, the C/C++ memory SCHECKER model checker. After implementing a concur- model is formulated as a graph of memory operations with rent data structure, developers annotate their code with a edges representing several partial orders. This complicates CDSSPEC specification. To test their implementation, de- the application of traditional approaches to correctness, e.g., velopers compile the data structure with the CDSSPEC spec- linearization cannot be applied. Specifically, approaches that ification compiler to extract the specification and generate relate the behaviors of concurrent data structures to analo- code that is instrumented with specification checks. Then, gous sequential data structures break down due to the ab- developers compile the instrumented program with a stan- sence of a total ordering of the memory operations. While dard C/C++ compiler. Finally, developers run the binary un- many dynamic tools [40, 50] for exploring code behavior der the CDSSPEC checker. CDSSPEC then exhaustively ex- under relaxed models do as a practical matter print out an plores the behaviors of the unit test and generates diagnostic execution in some order, this order insufficient as relaxed reports for executions that violate the specification. memory models generally make it possible for a data struc- Researchers have developed tools for exploring the be- ture operation (1) to see the effects of operations that appear havior of code under the C/C++ memory model including later in any such order (e.g., a load can read from a store that CDSCHECKER [40], CPPMEM [13], and Relacy [50], and appears later in the order) and (2) to see the effects of older, also under a revised C/C++ memory model [11]. While these stale updates. tools can be used to explore executions, they can be chal- Constraining Reorderings (Specifying Synchronization lenging to use for testing as they don’t provide support be- Properties): 2 Synchronization in C/C++ orders memory yond assertions for specifying data structure behavior. operations to different locations. Concurrent data structures often establish synchronization to avoid exposing their users 1.4 Contributions to highly non-intuitive behaviors that may break client code.