Automated and modular refinement reasoning for concurrent programs Chris Hawblitzel Erez Petrank Shaz Qadeer Serdar Tasiran Microsoft Technion Microsoft Ko¸cUniversity Abstract notable successes using the refinement approach in- clude the work of Abrial et al. [2] and the proof of full We present civl, a language and verifier for concur- functional correctness of the seL4 microkernel [37]. rent programs based on automated and modular re- This paper presents the first general and automated finement reasoning. civl supports reasoning about proof system for refinement verification of shared- a concurrent program at many levels of abstraction. memory multithreaded software. Atomic actions in a high-level description are refined We present our verification approach in the context to fine-grain and optimized lower-level implementa- of civl, an idealized concurrent programming lan- tions. A novel combination of automata theoretic guage. In civl, a program is described as a collection and logic-based checks is used to verify refinement. of procedures whose implementation can use the stan- Modular specifications and proof annotations, such dard features such as assignment, conditionals, loops, as location invariants and procedure pre- and post- procedure calls, and thread creation. Each procedure conditions, are specified separately, independently at accesses shared global variables only through invoca- each level in terms of the variables visible at that tions of atomic actions. A subset of the atomic ac- level. We have implemented as an extension to civl tions may be refined by new procedures and a new the language and verifier. We have used boogie civl program is obtained by replacing the invocation of an to refine a realistic concurrent garbage collection al- atomic action by a call to the corresponding proce- gorithm from a simple high-level specification down dure refining the action. Several layers of refinement to a highly-concurrent implementation described in may be performed until all atomic actions in the final terms of individual memory accesses. program are directly implementable primitives. Un- like classical program verifiers based on Floyd-Hoare 1 Introduction reasoning [24, 34] that manipulate a program and an- notations, the civl verifier manipulates multiple op- We present a technique for verifying a refinement re- erational descriptions of a program, i.e., several layers lation between two concurrent, shared-memory mul- of refinement are specified and verified at once. tithreaded programs. Our work is inspired by step- To prove refinement in civl, a simulation rela- wise refinement [53], where a high-level description is tion between a program and its abstraction is in- systematically refined, potentially via several inter- ferred from checks on each procedure, thus decom- mediate descriptions, down to a detailed implemen- posing a whole-program refinement problem into per- tation. Refinement checking is a classical problem in procedure verification obligations. The computation verification and has been investigated in many con- inside each such procedure is partitioned into \steps" texts, including hardware verification [15] and veri- such that one step behaves like the atomic specifica- fication of cache-coherence protocols and distributed tion and all other steps have no effect on the vis- algorithms [39]. In the realm of sequential software, ible state. This partitioning follows the syntactic 1 structure of the code in a way similar in spirit to threads. We build upon classical work [48, 36] on Floyd-Hoare reasoning. To be able to express the reasoning about non-interference with two distinct per-procedure verification obligation in terms of a col- innovations. First, we do not require the annota- lection of per-step verification tasks, the civl verifier tions to be strong enough to prove program correct- needs to address two issues. First, the notion of a ness but only strong enough to provide the context \step" in the implementation must be defined. The for refinement checking. Program correctness is es- definition of a step can deeply affect the number of tablished via a sequence of refinement layers from an checks that need to be performed and the number of abstract program that cannot fail. Second, to estab- user annotations. Second, it is typically not possible lish a postcondition of a procedure, we do not need to show the correctness of a step from an arbitrary to propagate a precondition through all the yield an- state. A precondition for the step in terms of shared notations in the procedure body. The correctness of variables must be supplied by the programmer and an atomic action specification gives us a simple frame mechanically checked by the verifier. rule|the precondition only needs to be propagated To address the first problem, civl lets the pro- across the atomic action specification. civl further grammer define the granularity of a step, allowing the simplifies the manual annotations required for logical user to specify a semantics with larger atomic actions. non-interference checking by providing a linear type A cooperative semantics for the program is explicitly system [52] that enables logical encoding of thread introduced by the programmer through the use of a identifiers, permissions [8], and disjoint memory [38]. new primitive yield statement; in this semantics a Finally, civl provides a simple module system. thread can be scheduled out only when it is about Modules can be verified separately, in parallel or at to execute a yield statement. The preemptive seman- different times, since the module system soundly does tics of the program is sequentially consistent execu- away with checks that pertain to cross-module inter- tion; all threads are imagined to execute on a single actions. This feature is significant since commutativ- processor and preemption, which causes a thread to ity checks and non-interference checks for location in- be scheduled out and a nondeterministically chosen variants are quadratic, whole program checks involv- thread to be scheduled in, may occur before any in- ing all pairs of yield locations and atomic blocks, or struction.1 Given a program P , civl verifies that the all pairs of actions from a program. Using the module safety of the cooperative semantics of P implies the system, the number of checks is reduced; they become safety of the preemptive semantics of P . This veri- quadratic in the number of yields and atomic blocks fication is done by computing an automata-theoretic within each module rather than the entire program. simulation check [30] on an abstraction of P in which We have implemented civl as a conservative ex- each atomic action of P is represented by only its tension of the boogie verifier. We have used it to mover type [43, 21]. The mover types themselves are verify a collection of microbenchmarks and bench- verified separately and automatically using an auto- marks from the literature [7, 17, 18, 19, 23, 33]. The mated theorem prover [10]. most challenging case study with civl was carried To address the second problem that refinement out concurrently with civl's development and served verification for each step requires invariants about as a design driver. We verified a concurrent garbage the program execution, civl allows the programmer collector, through six layers of refinement, down to to specify location invariants, attached either to a atomic actions corresponding to individual memory yield statement or to a procedure as its pre- or post- accesses. The level of granularity of the lowest-level condition. Each location invariant must be correct implementation distinguishes this verification effort for all executions and must continue to hold in spite from previous attempts in the literature. of potential interference from concurrently executing In conclusion, civl is the first automated verifier for shared-memory multithreaded programs that pro- 1In this paper, we focus our attention on sequential con- sistency and leave consideration of weak memory models to vides the capability to establish a multi-layered re- future work. finement proof. This novel capability is enabled by 2 two important innovations in core verification tech- niques for reducing the complexity of invariants sup- var Color: int; // WHITE=1, GRAY=2, BLACK=3 plied by the programmer and the verification condi- procedure WB(linear tid:Tid) tions solved by the prover. atomic [if (Color == WHITE) Color := GRAY]; { • Reasoning about preemptive semantics is re- var cNoLock:int; placed by simpler reasoning about cooperative cNoLock := GetColorNoLock(tid); operational semantics by exploiting automata- yield Color >= cNoLock; theoretic simulation checking. We are not aware if (cNoLock == WHITE) of any other verifier that combines automata- call WBSlow(tid); based and logic-based reasoning in this style. } procedure WBSlow(linear tid:Tid) • A linear type system establishes invariants about atomic [if (Color == WHITE) Color := GRAY]; disjointness of permission sets associated with { values contained in program variables. These var cLock:int; invariants, communicated to the prover as free call AcquireLock(tid); assumptions, significantly reduce the overhead cLock := GetColorLocked(tid); of program annotations. We are not aware of if (cLock == WHITE) any other verifier that combines type-based and call SetColorLocked(tid, GRAY); logic-based reasoning in this style. call ReleaseLock(tid); } 2 Overview procedure GetColorNoLock(linear tid:Tid) returns (cl:int) atomic [...]; We present an overview of our approach to refinement procedure AcquireLock(linear tid:Tid) on the program in Figure 1, a simplified version of the right [...]; write barrier in a concurrent garbage collector (GC). procedure ReleaseLock(linear tid:Tid) In a concurrent GC, a color (either WHITE, GRAY, or left [...]; BLACK) is associated with each object on the heap. procedure GetColorLocked(linear tid:Tid) Before writing to an address addr, a mutator executes returns (cl:int) both [...]; a write barrier. It checks addr has color WHITE and procedure SetColorLocked(linear tid:Tid, sets it to GRAY, indicating that the object at addr cl: int) atomic [...]; and objects reachable from it should not be garbage collected.