A Historical Perspective on Runtime Assertion Checking in Software Development Lori A. Clarke David S. Rosenblum Department of Computer Science Department of Computer Science University of Massachusetts University College London Amherst, MA 01003 London WC1E 6BT USA United Kingdom [email protected] [email protected] Abstract thoroughly a software system’s requirements are This report presents initial results in the area of documented, and no matter how carefully and software testing and analysis produced as part of the elegantly the system’s design has been constructed, Software Engineering Impact Project. The report inevitably latent faults, or incorrect program describes the historical development of runtime statements, are introduced in the system’s assertion checking, including a description of the implementation. These faults may be revealed during origins of and significant features associated with various levels of testing, or they may make a more assertion checking mechanisms, and initial findings inopportune appearance during field use by end- about current industrial use. A future report will users. In such situations the faults are typically provide a more comprehensive assessment of manifested externally as program failures, such as development practice, for which we invite readers of unexpected outputs, or other undesirable outcomes this report to contribute information. such as a program crash. Such failures provide developers with precious little information for initiating the task of correlating the simple external 1 Introduction evidence of failure with the complexity of searching numerous possible locations for the faults that caused The Software Engineering Impact Project is them. documenting the impact that software engineering research has had on computer science research and Assertions are one of the most useful automated on software development practice. The authors of techniques available for detecting faults and this paper are responsible for documenting the impact providing information about their locations, even for of research in software testing and analysis for the faults that are traversed during execution but do not Impact Project. One aspect of testing and analysis lead to failures. As described in this report, that has clearly had an impact is the widespread use assertions have a long and distinguished history in of assertions, particularly for use in automated the annals of software engineering and programming runtime detection of faults. This report documents language design. Initially developed as a means of the results of our initial assessment of assertions and stating expected or desired program properties as a narrates the history of software engineering research necessary step in constructing formal, deductive as it relates to the evolution and maturation of proofs of program correctness, assertions have found runtime assertion checking capabilities in many other applications in software engineering over programming languages and software development the years, albeit primarily in the later stages of support tools. development (particularly in the development and execution of source code). They are an important Despite decades of research into powerful element of model checking, an alternative and software engineering technologies, and despite the actively studied approach to program verification, in continual discovery of tenets of good software which the state space resulting from a program’s engineering practice, software development remains execution is checked against logical assertions an exceedingly complex endeavor. No matter how expressing temporal safety and liveness properties (e.g., SPIN [38]). They are embedded in the type handling, including the continuation semantics, may systems of many programming languages that alter the flow of control in the program considerably. support strong typing of data and objects (e.g., Ada Assertions, on the other hand, describe program [3, 30]). And they are frequently used in an informal invariants that are by definition expected always to fashion by developers to describe module interfaces hold. When the logical expression in an assertion is more precisely in order to assist understanding by found to be false, a fault has occurred. This fault is other developers. Yet the application of assertions reported and execution continues or terminates, having the greatest impact on development practice, depending on the severity of the fault. As discussed and the one on which we focus in this report, is their below, assertion capabilities often have expressive use for automated runtime fault detection, in which notations for representing an assertion and for formal assertion checks are instrumented into a indicating the scope or states in which it applies. program for execution along with the program’s Exceptions have often been used as a mechanism to application logic. implement assertions, as noted below. Assertions may be applied for automated fault Although evidence of institutionalized use of detection during any activity in which a program is assertions in software development projects is hard to executed, including debugging, testing, and come by, it is well-known that assertions have long production use. Assertions may be used for been used by seasoned software developers, who secondary purposes, such as documentation or to eventually come to learn the value of seeding their support static analysis of a program. For the code with automated defensive checks at the outset of purposes of our assessment, an assertion capability development, thereby avoiding the pain of belated, comprises at a minimum the following features: trial-and-error insertion of print statements during debugging [36]. Indirect evidence for more recent • a high-level language for representing logical growth in the popularity of assertions among expressions (typically Boolean-valued developers is to be found in the proliferation of expressions) to characterize invalid program assertion capabilities for widely-used programming execution states; languages, especially C++, Java, and C#. Thus, assertions have had a significant impact on software • a syntax for associating the logical expressions with a program and applying them to well- development for the past two to three decades, and defined states of the program; one can quite easily trace the research forbears of the various assertion capabilities developers have used • a means for automatic translation of the logical over the years. expressions into executable statements that evaluate the expressions on the appropriate state The next section of this report presents a brief or states of the associated program; and history of the research ideas that have contributed to the assertion capabilities that are available for use in • a predefined or user-defined runtime response current development practice. Section 3 summarizes that is invoked if the logical expression is the characteristics of the most widely-used assertion violated upon evaluation. capabilities, and Section 4 describes experimental evaluations about how the use of assertions impacts This combination of features has been shown to the software development process. Section 5 provide a powerful, flexible, high-level facility for concludes with a discussion of future plans for automated fault detection during program execution. further assessments of assertion capabilities, which Note that according to this description, the time- will be undertaken after receiving feedback from honored tradition of using “print” statements as readers of this report and from the community of debugging instrumentation does not qualify as an researchers and practitioners. assertion capability, since it represents a manual, low-level, and typically ad hoc implementation of assertion-like checks. Note also that exceptions are 2 A History of the Technology similar to the above description. Exceptions, however, are often intended to describe how the This section presents a concise history of assertions. program execution is to behave when an exceptional, The history is organized around the key ideas that but valid, event occurs [29]. It is often a matter of arose in the development of assertions. taste as to how exceptional the events that trigger an exception really are. When execution leads to signaling an exception, corrective actions are defined in an exception handler. The semantics of exception 2.1 Logical Assertions as Characterizations of establish the truth of Q in the state immediately Behavior following its execution. Hoare defined axiom schemas for primitive program statements (such as While the use of program instrumentation assignment statements) and inference rules for mechanisms for dumping low-level execution and compound statements (such as conditionals and memory traces is probably as old as computing itself loops) and compositions of statements. Hoare then (e.g., Evans and Darley [22]), the origins of formal, sketched a proof method whereby a programmer logical assertions about program behavior predates would supply a precondition P and postcondition Q computers [27, 90]. Goldstine’s and von Neumann’s for a program S and then apply the system of proof work on reasoning about programs used the term rules to establish the validity of the statement {P} S “assertion” for documenting invariants in algorithms. {Q}. Turing also advocated that assertions be used to document the states that can be associated with Several mechanical program verification efforts