sign in sign up
<<
Home , Formal methods, Formal system, Jeannette Wing, Jim Woodcock

Formal Methods: State of the Art and Future Directions EDMUND M. CLARKE, JEANNETTE M. WING, ET AL.1 Computer Department, Carnegie Mellon University, Pittsburgh, PA ͗[email protected]͘ and ͗[email protected]͘

1. INTRODUCTION The first part of this report assesses the state of the art in specification and Hardware and will in- verification. For verification, we high- evitably grow in scale and functionality. light advances in and Because of this increase in complexity, proving. In the three sections the likelihood of subtle errors is much on specification, model checking, and greater. Moreover, some of these errors theorem proving, we explain what we may cause catastrophic loss of money, mean by the general technique and time, or even life. A major goal of is to enable de- briefly describe some successful case velopers to construct systems that oper- studies and well-known tools. The sec- ate reliably despite this complexity. One ond part of this report outlines future way of achieving this goal is by using directions in fundamental concepts, new methods, which are mathemati- methods and tools, integration of meth- cally based , techniques, and ods, and education and technology tools for specifying and verifying such transfer. We close with summary re- systems. Use of does marks and pointers to resources for not a priori guarantee correctness. more information. However, they can greatly increase our understanding of a by revealing inconsistencies, ambiguities, and incom- 2. STATE OF THE ART pleteness that might otherwise go unde- In the past, the use of formal methods tected. in practice seemed hopeless. The nota- tions were too obscure, the techniques did not scale, and the tool support was 1 Working Group members include Rajeev Alur, inadequate or too hard to use. There Edmund Clarke (Co-Chair), Rance Cleaveland, were only a few nontrivial case studies David Dill, Allen Emerson, Stephen Garland, Steven German, , Anthony Hall, and together they still were not convinc- Thomas Henzinger, Gerard Holzmann, Cliff ing enough to the practicing software or Jones, Robert Kurshan, , Kenneth hardware engineer. Few people had the McMillan, J Moore, Doron Peled, Amir Pnueli, training to use them effectively on the John Rushby, Natarajan Shankar, Joseph Sifakis, Prasad Sistla, Bernhard Steffen, Pierre Wolper, job. (Co-Chair), , and Only recently have we begun to see a Pamela Zave. more promising picture of formal meth-

This research is sponsored in part by the Wright Laboratory, Aeronautical Systems Center, Air Force Materiel Command, USAF, and the Advanced Research Projects Agency (ARPA) under grant number F33615-93-1-1330. Views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing official policies or endorsements, either expressed or implied, of Wright Laboratory or the Government. Permission to make digital/hard copy of part or all of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication, and its date appear, and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. © 1996 ACM 0360-0300/96/0400–0626 $03.50

ACM Surveys, Vol. 28, No. 4, December 1996 Formal Methods • 627 ods. For software specification, industry [ISO 1987] wed two different methods, is open to trying out such as Z one for handling rich state spaces and to document a system’s properties more one for handling complexity due to con- rigorously. For hardware verification, currency. Common to all these methods industry is adopting techniques such as is the use of the mathematical concepts model checking and theorem proving to of and composition. complement the more traditional one of The process of specification is the act simulation. In both areas, researchers of writing things down precisely. The and practitioners are performing more main benefit in so doing is intangible— and more industrial-sized case studies, gaining a deeper understanding of the and thereby gaining the benefits of us- system being specified. It is through ing formal methods. this specification process that develop- ers uncover design flaws, inconsisten- 2.1 Specification cies, ambiguities, and incompletenesses. A tangible byproduct of this process, Specification is the process of describing however, is an artifact that can itself be a system and its desired properties. For- formally analyzed, for example, checked mal specification uses a with a to be internally consistent or used to mathematically defined and se- derive other properties of the specified mantics. The kinds of system properties system. The specification is a useful might include functional behavior, tim- communication device between cus- ing behavior, performance characteris- tomer and designer, between designer tics, or internal structure. So far, speci- and implementor, and between imple- fication has been most successful for mentor and tester. It serves as a com- behavioral properties. One current panion document to the system’s source trend is to integrate different specifica- code, but at a higher level of descrip- tion languages, each able to handle a tion. different aspect of a system. Another is to handle nonbehavioral aspects of a Notable Examples system such as its performance, real- CICS. Oxford University and IBM time constraints, policies, and Hursley Laboratories collaborated in architectural design. the 1980s on using Z to formalize part Some formal methods such as Z of IBM’s Customer Information Con- [Spivey 1988], VDM [Jones 1986], and trol System, an online transaction Larch [Guttag and Horning 1993] focus processing system with thousands of on specifying the behavior of sequential installations worldwide [Houston and systems. States are described in terms King 1991]. Measurements taken by of rich mathematical structures such as IBM throughout the development pro- sets, relations, and functions; state cess indicated an overall improve- transitions are given in terms of pre- ment in the quality of the product, a and post-conditions. Other methods reduction in the number of errors dis- such as CSP [Hoare 1985], CCS [Milner covered, and earlier detection of er- 1980], Statecharts [Harel 1987], Tempo- rors found in the process. IBM also ral [Pnueli 1981; Manna and estimated a 9% reduction in the total Pnueli 1991; Lamport 1984], and I/O development cost of the new release. automata [Lynch and Tuttle 1987] focus The success of this work is well on specifying the behavior of concurrent known and resulted in the Queen’s systems; states typically range over Award for Technological Achieve- simple domains like integers or are left ment. It inspired many others to fol- uninterpreted, and behavior is defined low suit. in terms of sequences, trees, or partial orders of events. Still others such as CDIS. In 1992 Praxis delivered to the RAISE [Nielsen et al. 1989] and LOTOS UK Civil Aviation Authority the CCF

ACM Computing Surveys, Vol. 28, No. 4, December 1996 628 • E. M. Clarke et al.

Display , a part of ular specifications [Heninger 1980; the new air traffic management sys- Janicki et al. 1996]. Many would ex- tem for London’s airspace [Hall 1996]. pect that the use of SPARK would add CDIS is a distributed fault-tolerant to the cost of the software, while im- system implemented on nearly 100 proving its quality. The added qual- computers linked in a dual local-area ity, however, decreased the overall network. Praxis used formal methods cost of because as an integral part of the development of the huge savings in testing. The process and in conjunction with other use of SPARK annotations to specify software engineering, project manage- the behavior of the modules led to ment, and quality assurance tech- software that is close to being “correct niques. During requirements analy- by construction,” and hence passes its sis, formal description supplemented tests instead of requiring expensive informal and structured requirements rework. notations. At the system specification stage, an abstract VDM model was TCAS. In the early 1990s, the Safety- developed in conjunction with con- Critical Systems Research Group at crete definitions, semi- the University of California, Irvine formal definitions of the concurrent (now at the University of Washington) produced a formal requirements spec- behavior, and definitions of external ification for the Traffic Collision interfaces. During design, the ab- Avoidance System (TCAS) II, required stract VDM was refined into more on all commercial aircraft flying in concrete module specifications. At a US airspace. They used the Require- lower level, the software for the dual ments State Machine Language LAN was specified and developed for- (RSML), which is based on State- mally using CCS. charts with changes made to over- Productivity on the project was the come difficulties found during the same as or better than on comparable specification process. Although an in- projects carried out using informal dustry group was attempting to pro- methods. There was, in other words, vide an English language specifica- no net cost in using formal methods. tion at the same time, the complexity However, the perceived and measured of TCAS impeded that process; even- quality of the software was much tually the English specification effort higher. The delivered software had a was abandoned and the RSML specifi- defect rate of about 0.75 faults per cation was adopted instead. After a thousand lines of code, a figure two to group of industry and university rep- ten times better than that for pub- resentatives produced a first draft of lished projects and comparable soft- the TCAS II specification, a private ware in air traffic control applications company on behalf of the Federal Avi- that did not use formal methods. ation Administration took over the specification effort; official TCAS II Lockheed C130J. Praxis has been re- documentation still uses RSML. Both cently working with Lockheed on ana- the private company and the original lyzing the code for the avionic soft- university researchers have produced ware of the Lockheed C130J [Croxford automated tools for RSML including and Sutton 1995] being supplied to simulators, generators and the US Air Force and the RAF. The other test tools, and safety analysis software is coded in the SPARK-anno- tools. The TCAS II specification has tated of Ada. Specifications are been automatically checked for math- written in the Software Productivity ematical and consis- Consortium’s CORE [SPC tency [Heimdahl and Leveson 1996] 1993], which is based on Parnas’s tab- and provably correct code can now be

ACM Computing Surveys, Vol. 28, No. 4, December 1996 Formal Methods • 629

automatically generated from RSML Institute of Standards and Technol- specifications. ogy [Kuhn and Dray 1990]. The TCAS II project demonstrated —Telephony. Various features of (1) the practicality of writing a formal AT&T’s 5ESS telephone switching requirements specification for a com- system using Esterel [Jagadeesan et plex process-control system and (2) al. 1996] and combinations of Z and the feasibility of building a formal CSP [Mataga and Zave 1995; Zave model of a system that can be read 1995; Zave and Jackson 1996]; the and reviewed by application experts University of Passau and without special training. Nixdorf’s joint work on customizable telephone services and features [Stef- Other case studies in formal specifica- fen et al. 1996], recently done for tion have been performed primarily on Deutsche Telekom. commercial and safety-critical systems. Some are proprietary or lack documen- —Transportation. The automatic train tation that we can cite. To give the protection system for the Paris Metro reader a sense of the applicability of [Carnot et al. 1992; Guiho and Hen- formal methods, we list some for which nebert 1990]; British Rail’s signaling we can provide references. rules [King 1994]; and the on-board avionics software for an Israel air- —. An HP Medical Instru- craft [Harel 1992]. ments real-time for storing See also Craigen et al. [1993a, 1993b, patient monitoring information [Bear 1994, 1995] for a description of 12 case 1991]. studies in formal methods (most cited in —Devices. A Tektronix family of oscillo- the preceding). scopes [Delisle and Garlan 1990]; a Schlumberger line of household elec- tricity meters [Arnold et al. 1996]. 2.2 Verification —Hardware. An INMOS floating-point Two well-established approaches to ver- processor [Barrett 1989]; the virtual ification are model checking and theo- channel processor in INMOS’s T9000 rem proving. They go one step beyond transputer [Barrett 1995]. (Also see specification; these formal methods are Section 2.2.2.) used to analyze a system for desired —Medical. The Clinical Neutron Ther- properties. apy System at the University of 2.2.1 Model Checking. Model check- Washington (cyclotron controller) ing is a technique that relies on build- [Jacky 1995]. ing a finite model of a system and —Nuclear. Argonne National Laborato- checking that a desired property holds ry’s work on the Reactor Safety Sys- in that model. Roughly speaking, the tem for the Experimental Breeder Re- check is performed as an exhaustive actor-II [Chisolm et al. 1987; Kljaich state space search that is guaranteed to et al. 1989]; the shutdown system of terminate since the model is finite. The the Darlington Nuclear Generating technical challenge in model checking is System in Canada [Archinoff et al. in devising and data struc- 1990]. tures that allow us to handle large —Security. The security policy model for search spaces. Model checking has been the NATO Air Command and Control used primarily in hardware and proto- System [Boswell 1995]; the secure col verification [Clarke and Kurshan transmission of datagrams in the 1996]; the current trend is to apply this Multinet Gateway System [Dinolt et technique to analyzing specifications of al. 1984]; the Token-based Access software systems. Control System of the US National Two general approaches to model

ACM Computing Surveys, Vol. 28, No. 4, December 1996 630 • E. M. Clarke et al. checking are used in practice today. The systems efficiently, thereby increasing first, temporal model checking, is a tech- the size of the systems that could be nique developed independently in the verified. Other promising approaches to 1980s by Clarke and Emerson [1981] alleviating state explosion include the and by Queille and Sifakis [1982]. In exploitation of partial order information this approach specifications are ex- [Peled 1996], localization reduction pressed in a temporal logic [Pnueli [Kurshan 1994a; Kurshan 1994b] and 1981] and systems are modeled as finite semantic minimization [Elseaidy et al. state transition systems. An efficient 1996] to eliminate unnecessary states search procedure is used to check if a from a system model. given finite state transition system is a Model checkers today are routinely model for the specification.2 expected to handle systems with be- In the second approach, the specifica- tween 100 and 200 state variables. They tion is given as an automaton; then the have checked interesting systems with system, also modeled as an automaton, 10120 reachable states [Burch et al. is compared to the specification to de- 1994] and, by using appropriate ab- termine whether its behavior conforms straction techniques, they can check to that of the specification. Different systems with an essentially unlimited notions of conformance have been ex- number of states [Clarke et al. 1992]. As plored, including language inclusion a result, model checking is now power- [Har’El and Kurshan 1990; Kurshan ful enough that it is becoming widely 1994a], orderings [Cleave- used in industry to aid in the verifica- land et al. 1993; Roscoe 1994], and ob- tion of newly developed designs. servational equivalence [Cleaveland et al. 1993; Fernandez et al. 1996; Roy and Notable Examples de Simone 1990]. Vardi and Wolper [1986] showed how the temporal-logic IEEE Futurebusϩ. In 1992 Clarke and model-checking problem could be recast his students at Carnegie Mellon used in terms of automata, thus relating SMV [McMillan 1993] to verify the these two approaches. cache coherence protocol described in In contrast to theorem proving, model the IEEE Futurebusϩ Standard checking is completely automatic and 896.1-1991 [Clarke et al. 1993; Long fast, sometimes producing an answer in 1993]. They constructed a precise a matter of minutes. Model checking model of the protocol in the SMV in- can be used to check partial specifica- put language and then used SMV to tions, and so can provide useful infor- show that the resulting transition mation about a system’s correctness system satisfied a even if the system has not been com- of cache coherence. They found a pletely specified. Above all, model number of previously undetected er- checking’s tour de force is that it pro- rors and potential errors in the design duces counterexamples, which usually of the protocol. This appears to be the represent subtle errors in design, and first time that an automatic verifica- thus can be used to aid in debugging. tion tool has been used to find errors The main disadvantage of model in an IEEE standard. Although the checking is the state explosion problem. development of the protocol began in In 1987 McMillan used Bryant’s ordered 1988, all previous attempts to vali- binary decision diagrams (BDDs) [Bry- date it were based entirely on infor- ant 1986] to represent state transition mal techniques. IEEE SCI. In 1992 Dill and his col- 2 Exhaustive state space search, or reachability analysis, dates back to the earliest papers on Petri leagues at Stanford developed the nets. The term “model checking” was coined by Murphi finite state verification sys- Clarke and Emerson [1981]. tem and verified the cache coherence

ACM Computing Surveys, Vol. 28, No. 4, December 1996 Formal Methods • 631

protocol of the Scalable Coherent In- methods in a rou- terface, IEEE Standard 1596-1992 tine project within [Dill et al. 1992]. The SCI standard AT&T [Chaves 1992; Holzmann defines several protocols, each a sub- 1994]. The project lasted from 1989 of the next. They constructed a until 1992. Formal modeling and au- model of a “typical” protocol and sup- tomated verification were applied to plied a specification of properties nec- the development of the International essary for cache coherence. To avoid Telecommunications Union (formerly errors in the translation, they based CCITT) ISDN/IUPP (ISDN User Part their model directly on the C code Protocol). A team of five “verification that is given as a definition of the SCI engineers” formalized 145 require- standard. Since the number of states ments in temporal logic, and rendered of the model could be very large, they the proofs with the help of a special- verified only small instances of the purpose model checker [Holzmann system. Even with this simplification, 1992; Holzmann and Patti 1989]. A they found several errors in the proto- total of 7,500 lines of Specification col, ranging from omissions of vari- and Description Language (SDL) able initializations to subtle logical source code (excluding comments) was errors. These errors existed in the verified; 112 errors were revealed rather basic subset that they defined, (and fixed) in the high-level designs; although the protocol had been exten- approximately 55% of the original de- sively discussed, simulated, and even sign requirements were discovered to implemented. be logically inconsistent.

Stereo components. One of the emerg- HDLC. A high-level data link control- ing application domains of automatic ler (HDLC) transmitter core was de- verification is the design of hybrid signed at the Bell Labs Microelectron- systems, which consist of both dis- ics Design Center in Madrid, Spain crete and continuous components. In for an Application-Specific Integrated 1994, Bosscher, Polak, and Vaan- Circuit of telecommunication drager won a best-paper award for macrocells. The standard design pro- proving manually the correctness of a cess included capture at the register- control protocol used in Philips stereo transfer level using VHDL, simula- components [Bosscher et al. 1994]. In tion, and synthesis. In 1996, late in 1995, Ho and Wong-Toi [1995] veri- the process, the formal verification fied an abstraction of the protocol us- team at Bell Labs offered to run some ing the symbolic model checker additional functional verification on HyTech and inferred, fully automati- the design [Calero et al. 1996]. Since cally, a more efficient timing of the this design was considered to be al- protocol than the one used by Philips. most finished, it was not expected Also in 1995, Daws and Yovine [1995] that any errors would be found. used the verification tool Kronos to Within five hours of work, six proper- check automatically all the properties ties were specified and five were veri- stated and handproved by Bosscher et fied, using the FormalCheck verifica- al. In 1996, Bengtsson and his col- tion tool [DePalma and Glaser 1996]. leagues model checked the entire pro- The sixth property was found by For- tocol, thus completing the quest of malCheck to fail, uncovering a bug fully automating a human that that would have at least reduced the as little as two years ago was consid- throughput of the HDLC channel. ered far out of reach for algorithmic More likely, this bug would have con- methods [Bengtsson et al. 1996]. fused the higher-level protocols, caus- ISDN/ISUP. The NewCoRe Project ing lost transmissions. It took just a was the first full-scale application of few minutes to identify and propose a

ACM Computing Surveys, Vol. 28, No. 4, December 1996 632 • E. M. Clarke et al.

fix for a design error that managed to not directly analyzable. However, by escape many hours of logic simula- using the semantic minimization fea- tion. The error was corrected and the ture of the Concurrency Workbench, correction was formally verified using they were able to construct automati- FormalCheck. Plans are now in the cally a much smaller system with the works at the Madrid design center to same timing properties that could be include model checking as part of the analyzed. In the course of their analy- standard design process. sis they uncovered an error in a timer setting that, if undetected, could have PowerScale. In 1995 a group at Bull in caused the active structural control collaboration with researchers of the component to worsen, rather than Verimag Laboratory used LOTOS to dampen, the vibration experienced by describe the processors, memory con- buildings during earthquakes. troller, and bus arbiter of the multi- processor architecture called Power- Other successful industrial-sized case Scale. This architecture is based on studies in model checking are too nu- IBM’s PowerPC microprocessor and is merous to list. Evidence that model used in Bull’s Escala series of servers checking has come of age is that indus- and workstations.௢ They identified try is building its own model checkers four correctness properties, which ex- or simply using existing ones. Listed in press the essential requirements for a the following are some well-known proper functioning of the arbitration model checkers, roughly categorized ac- , and formalized the proper- cording to whether the specification ties and algorithm in terms of bisimu- they check is given as a logical formula lation relations (modulo ) or as a machine: between finite labelled transition sys- Temporal logic model checkers. The tems. Using the compositional and on- very first two model checkers were the-fly model-checking techniques im- EMC [Clarke and Emerson 1981; plemented in the Cæsar/Alde´baran Clarke et al. 1986; Browne et al. Development Package (CADP) tool- 1986] and CÆSAR [Queille and Si- box, the correctness of the arbitration fakis 1981; Fernandez et al. 1996]. algorithm was established automati- SMV [McMillan 1993] is the first cally in a few minutes [Chehaibar et model checker to use BDDs. The Spin al. 1996]. system [Gerth et al. 1995; Holzmann Buildings. In 1995 civil engineers at 1991] uses partial order reduction to North Carolina State University used reduce the state explosion problem the Concurrency Workbench to ana- [Holzmann and Peled 1994; Peled lyze the timing properties of a distrib- 1996]. Murphi [Dill et al. 1992] and uted active structural control system UV [Kaltenbach 1994] are based on [Elseaidy et al. 1996]. The system in the Unity question was designed to make build- [Chandy and Misra 1988]. The Con- ings more resistant to earthquakes by currency Workbench [Cleaveland et sampling the forces being applied to al. 1993] verifies CCS processes for the structure and using hydraulic ac- properties expressed as mu-calculus tuators to exert countervailing forces. formulas. SVE [Filkorn et al. 1994], The engineers first coded their design FORMAT [Damm et al. 1995; Damm in a timed version of the CCS lan- and Delgado-Kloos 1996], and CV [De´- guage; the resulting model contained harbe and Borrione 1995] all focus on states and was hardware verification. HyTech [Alur 1019 ء in excess of 2.12 et al. 1996] is a model checker for hybrid systems; Kronos [Daws and ௢ PowerScale and Escala are registered trade- Yovine 1995; Henzinger et al. 1994], marks of Bull. for real-time systems.

ACM Computing Surveys, Vol. 28, No. 4, December 1996 Formal Methods • 633

Behavior conformance checkers. The sified in a spectrum from highly auto- Cospan/FormalCheck system [De- mated, general-purpose programs to in- Palma and Glaser 1996; Har’El and teractive systems with special-purpose Kurshan 1990] is based on showing capabilities. The automated systems inclusion between omega automata. have been useful as general search pro- FDR [Roscoe 1994] checks refinement cedures and have had noteworthy suc- between CSP programs; most re- cess in solving various combinatorial cently, it has been used to verify and problems. The interactive systems have debug the Needham-Schroeder au- been more suitable for the systematic thentication protocol [Lowe 1996]. formal development of and The Concurrency Workbench [Cleave- in mechanizing formal methods. land et al. 1993] checks a similar no- In contrast to model checking, theo- tion of refinement between CCS pro- rem proving can deal directly with infi- grams; it and the tool Auto [Roy and nite state spaces. It relies on techniques de Simone 1990] may also be used to such as structural induction to prove minimize systems with respect to ob- over infinite domains. Interactive theo- servational equivalence and to deter- rem provers, by definition, require in- mine if two systems are observably teraction with a human, so the theorem- equivalent. proving process is slow and often error- Combination checkers. Berkeley’s HSIS prone. In the process of finding the [Hojati et al. 1993] combines model proof, however, the human user often checking with language inclusion; gains invaluable insight into the system Stanford’s STeP [Bjørner et al. 1996] or the property being proved. system, with deductive methods; and VIS [Brayton et al. 1996], with logic Notable Examples synthesis. The PVS theorem prover [Owre et al. 1992] has a model checker SRT division algorithm. In 1995 for the modal mu-calculus [Rajan et al. Clarke, German, and Zhao used auto- 1995]. METAFrame [Steffen et al. matic theorem-proving techniques 1996] is an environment that supports based on symbolic algebraic manipu- model checking in the entire software lation to prove the correctness of an development process. SRT division algorithm similar to the one in the Pentium [Clarke et al. 2.2.2 Theorem Proving. Theorem prov- 1996]. This verification method runs ing is a technique by which both the automatically and could have de- system and its desired properties are tected the error in the Pentium, expressed as formulas in some mathe- which was caused by a faulty quotient matical logic. This logic is given by a digit selection table. Later Ruess, , which defines a set of Shankar, and Srivas used SRI’s gen- and a set of rules. eral-purpose theorem prover, PVS Theorem proving is the process of find- [Owre et al. 1992], on this same ex- ing a proof of a property from the axi- ample [Ruess et al. 1996]. oms of the system. Steps in the proof appeal to the axioms and rules, and Processor designs. The Verity verifica- possibly derived definitions and inter- tion tool [Kuehlmann et al. 1995] is mediate lemmas. Although proofs can widely used within IBM in the design be constructed by hand, here we focus of many processors such as the Pow- only on machine-assisted theorem prov- erPC and System/390. Applied in a ing. Theorem provers are increasingly hierarchical manner, the tool can being used today in the mechanical ver- handle entire processor designs con- ification of safety-critical properties of taining millions of transistors [Appen- hardware and software designs. zeller and Kuehlmann 1995]. Using Theorem provers can be roughly clas- this tool, the functional behavior of a

ACM Computing Surveys, Vol. 28, No. 4, December 1996 634 • E. M. Clarke et al.

hardware system at the register- collaboration with Motorola design- transfer level, gate level, or transistor ers, developed an ACL2 specification level is modeled as a Boolean state of the entire Motorola Complex Arith- transition . Algorithms based metic Processor (CAP), a microproces- on BDDs are used to check the equiv- sor for digital signal processing alence of the state transition func- (DSP). The CAP is the most compli- tions for different design levels. cated microprocessor yet formalized, with a three-stage pipeline, six inde- Motorola 68020. In 1991 Boyer and Yu pendent memories, four multiplier-ac- constructed an Nqthm [Boyer and cumulators, over 250 programmer- Moore 1979, 1988] specification of the visible registers, and an instruction Motorola 68020 microprocessor (in- set allowing the simultaneous modifi- cluding 80% of the user-mode instruc- cation of well over 100 registers in a tions) [Boyer and Yu 1996]. They used single instruction. The formal specifi- the specification to prove the correct- cation tracked the evolving design ness of many binary machine code and included a simpler nonpipelined programs produced by commercial view that was proved equivalent on a from source code in such certain of programs. Finally, high-level languages as Ada, Lisp, Brock used ACL2 to verify the binary and C. For example, Yu verified the microcode for several DSP algorithms MC68020 binary code produced by the [Brock et al. 1996]. “gcc” for 21 of the 22 C pro- grams in the Berkeley string library. AAMP5. During 1993–1995, Srivas of the Stanford Research Institute and AMD5K86. In 1995 Moore and Kauf- Miller of Rockwell International col- mann of Computational Logic, Inc., laborated on the specification and ver- and Lynch of Advanced Micro De- ification of the Collins Commercial vices, Inc., collaborated to prove the Avionics AAMP5 microprocessor. correctness of Lynch’s microcode for They used PVS to specify 108 of the floating-point division on the 209 AAMP5 instructions and verified AMD5K86. Starting from an informal the microcode for 11 representative proof of correctness, they formalized instructions [Miller and Srivas 1995]. their in the ACL2 logic [Kaufmann and Moore 1995] and As with model checking, an increase checked it with the ACL2 mechanical in the number and kinds of theorem theorem prover. Gaps and mistakes provers provides evidence for a growing were found in the informal “proof” but interest in theorem proving. There has in the end the microcode was mechan- been a corresponding increase in the ically shown to be correct [Moore et number and kinds of examples to which al. 1996]. The entire effort took about theorem provers have been applied. Fol- nine weeks. The mechanical proof lowing are some well-known theorem ended doubt about the code’s correct- provers, categorized roughly by their ness and allowed testers to focus on degree of automation: other routines. In 1996 Russinoff used ACL2 to check the correctness of the User-guided automatic deduction tools. floating-point square root microcode Systems like ACL2 [Kaufmann and [Russinoff 1996]. He found bugs in Moore 1995], Eves [Craigen et al. the microcode itself; after they were 1988], LP [Garland and Guttag 1988], fixed, the final version of the square Nqthm [Boyer and Moore 1979], Reve root microcode was also mechanically [Lescanne 1983], and RRL [Kapur proved correct. and Musser 1987] are guided by a sequence of lemmas and definitions Motorola CAP. During 1992–1996, but each theorem is proved automati- Brock of Computational Logic, Inc., in cally using built-in heuristics for in-

ACM Computing Surveys, Vol. 28, No. 4, December 1996 Formal Methods • 635

duction, lemma-driven , and 3.1 Fundamental Concepts simplification. Nqthm, the Boyer- Moore theorem prover, has been used Significant advances in the practical to check a proof of Go¨del’s first incom- use of formal methods have relied on pleteness theorem, and in a variety of fundamental results drawn from all ar- large-scale verification efforts. eas in , not necessarily directly intended for formal methods. Proof checkers. Examples include Coq Further work needs to be done in the [Cornes et al. 1995], HOL [Gordon following areas. 1987], LEGO [Luo and Pollack 1992], LCF [Gordon et al. 1979], and Nuprl —Composition. We need to understand [Constable et al. 1986]. They have how to compose methods, specifica- been used to formalize and verify tions, models, , and proofs. hard problems in mathematics and in —Decomposition. We need to develop program verification. more efficient methods for decompos- Combination provers. Analytica [Clarke ing a computationally demanding and Zhao 1993], which combines theo- global property into local properties rem proving with the symbolic alge- whose verification is computationally bra system Mathematica, has success- simple (e.g., task decomposition and fully proved some hard number- localization reduction methods [Kurs- theoretic problems due to Ramanujam. han 1994b]). Both PVS [Owre et al. 1992] and STeP —Abstraction. Real systems are difficult [Bjørner et al. 1996] combine powerful to specify and verify without abstrac- decision procedures and model check- tions. We need to identify different ing with interactive proof. PVS has kinds of abstractions, perhaps tai- been used to verify a number of hard- lored for certain kinds of systems or ware designs and reactive, real-time, problem domains, and we need to de- and fault-tolerant algorithms. velop ways to justify them formally, perhaps using mechanical help. 3. FUTURE DIRECTIONS —Reusable models and theories. Rather than defining models and theories The overarching goal of formal methods from scratch each time a new applica- is to help engineers construct more reli- tion is tackled, it would be better to able systems. Formal methods is thus have reusable and parameterized an area that cuts across almost all other models and theories. areas in computer science. Its founda- tions lie squarely in mathematics, its —Combinations of mathematical theo- intended applications are hardware and ries. Many safety-critical systems software systems, and its potential us- have both digital and analog compo- ers are all developers involved in the nents. These hybrid systems require system engineering process. reasoning about both discrete and Tremendous advances in the past de- continuous mathematics. cade have been made on all fronts. As System developers would like to be technology improves, it becomes more able to predict how well their system feasible to attack harder and larger will operate in the field. Indeed, they problems. Progress in the area depends often care more about performance on doing fundamental research, invent- than correctness. Performance model- ing new methods and building new ing borrows strongly from , tools, integrating different methods to , and queueing . work together, and making concerted —Data structures and algorithms.To efforts by researchers to work with handle larger search spaces and practitioners to transfer technology ef- larger systems, new data structures fectively. and algorithms, for example, more

ACM Computing Surveys, Vol. 28, No. 4, December 1996 636 • E. M. Clarke et al.

concise data structures for represent- likely to be more patient, however, ing Boolean functions, are needed. with completely automatic tools that perform more extensive analysis. 3.2 Methods and Tools —Ease of learning. Notations and tools should provide a starting point for No one method or tool can serve all writing formal specifications for de- purposes. We need to support all differ- velopers who would not otherwise ent kinds. From past experience, we write them. The knowledge of formal have learned what kinds can have the specifications needed to start realiz- most impact. To be attractive to practi- ing benefits should be minimal. tioners, methods and tools should sat- —Orientation toward error detection. isfy the following criteria. We realize Methods and tools should be opti- that some of these criteria are ideals, mized for finding errors, not for certi- but it is still good to strive for them. fying correctness. They should sup- port generating counterexamples as a —Early payback. Methods and tools means of debugging. should provide significant benefits al- —Focused analysis. Methods and tools most as soon as people begin to use should be good at analyzing at least them. one aspect of a system well, for exam- —Incremental gain for incremental ef- ple, the control flow of a protocol. fort. Benefits should increase as de- They need not be good at analyzing velopers get more adept or put more all aspects of a system. effort into writing specifications or —Evolutionary development. Methods using tools. and tools should support evolutionary —Multiple use. It should be possible to system development by allowing par- amortize the cost of a method or tool tial specification and analysis of se- over many uses. For example, it lected aspects of a system. should be possible to derive benefits from a single specification at several More ambitiously, rather than build a points in a program’s life cycle: in single tool, we can build “meta-tools” design analysis, code optimization, which themselves produce tools custom- test case generation, and regression ized for a particular problem domain testing. [Steffen et al. 1996], formal notation —Integrated use. Methods and tools [Cleaveland et al. 1995], or logic [Gor- should work in conjunction with each don 1987; Kindred and Wing 1996]. other and with common programming These metatools, like compiler genera- languages and techniques. Developers tors, provide an automatic way to build should not have to “buy into” a new specialized model checkers or proof methodology completely to begin re- checkers. ceiving benefits. The use of tools for Finally, for any new method or tool, formal methods should be integrated its developer should state explicitly with that of tools for traditional soft- what its strengths, limitations, model- ware development, for example, com- ing assumptions, ease of integration pilers and simulators. with other methods and tools, and start-up costs are. Clear selection crite- —Ease of use. Tools should be as easy to ria help potential users decide what use as compilers, and their output method or tool is most appropriate for should be as easy to understand. the problem at hand. —Efficiency. Tools should make effi- cient use of a developer’s time. Turn- 3.3 Integration of Methods around time with an interactive tool should be comparable to that of nor- Given that no one formal method is mal compilation. Developers are likely to be suitable for describing and

ACM Computing Surveys, Vol. 28, No. 4, December 1996 Formal Methods • 637 analyzing every aspect of a complex sys- done in tools such as PVS and STeP. tem, a practical approach is to use dif- For example, a sufficiently expressive ferent methods in combination. When logic can be used to define temporal combining methods it is important to operators over finite state transition consider both systems in terms of maximal or minimal fixed points. For finite state transition —finding a suitable style for using dif- systems, these fixed points can be eval- ferent methods together; and uated using a model checker as a deci- —finding a suitable for using sion procedure. For structures with un- different methods together. bounded state spaces, the temporal Very often neither is addressed ade- properties can be verified by means of quately. Failure to find a suitable style fixed point induction. misses out on the true advantages of Another way of combining deductive combining methods. For example, the Z and model checking approaches is to use school stresses the importance of pre- deduction to obtain a finite state ab- sentation of specifications in an accessi- straction of an implementation that can ble form, with plenty of natural lan- be verified using model checking. Such guage. This emphasis has helped in abstractions are commonly used in pre- popularizing its notation. Any combina- paring a problem for model checking tion must preserve this style of presen- but are seldom rigorously verified. De- tation. duction can also be used to verify as- Failure to attend to the theoretical sumption-commitment proof obligations foundations of the combination misses generated by composing component im- out on the true advantages of formality. plementations that have been sepa- In chemistry, a distinction is drawn be- rately verified by means of model check- tween a mixture and a compound.Ina ing. Induction can be combined with mixture, the ingredients merely mingle model checking to verify systems com- together; in a compound, the ingredi- posed of networks of finite state pro- ents become chemically united. So it is cesses. with combining different formal meth- 3.3.2 Integration with the System De- ods. If the meaning of the combination velopment Process. Formal methods can is not properly explained, then the re- complement less formal methods that sult is merely a mixture: nothing more are used in the overall system develop- can be deduced from the joint descrip- ment process. They could be used not tion than from the separate ones. If the instead of, but in addition to, informal meaning of the combination is ex- methods, as was done by Praxis in the plained, then the result is much more CDIS example. So far formal methods powerful. It then becomes possible to have shown their strength in their use have two views of a system specifica- in specification and verification. It is tion, and to reason with and refine one worth exploring how they can be used in view, and to understand the conse- , refinement, and quences in the other view. testing. 3.3.1 Model Checking and Theorem Proving. One of the most promising di- —Requirements analysis necessarily rections in method integration is in deals with customers who often have combining model checking and theorem an imprecise idea of what they want; proving [Kurshan and Lamport 1993; formal methods can help customers Rajan et al. 1995; Bjørner et al. 1996], nail down their system requirements ideally to benefit from the advantages of more precisely. both approaches. One way is to employ —Refinement is the reverse of verifica- model checking as a decision procedure tion; it is the process of taking one within a deductive framework, as is level of specification (or implementa-

ACM Computing Surveys, Vol. 28, No. 4, December 1996 638 • E. M. Clarke et al.

tion) and through a series of “correct- gle stand-alone programs from ness-preserving transformations” syn- scratch, but also how to construct thesizing a lower-level specification large systems, perhaps using off-the- (or implementation). Although much shelf components, and how to main- theoretical work on refinement has tain legacy code; they need to know been done, the results have not yet not just how to code, but also how to transferred to practice. do high-level system design. —Testing is one of the most costly areas in all software projects. Formal meth- 4. CONCLUDING REMARKS ods can play a role in the validation process, for example, using formal Commercial pressure to produce higher- specifications to generate test suites quality software is always increasing. [Richardson et al. 1989], and using Formal methods have already demon- model and proof-checking tools to de- strated success in specifying commercial termine formal relationships between and safety-critical software, and in ver- specifications and test suites and be- ifying protocol standards and hardware tween test suites and code. designs. In the future, we expect that the role of formal methods in the entire 3.4 Education and Technology Transfer system-development process will in- crease, especially as the tools and meth- Education is vital to the success of the ods successful in one domain carry over formal methods. There are different to others. Progress, however, will de- kinds of audiences: pend strongly on continued support for basic research on new specification lan- —Our research peers. Some of our great- guages and new verification techniques. est skeptics are our own colleagues. Ideally, system developers would all We can overcome this skepticism by be trained sufficiently well that they collaborating with them and their would not even think that they are us- students on systems that they care ing a formal method or tool. They would about. routinely use the mathematics underly- —Practitioners. Technology transfer ing the notation of a formal specifica- should be taken very seriously from tion language simply as a means of com- the very beginning. The recent spread municating ideas to others on their of formal methods is directly related team or of documenting their own de- to efforts made by researchers in sign decisions. They would routinely use teaching their techniques to industry. tools such as model and proof checkers For effective technology transfer, with as much ease as they use compil- however, we must keep in mind that ers. Therefore, as researchers and edu- success for industry depends on cators in formal methods, we should timely delivery, continuously en- strive to make our notations and tools hanced functionality, understanding accessible to nonexperts. customers’ needs, reuse of legacy Towards this ideal, however, it makes code, commitment to quality, elimina- sense to cultivate a new career path for tion of errors, cost-effective develop- specialists in formal methods. They ment, and real-time performance. could be experts in the use of one —Students at all levels. Some graduate method or tool or they could be knowl- programs now incorporate formal edgeable in many, offering their advice methods in their curricula [Garlan et on which to use for a given application. al. 1995; Oxford University 1996]. Ed- Wing [1985] envisioned over ten years ucators are starting to consider teach- ago the idea of specification firms, anal- ing formal methods at the undergrad- ogous to architecture and law firms, uate level. Students need to whose employees would be hired for understand not just how to build sin- their skills in formal methods. This vi-

ACM Computing Surveys, Vol. 28, No. 4, December 1996 Formal Methods • 639 sion has been realized by the growth The t9000 virtual channel processor. IEEE both in the number of in-house teams Trans. Softw. Eng. 21, 2 (Feb.), 69–78. that consult on projects within large BEAR, S. 1991. An overview of HP-SL. In Pro- corporations (e.g., AT&T and ) and ceedings of VDM’91: Formal Development Methods, Volume 551 of Lecture Notes in in the number of independent compa- Computer Science. Springer-Verlag. nies (e.g., Computational Logic, Inc., BENGTSSON, J., GRIFFIOEN, W., KRISTOFFERSEN, K., Kestrel Institute, and ORA) that spe- LARSEN, K., LARSSON, F., PETTERSSON, P., AND cialize in the use of formal methods and YI, W. 1996. Verification of an audio proto- do contract work for industry and gov- col with bus collision using UppAal. In Com- ernment agencies. Some companies, puter-Aided Verification ’96, Lecture Notes in Computer Science 1102, R. Alur and T. Henz- such as Praxis, use formal methods as a inger, Eds., Springer-Verlag, 244–256. routine part of their development pro- BJØRNER,N.ET AL. 1996. STeP: Deductive-algo- cess. rithmic verification of reactive and real-time Finally, for further reading, see the systems. In Proceedings of the Eighth Interna- April 1996 issue of IEEE Computer, tional Conference on Computer-Aided Verifi- which contains a roundtable discussion cation, Number 1102 in Lecture Notes in on formal methods, and the June 1996 Computer Science (July), Springer-Verlag, 415–418. issue of IEEE Spectrum, which gives an BOSSCHER, D., POLAK, I., AND VAANDRAGER,F. overview of model checking. Online fo- 1994. Verification of an audio-control proto- rums include the net newsgroup comp. col. In FTRTFT 94: Formal Techniques in specification and its subnewsgroups for Real-Time and Fault-Tolerant Systems, Lec- specific methods, and the formal meth- ture Notes in Computer Science 863, H. Lang- ods mailing list, [email protected]. The maack, W.-P. de Roever, and J. Vytopil Eds., Oxford University’s web page http:// www. Springer-Verlag, 170–192. comlab.ox.ac.uk/archive/formalmethods. BOSWELL, A. 1995. Specification and validation of a security policy model. IEEE Trans. Softw. html points to a wealth of information Eng. 21, 2 (Feb.), 63–68. about formal methods, including papers, BOYER,R.AND YU, Y. 1996. Automated proofs of reports, tools, conferences, journals, object code for a widely used microprocessor. projects, and people. J. ACM 43, 1 (Jan.), 166–192. BOYER,R.S.AND MOORE, J. S. 1979. A Compu- tational Logic. Academic Press, New York. REFERENCES BOYER,R.S.AND MOORE, J. S. 1988. A Compu- tational Logic Handbook. Academic Press, ALUR, R., HENZINGER, T., AND HO, P.-H. 1996. New York. Automatic symbolic verification of embedded systems. IEEE Trans. Softw. Eng. 22, 3, 181– BRAYTON,R.ET AL. 1996. VIS: A system for 201. verification and synthesis. In Proceedings of the Eighth International Conference on Com- APPENZELLER,D.P.AND KUEHLMANN, A. 1995. puter-Aided Verification, Number 1102 in Formal verification of a PowerPC micropro- Lecture Notes in Computer Science, Springer- cessor. In Proceedings of the IEEE Interna- Verlag, 423–427. tional Conference on Computer Design (ICCD’95) (Austin, TX, Oct.), 79–84. BROCK, B., KAUFMANN, M., AND MOORE,J.S. 1996. Heavy inference: about com- ARCHINOFF,G.ET AL. 1990. Verification of the shutdown system software at the Darlington mercial microprocessors. In Formal Methods Nuclear Generating System. In International in Computer-Aided Design (FMCAD’96) Conference on Control and Instrumentation in (Nov.), M. Srivas and A. Camilleri Eds., Nuclear Installations (Glasgow, Scotland, Springer-Verlag. May). BROWNE,M.C.,CLARKE,E.M.,DILL,D.L.,AND ARNOLD, A., BEGAY, D., AND RADOUX, J.-P. 1996. MISHRA, B. 1986. Automatic verification of The embedded software of an electricity sequential circuits using temporal logic. IEEE meter: An experience in using Formal Meth- Trans. Comput. C-35, 12, 1035–1044. ods in an industrial project. Sci. Comput. Pro- BRYANT, R. E. 1986. Graph-based algorithms gram. for manipulation. IEEE BARRETT, G. 1989. Formal methods applied to a Trans. Comput. C-35,8. floating-point number system. IEEE Trans. BURCH,J.R.,CLARKE,E.M.,LONG,D.E.,MCMIL- Softw. Eng. 15, 5 (May), 611–621. LAN,K.L.,AND DILL, D. L. 1994. Symbolic BARRETT, G. 1995. Model checking in practice: model checking for sequential circuit verifica-

ACM Computing Surveys, Vol. 28, No. 4, December 1996 640 • E. M. Clarke et al.

tion. IEEE Trans. Comput. Aided Des. Integr. CLARKE,E.M.,GRUMBERG, O., AND LONG,D.E. Circuits Syst. 13, 4 (April), 401–424. 1992. Model checking and abstraction. In CALERO, J., ROMAN, C., AND PALMA, G. D. 1997. Proceedings of Principles of Programming A practical design case using formal verifica- Languages. tion. In Proceedings of Design-SuperCon’97. CLEAVELAND, R., MADELAINE, E., AND SIMS,S. To appear. 1995. Generating front ends for verification CARNOT, M., DASILVA, C., DEHBONEI, B., AND tools. In Tools and Algorithms for the Con- MEIJA, F. 1992. Error-free software devel- struction and Analysis of Systems (TACAS opment for critical systems using the B-meth- ’95), Vol. 1019 of Lecture Notes in Computer odology. In Third International IEEE Sympo- Science, E. Brinksma, R. Cleaveland, K. sium on Software . Larsen, and B. Steffen Eds., Springer-Verlag, 153–173. CHANDY,K.AND MISRA, J. 1988. Parallel Pro- gram Design. Addison-Wesley, Reading, MA. CLEAVELAND, R., PARROW, J., AND STEFFEN,B. 1993. The Concurrency Workbench: A se- CHAVES, J. 1992. Formal methods at AT&T: An mantics-based tool for the verification of con- industrial usage report. In Proceedings For- current systems. ACM Trans. Program Lang. mal Description Techniques IV - 1991, North- Syst. 15, 1 (Jan.), 36–72. Holland, Amsterdam, 83–90. CONSTABLE,R.ET AL. 1986. Implementing CHEHAIBAR, G., GARAVEL, H., MOUNIER, L., TAWBI, Mathematics with the NuPRL Proof Develop- N., AND ZULIAN, F. 1996. Specification and ment Environment. Prentice-Hall, Englewood verification of the powerscale bus arbitration Cliffs, NJ. protocol: An industrial experiment with LO- CORNES, C., COURANT, J., FILLIATREˆ , J.-C., HUET, TOS. In Proceedings of FORTE/PSTV’96 G., MANOURY, P., PAULIN-MOHRING, C., MUNOZ, (Kaiserslautern, Germany). Chapman & Hall, C., MURTHY, C., PARENT, C., SA¨ıBI, A., AND London. WERNER, B. 1995. The coq proof assistant CHISOLM, G., KLJAICH, J., SMITH, B., AND WOJCIK, reference manual version 5.10. Tech. Rep. 177 A. 1987. An approach to the verification of (July), INRIA. http://pauillac.inria.fr/coq/syste- a fault-tolerant, computer-based reactor me_coq-eng.html. safety system: A case study using automated CRAIGEN, D., GERHART, S., AND RALSTON,T. reasoning (Vol. 1, interim report). Tech. Rep. 1993a. An international survey of industrial NP-4924 (Jan.), Electric Power Research In- applications of formal methods. Tech. Rep. stitute, Palo Alto, CA. Prepared by Argonne NIST GCR 93/626 (Vols. 1 and 2) (March), National Laboratory. U.S. National Institute of Standards and CLARKE, E., GERMAN, S., AND ZHAO, X. 1996. Technology. Also published by the U.S. Naval Verifying the SRT division algorithm using Research Laboratory (Formal Rep. 5546-93- theorem proving techniques. In Proceedings of 9582, Sept.), and the Atomic Energy Control the Eighth International Conference on Com- Board of Canada. puter-Aided Verification, Number 1102 in CRAIGEN, D., GERHART, S., AND RALSTON,T. Lecture Notes in Computer Science, Springer- 1993b. Observations on industrial practice Verlag, 111–122. using formal methods. In Proceedings of the CLARKE,E.AND KURSHAN, R. 1996. Computer- Fifteenth International Conference on Soft- aided verification. IEEE Spectrum 33,6,61– ware Engineering (May). 67. CRAIGEN, D., GERHART, S., AND RALSTON, T. 1994. CLARKE,E.AND ZHAO, X. 1993. Analytica: A Formal methods in critical systems. IEEE theorem prover for Mathematica. Math- Softw. 11, 1 (Jan.). ematica J., 56–71. CRAIGEN, D., GERHART, S., AND RALSTON, T. 1995. CLARKE,E.M.AND EMERSON, E. A. 1981. Syn- Formal methods reality check: Industrial us- thesis of synchronization skeletons for age. IEEE Trans. Softw. Eng. 21, 2 (Feb.), branching time temporal logic. In Logic of 90–98. Programs: Workshop, (Yorktown Heights, CRAIGEN, D., KROMODIMOELJO, S., MEISELS, I., NY), Vol. 131 of Lecture Notes in Computer NEILSON, A., PASE, B., AND SAALTINK,M. Science, Springer-Verlag. 1988. m-EVES: A tool for verifying software. CLARKE,E.M.,EMERSON,E.A.,AND SISTLA,A. In Proceedings of the Tenth International Con- P. 1986. Automatic verification of finite- ference on Software Engineering (Singapore, state concurrent systems using temporal logic April), 324–333. specifications. ACM Trans. Program Lang. CROXFORD,M.AND SUTTON, J. 1995. Breaking Syst. 8, 2, 244–263. through the V and V bottleneck. In Proceed- CLARKE,E.M.,GRUMBERG, O., HIRAISHI, H., JHA, ings of Ada in Europe 1995. Springer-Verlag. S., LONG,D.E.,MCMILLAN,K.L.,AND NESS,L. DAMM,W.AND DELGADO-KLOOS, C. 1996. A. 1993. Verification of the Futurebusϩ Practical Formal Methods for Hardware De- cache coherence protocol. In Proceedings sign. Lecture Notes in Computer Science. CHDL. Springer-Verlag.

ACM Computing Surveys, Vol. 28, No. 4, December 1996 Formal Methods • 641

DAMM, W., JOSKO, B., AND SCHLOR¨ , R. 1995. teenth Symposium on Principles of Program- Specification and Validation Methods for Pro- ming Languages, 219–228. gramming Languages and Systems, Chap. GERTH, R., PELED, D., VARDI,M.Y.,AND WOLPER, Specification and verification of VHDL-based P. 1995. Simple on-the-fly automatic verifi- system-level hardware designs, Oxford Uni- cation of linear temporal logic. In Proceedings versity Press, New York, 331–410. IFIP/WG6.1 Symposium on Protocol Specifi- DAWS,C.AND YOVINE, S. 1995. Two examples of cation, Testing, and Verification (Warsaw, Po- verification of multirate timed automata with land, June). KRONOS. In Proceedings of 1995 IEEE Real- GORDON, M. 1987. HOL: A proof generating Time Systems Symposium, RTSS’95 (Pisa, It- system for higher-order logic. In VLSI Specifi- aly, Dec.). IEEE Computer Society Press, Los cation, Verification and Synthesis. Kluwer. Alamitos, CA. GORDON,M.J.,MILNER,A.J.,AND WADSWORTH,C. DEHARBE´ ,D.AND BORRIONE, D. 1995. P. 1979. Edinburgh LCF, Vol. 78 of Lec- of a verification-oriented subset of VHDL. In ture Notes in Computer Science. Springer- CHARME’95, Correct Hardware Design and Verlag. Verification Methods, P. Camurati and H. GUIHO,G.AND HENNEBERT, C. 1990. SACEM Eveking, Eds., Vol. 987 of Lecture Notes in software validation. In Twelfth International Computer Science Springer-Verlag, 293–310. Conference on Software Engineering. DELISLE,N.AND GARLAN, D. 1990. A formal GUTTAG,J.AND HORNING, J. 1993. Larch: Lan- specification of an oscilloscope. IEEE Softw. 7, guages and Tools for Formal Specification. 5 (Sept.), 29–36. Springer-Verlag. Written with S. J. Garland, DEPALMA,G.AND GLASER, A. 1996. Formal ver- K. D. Jones, A. Modet, and J. M. Wing. ification augments simulation. Electr. Eng. HALL, A. 1996. Using formal methods to de- Times, 56. velop an ATC information system. IEEE DILL,D.L.,DREXLER,A.J.,HU,A.J.,AND YANG,C. Softw. 12, 6 (March), 66–76. H. 1992. Protocol verification as a hard- HAREL, D. 1987. Statecharts: A visual formal- ware design aid. In IEEE International Con- ism for complex systems. Sci. Comput. Pro- ference on Computer Design: VLSI in Comput- gram. 8, 231–274. Preliminary version: Tech. ers and Processors, 522–525. Rep. CS84-05, The Weizmann Institute of Sci- DINOLT,G.ET AL. 1984. Multinet gateway—to- ence, Rehovot, Israel, Feb. 1984. wards A1 certification. In IEEE Symposium HAREL, D. 1992. Biting the silver bullet: To- on Security and Privacy (1984). ward a brighter future for system develop- ELSEAIDY, W., CLEAVELAND, R., AND BAUGH,J. ment. IEEE Comput. 25, 1 (Jan.), 8–20. 1996. Modeling and verifying active struc- HAR’EL,Z.AND KURSHAN, R. P. 1990. Software tural control systems. Sci. Comput. Program. for analytical development of communications (to appear). A preliminary version of this pa- protocols. AT&T Bell Lab. Tech. J. 69, 1 (Jan.– per appears in the Proceedings of the 1994 Feb.), 45–59. Real-Time Systems Symposium. HEIMDAHL,M.AND LEVESON, N. 1996. Com- FERNANDEZ, J.-C., GARAVEL, H., KERBRAT, A., MA- pleteness and consistency in hierarchical state- TEESCU, R., MOUNIER, L., AND SIGHIREANU, based requirements. IEEE Trans. Softw. Eng. M. 1996. CADP (CÆSAR/ALDEBARAN SE-22, 6 (June), 363–377. development package): A protocol validation and international verification toolbox. In Pro- HENINGER, K. 1980. Specifying software re- ceedings of the 8th Conference on Computer- quirements for complex systems: New tech- Aided Verification, Number 1102 in Lecture niques and their application. IEEE Trans. Notes in Computer Science. R. Alur and T. A. Softw. Eng. 6, 1 (Jan.), 2–13. Henzinger, Eds., Springer-Verlag. HENZINGER,T.A.,NICOLLIN, X., SIFAKIS, J., AND FILKORN, T., SCHNEIDER, H., SCHOLZ, A., STRASSER, YOVINE, S. 1994. Symbolic model checking A., AND WARKENTIN, P. 1994. SVE User’s for real-time systems. Inf. Comput. 111, 111– Guide. Tech. Rep. ZFE BT SE 1-SVE-1, Sie- 244. mens AG, Corporate Research and Develop- HO, P.-H. AND WONG-TOI, H. 1995. Automated ment, Munich. analysis of an audio control protocol. In Com- GARLAN, D., ABOWD, G., JACKSON, D., TOMAYKO, J., puter-Aided Verification ’95, Lecture Notes in AND WING, J. 1995. The CMU Master of Computer Science 939, P. Wolper Ed., Spring- Software Engineering Core Curriculum. In er-Verlag, 381–394. Proceedings of the Eighth SEI Conference on HOARE, C. A. R. 1985. Communicating Sequen- Software Engineering Education (CSEE) Vol. tial Processes. Prentice-Hall International, 895 of Lecture Notes in Computer Science, Englewood Cliffs, NJ. Springer-Verlag, 65–86. HOJATI, R., BRAYTON, R., AND KURSHAN, R. 1993. GARLAND,S.J.AND GUTTAG, J. V. 1988. BDD-based debugging of designs using lan- Inductive methods for reasoning about ab- guage containment and fair CTL. In Proceed- stract data types. In Proceedings of the Fif- ings of the Fifth International Conference on

ACM Computing Surveys, Vol. 28, No. 4, December 1996 642 • E. M. Clarke et al.

Computer-Aided Verification, Number 697 in ceedings of the USENIX Workshop on Elec- Lecture Notes in Computer Science, C. Cour- tronic Commerce Protocols (1996). coubetis Ed., Springer-Verlag, 41–57. KING, T. 1994. Formalising British Rail’s sig- HOLZMANN, G. 1991. Design and Validation of nalling rules. In FME’94: Industrial Benefit of Computer Protocols. Prentice-Hall, Englewood Formal Methods, Volume 873 of Lecture Cliffs, New Jersey. Notes in Computer Science (1994), Springer- HOLZMANN, G. 1992. Practical methods for the Verlag, 45–54. formal validation of SDL specifications. Com- KLJAICH, J., SMITH, B., AND WOJCIK, A. 1989. put. Commun. Special issue on Practical Uses Formal verification of fault tolerance using of FDT’s. theorem-proving techniques. IEEE Trans. Comput. 38, 366–376. HOLZMANN, G. 1994. The theory and practice of a formal method: NewCoRe. In Proceedings of KUEHLMANN, A., SRINIVASAN, A., AND LAPOTIN,D.P. IFIP World Computer Congress (Hamburg, 1995. Verity—a formal verification program Germany, Aug.). for custom CMOS circuits. IBM J. Res. Dev. 39, 1/2, 149–165. HOLZMANN,G.AND PATTI, J. 1989. Validating SDL specifications: An experiment. In Pro- KUHN,D.AND DRAY, J. 1990. Formal specifica- ceedings of the Ninth International Conference tion and verification of control software for on Protocol Specification, Testing, and Verifi- cryptographic equipment. In Sixth Computer cation, INWG/IFIP (Twente, Netherlands, Security Applications Conference (1990). June) C. Vissers and E. Brinksma, Eds. KURSHAN,R.AND LAMPORT, L. 1993. Verifi- HOLZMANN,G.AND PELED, D. 1994. An improve- cation of a multiplier: 64 Bits and beyond. In ment in formal verification. In Proceedings of Computer Aided Verification, Volume 697 of FORTE94 (Berne, Switzerland, Oct.). Lecture Notes in Computer Science, C. Cour- coubetis, Ed., Springer-Verlag, 166–179. HOUSTON,I.AND KING, S. 1991. CICS project report: Experiences and results from using Z. KURSHAN, R. P. 1994a. Computer-Aided Verifi- In Proceedings of VDM’91: Formal Develop- cation of Coordinating Processes. Princeton ment Methods, Volume 551 of Lecture Notes University Press, Princeton, NJ. in Computer Science, Springer-Verlag. KURSHAN, R. P. 1994b. The complexity of verifi- cation. In Proceedings 26th ACM Symposium ISO. 1987. Information Systems Processing— on Theory of Computing (STOC) (Montreal), Open Systems Interconnection—LOTOS. 365–371. Tech. Rep. International Standards Organiza- tion DIS 8807. LAMPORT, L. 1984. The temporal logic of ac- tions. ACM Trans. Program. Lang. Syst., 872– JACKY, J. 1995. Specifying a safety-critical con- 923. trol system in Z. IEEE Trans. Softw. Eng. 21, 2 (Feb.), 99–106. LESCANNE, P. 1983. Computer experiments with the REVE term rewriting system gener- JAGADEESAN, L., PUCHOL, C., AND OLNHAUSEN,J. ator. In Proceedings of the Tenth Symposium V. 1996. A formal approach to reactive sys- on Principles of Programming Languages tems software: A telecommunications applica- (Austin, Texas, Jan.), 99–108. tion in Esterel. Formal Aspects Comput. 8,2 (March), 123–151. LONG, D. L. 1993. Model checking, abstraction, and compositional reasoning. Ph.D. Thesis, JANICKI, R., PARNAS,D.L.,AND ZUCKER, J. 1996. Carnegie Mellon Univ., Computer Science Tabular representations in relational docu- Dept. ments. In Relational Methods in Computer LOWE, G. 1996. Breaking and fixing the Need- Science. C. Brink, Ed., Springer-Verlag (to ham-Schroder public-key protocol using FDR. appear). In Tools and Algorithms for the Construction JONES, C. B. 1986. Systematic Software Devel- and Analysis of Systems, Vol. 1055 of Lecture opment Using VDM. Prentice-Hall Interna- Notes in Computer Science. Springer-Verlag. tional, New York. LUO,Z.AND POLLACK, R. 1992. LEGO proof de- KALTENBACH, M. 1994. Model checking for velopment system: User’s manual. Tech. Rep. UNITY. Tech. Rep. TR94-31 (Dec.), The Uni- ECS-LFCS-92-211 (May), Computer Science versity of Texas at Austin. Dept., Univ. of Edinburgh. KAPUR,D.AND MUSSER, D. 1987. Proof by con- LYNCH,N.AND TUTTLE, M. 1987. Hierarchical sistency. Artif. Intell. 31, 125–157. correctness proofs for distributed algorithms. KAUFMANN,M.AND MOORE, J. S. 1995. ACL2: A Tech. Rep. (April), MIT Laboratory for Com- Computational Logic for Applicative Common puter Science, Cambridge, MA. Lisp, The User’s Manual (Version 1.8). ftp:// MANNA,Z.AND PNUELI, A. 1991. The Temporal ftp.cli.com/pub/acl2/v1-8/acl2-sources/doc/ Logic of Reactive and Concurrent Systems, HTML/acl2-doc.html. Springer-Verlag, New York. KINDRED,D.AND WING, J. 1996. Fast, auto- MATAGA,P.AND ZAVE, P. 1995. Multiparadigm matic checking of security protocols. In Pro- specification of an AT&T switching system. In

ACM Computing Surveys, Vol. 28, No. 4, December 1996 Formal Methods • 643

Applications of Formal Methods,M.G. Classical Mind: Essays in Honour of C.A.R. Hinchey and J. P. Bowen, Eds., Prentice-Hall Hoare, A. Roscoe, Ed., Prentice-Hall, Engle- International, Englewood Cliffs, NJ, 375–398. wood Cliffs, NJ. MCMILLAN, K. L. 1993. Symbolic Model Check- ROY,V.AND DE SIMONE, R. 1990. Auto/Auto- ing: An Approach to the State Explosion Prob- graph. In Computer-Aided Verification ’90, lem. Kluwer. Vol. 3 of DIMACS Series on Discrete Mathe- MILLER,S.P.AND SRIVAS, M. 1995. Formal ver- matics and Theoretical Computer Science ification of the AAMP5 microprocessor: A case (Piscataway, NJ, June), E. Clarke and R. Kur- study in the industrial use of formal methods. shan, Eds., American Mathematical Society, In WIFT’95: Workshop on Industrial-Strength Providence, RI, 235–250. Formal Specification Techniques (Boca Raton, RUESS, H., SHANKAR, N., AND SRIVAS, M. 1996. FL), IEEE Computer Society, Washington, Modular verification of SRT division. In Pro- DC, 2–16. ceedings of the Eighth International Confer- MILNER, A. 1980. A Calculus of Communicating ence on Computer-Aided Verification, No. Systems, Vol. 92 of Lecture Notes in Com- 1102 in Lecture Notes in Computer Science puter Science. Springer-Verlag. (July), Springer-Verlag, 123–134. MOORE,J.S.,LYNCH, T., AND KAUFMANN, M. 1996. RUSSINOFF, D. 1996. A mechanically checked A mechanically checked proof of the correct- proof of the correctness of the AMD K5 float- ness of the AMD5K86 floating point division ing-point square root algorithm (submitted). algorithm. http://devil.ece.utexas.edu:80/lynch/ SPC. 1993. Consortium requirements engineer- divide/divide.html. ing guidebook. Tech. Rep. SPC-92060-CMC NIELSEN, M., HAVELUND, K., WAGNER, K., AND version 01.00.09, Software Productivity Con- GEORGE, C. 1989. The RAISE language, sortium, Herndon, VA. method and tools. Formal Aspects Comput. 1, SPIVEY, J. M. 1988. Introducing Z: a Specifica- 85–114. tion Language and its Formal Semantics. OWRE, S., RUSHBY, J., AND SHANKAR, N. 1992. Cambridge University Press, New York. PVS: A prototype verification system. In Elev- STEFFEN, B., MARGARIA, T., CLASSEN, A., AND enth International Conference on Automated BRAUN, V. 1996. The Meta ’95 environ- Deduction (CADE), Vol. 607 of Lecture Notes ment. In Proceedings of Computer-Aided Veri- in , D. Kapur Ed., fication ’96, Lecture Notes Computer Science, Springer-Verlag, 748–752. Springer-Verlag. OXFORD UNIVERSITY. 1996. http://www.comlab. STEFFEN, B., MARGARIA, T., CLASSEN, A., BRAUN, V., ox.ac.uk/igdp/. Master’s of Science in Soft- AND REITENSPIESS, M. 1996. An environ- ware Engineering. ment for the creation of intelligent network PELED, D. 1996. Combining partial order reduc- services. In Intelligent Networks: IN/AIN tions with on-the-fly model-checking. J. For- Technologies, Operations, Services, and Appli- mal Meth. Syst. Des. 8 (1), 39–64. Also ap- cations—A Comprehensive Report (Chicago), I. peared in the Proceedings of the Sixth E. Consortium Ed., 287–300. Invited contribu- International Conference on Computer Aided tion. Also invited to the Annual Review of Verification 1994 (Stanford, CA), Lecture Communications, IEC, 919–935. Notes in Computer Science 818, Springer-Ver- VARDI,M.Y.AND WOLPER, P. 1986. An automa- lag, 377–390. ta-theoretic approach to automatic program PNUELI, A. 1981. A temporal logic of concurrent verification. In Proceedings of Logic in Com- programs. Theor. Comput. Sci. 13, 45–60. puter Science. QUEILLE,J.AND SIFAKIS, J. 1982. Specification WING, J. 1985. Specification firms: A vision for and verification of concurrent systems in the future. In Proceedings of the Third Inter- CÆSAR. In Proceedings of Fifth ISP. national Workshop on Software Specification RAJAN, S., SHANKAR, N., AND SRIVAS, M. 1995. and Design (London, Aug.), 241–243. An integration of model-checking with auto- ZAVE, P. 1995. Secrets of call forwarding: A mated proof checking. In Computer-Aided specification case study. In Proceedings of the Verification, ’95, Volume 939 of Lecture Notes Eighth International IFIP Conference on For- in Computer Science P. Wolper, Ed., Spring- mal Description Techniques for Distributed er-Verlag, 84–97. Systems and Communications Protocols RICHARDSON, D., O’MALLEY, T., AND MOORE,C.T. (FORTE ’95) Chapman & Hall, London, 153– 1989. Approaches to specification-based 168. testing. In ACM SIGSOFT 89: Third Sympo- ZAVE,P.AND JACKSON, M. 1996. Where do oper- sium on , Analysis, and Veri- ations come from? A multiparadigm specifica- fication (Dec.). tion technique. IEEE Trans. Softw. Eng. 22,7 ROSCOE, A. 1994. Model-checking CSP. In A (July), 508–528.

ACM Computing Surveys, Vol. 28, No. 4, December 1996