Conception formelle Frédéric Herbreteau Slides by Igor Walukiewicz, thanks! Factorial(x) := if x=0 then 1 else x*Factorial(x-1) Is this program correct? Some examples where it matters Therac-25 Radiation Overdosing (1985-87) • Radiation machine for treatement of cancer patients • At least 6 cases of overdoses in period 1985-1987 • Three death cases • Source: Design error in the control software (race condition) AT&T Telephone Network Outage (1990) • 9 hours otage of large parts of US telephone network • Cost: several 100 million US$ • Source: software flaw (wrong interpretation of break statement in C) Ariane 5 Crash (1996) • Crash of Ariane 5 missle in June 1996 • Cost: more than 500 milion US$ • Source: software flaw • A data conversion from 64-bit floating to 16-bit signed integer Pentium FDIV Bug (1994) • FDIV= floating point division unit • 1 in 9 bilion floating point dividers would produce inacurate results • Cost: 500 million in replaced processors • Source flaw in a division table BOEING 737-MAX (2019) • 356 morts (2 crashes) • coût en dizaines de milliards de dollars • problème de design du “flight control software (MCAS)” (et autre) Why it is difficult to verify computer systems? •Analog systems are continuous •Digital systems are not •Big number of components interacting together Some analysis “The role of software in recent Aerospace Accidents” Nancy G. Leveson Aeronautic and Astronautic Department MIT Accidents analysed •Explosion of Ariane 5 •Loss of Mars Climate Orbiter •Destruction of Mars Polar Lander •Placing Milstar satellite in an incorrect orbit •American Airlines B-757 crash into a mountain near Cali •Collision of Lufthansa A320 into earth bank in Warsaw •Crash of China Airlines A320 near Nagoya •Overconfidence on digital •Flawed review process automation •Revision for undesired •Not understanding the risks behaviours is very productive associated to software •Inadequate safety engineering •Almost all errors were due to flaws in specification and not in •50%-70% safety decisions are coding made in early stages of development •Reliability techniques (like redundancy) not effective for •Software reuse without safety software analysis •Assuming the risk decreases •Unnecessary complexity of over time software •Inadequate specifications Formal verification at work The transputer project • T9000 transputer (still widely used: GPS) • Formal development of floating- point unit • Formalisation revealed problems with the standard • Diagnostic information should be propagated, but sometimes it was impossible. • Verification with Hoare logic revealed errors (on very small class of vectors) • Formal development turned out to be faster • Queen’s Award for Technology Achievement in 1990 Signalling system for RER • Increase traffic by 25% preserving safety levels • 21K of Modula-2 code have been formally specified and verified using B method • Later the same method has been used for line 14 and Roissy Shuttle • No unit test where preformed, just some global tests B method AAMP Microprocessor • AAMP5 widely used processor (Rockwell Collins) • .5 M transistors • Completely verified in PVS (300 hours per instruction) • Later verified AAMP-FV showing dramatic reduction in verification costs • National Security Agency certification for use in cryptographic applications. Airbus • Development of Level A controllers for A340/600 series (Esterel technologies) • 70% of code generated automatically • Quick management of requirements changes • Major productivity improvement (each new project requires twice as much software as its predecessor) • SCADE has been adopted for A380 for most of on-board computers. What are formal methods Specification + Analysis of the system Customer needs Requirements (safety,…) Specification Source code Machine code Specification Giving precise statement of what the system must do while avoiding constraints on how it is achieved Ex: Everybody should be happy Many observers have noticed that the process of developing a formal specification is often as effective for finding errors as the verification effort in which the specification is to be used. Analysing system properties Formal methods allow to reason about systems Formal specification is a mathematical object. One can analyse it, manipulate it, reason about it A model of the system that suppresses implementation detail allows the architect to concentrate on the analyses and decisions that are most crucial. Problem: formal specifications are typically large (thousands of lines) Verification is impossible (algorithmically) Halting problem for Turing machines Alain Turing (1912-1954) Mathematician, Logician, crypto-specialist Computational model: Turing machine Program termination is not decidable: There is no algorithm to decide if a TM stops on the empty input Methods of system verification Peer reviewing • Manual code inspection. • On average 60% of errors caught. • subtle errors (concurrency, algorithm defects) hard to catch. • Used in 80% of all software engineering projects. • Refinement of this method: parallel developpement Testing 30%-50% of development cost. Programmers have to provide insights what to test, and what should be system response. When to stop testing? Formal specifications help here: One of the most cost-effective uses of specifications New tools provide as good coverage as manual test cases. They avoid programming test cases Get them as soon as you can Theorem proving Doing large proofs semi-automatically Model Checking Model checking flow-graph Requiremen The ACM Turing Award in 2007 for model-checking Some Turing Award Winners •Edsger Dijkstra (1972) •Donald Knuth (1974) •Michael Rabin and Dana Scott (1976) •Tony Hoare (1980) •Thompson & Ritchie (1983) •Hopcroft & Trajan (1986) •Rivest, Shamir, Adleman (2002) The ACM Turing Award in 2007 Edmund M. Clarke Jr. (CMU USA) Allen E. Emerson (U. of Texas at Austin, USA) Joseph Sifakis (IMAG, Grenoble) Jury justification: For their roles in developing Model-Checking into a highly effective verification technology, widely adopted in the hardware and software industries Advantages Makes formal techniques available to broad audience Automatic procedure taking on an input: • a finite state model • and a set of required properties A transition system Properties The program terminates The program does not terminate The computed value is correct A sequence of events is correct Mutual exclusion No deadlock No starvation No starvation Model Checker Yes No Insufficient memory Counterexample Advantages •Widely applicable •Allows for partial verification •Automatic •Produces counterexamples •Sound and interesting mathematical foundations Disadvantages •Analyses an abstraction of the system •Focused on control an not data oriented applications •No guarantee about the completeness of the results •Impossible to generalise Milestones towards MC Mathematical program correctness: Turing, Church (1949) Syntax based technique for sequential programs Floyd Hoare (69) Dikstra, Hoare: invariant method and program refinement (70ties) Program logics •Algorithmic and dynamic logics •Hoare logics •Linear temporal logic Model checking using temporal logics Clarke Emersion 1981, Sifakis at the same time First appearance of MEC (Alaiwan, Arnold 1983) Who uses formal methods? • Microsoft Azure (cloud computing) • Amazon S3 (cloud computing) • Intel (processors) • Airbus (aircrafts & space) • Dassault (aircrafts, systems 3DS) • ConsenSys (blockchain, smart contracts) • Oracle (“Parfait” static analysis) • Framatome (nuclear reactors) • Bosch (automotive) • … Any verification using model-based techniques is only as good as the model of the system Conclusions Formal methods are not silver bullet to eliminate errors They are not beyond budget of system developers They are the only practical means to demonstrate the absence of undesired behaviour The process of developing a specification is often the most valuable phase Leading hardware developers continue to apply model chekcing and proof technology In spite of their successes, verification technology and formal methods have not seen widespread adoption as a routine part of systems development practice, except, arguably, in the development of critical systems in certain domains. Indeed, we expect diffusion of rigorous verification and design technology to take place gradually, and not result in their explicit adoption as a distinct technology Hoare's vision: computer software as a most reliable component in any system it controls, and no one can blame the software anymore Plan of the course 1. Test vs. Model-checking vs. Proof 2. Formal modelling 3. Linear Temporal Logic (LTL) 4. LTL model-checking algorithms 5. Verification of algorithms with TLA+ 6. Fairness 7. Incremental modelling 8. Verification of infinite state algorithms.