Edmund M. Clarke Thomas A. Henzinger Helmut Veith Roderick Bloem Editors Handbook of Model Checking Handbook of Model Checking Edmund M

Home , Dana Scott

Edmund M. Clarke Thomas A. Henzinger Helmut Veith Roderick Bloem Editors Handbook of Model Checking Handbook of Model Checking Edmund M. Clarke r Thomas A. Henzinger r Helmut Veith r Roderick Bloem Editors

Handbook of Model Checking Editors Edmund M. Clarke Helmut Veith Carnegie Mellon University Arbeitsbereich Formal Methods in Systems Pittsburgh, Pennsylvania, USA Engineering Technische Universität Wien Wien, Austria

Roderick Bloem Institut für angewandte Thomas A. Henzinger Informationsverarbeitung und Institute of Science and Technology Austria Kommunikationstechnologie (IAIK) (IST Austria) Technische Universität Graz Klosterneuburg, Austria Graz, Austria

ISBN 978-3-319-10574-1 ISBN 978-3-319-10575-8 (eBook) DOI 10.1007/978-3-319-10575-8

Library of Congress Control Number: 2018943403

© Springer International Publishing AG, part of Springer Nature 2018 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made.

The image on the book cover was designed by Anna Petukhova

Printed on acid-free paper

This Springer imprint is published by the registered company Springer International Publishing AG part of Springer Nature. The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland To Helmut Foreword

With 32 technical articles and 76 authors, this handbook represents a full post- graduate course in Model Checking. If a reader can verify that he or she has read and studied every article, then Springer should certainly award that reader a Mas- ter’s Degree in Model Checking! Departments in Computer Science everywhere will certainly welcome access to this major resource. Model Checking has become a major area of research and development both for hardware and software verification owing to many factors. First, the improved speed and capacity of computers in recent times have made all kinds of problem solving both practical and efficient. Moreover, in the area of Model Checking the methods of design of models have contributed to the best formulation of problems. Then we have seen SAT solvers gain unexpected and truly remarkable efficiency improvements—despite theoretical limitations. Additionally, the methodology of Satisfiability Modulo Theories (SMT) has contributed to finding excellent ways to pose and solve problems. Uses of temporal logic and data-flow-analysis techniques have also made model checking more naturally efficient. All these contributions have helped solve the ever-present “state explosion problem.” The urgency to make greater strides has increased because new applications in such diverse areas as health care, transportation, security, and robotics require work in the field to achieve greater scale, expressivity, and automation. I would definitely recommend new Ph.D. candidates look seriously into going into research in this field, because success in Model Checking can directly lead to future success in many other activities in Computer Science. Finally, the recent tragic loss of Helmut Veith has been a dreadful blow to his family, friends, colleagues, and students. Let’s take up the flag in his honor to help promote and expand the field in which he was poised to become a recognized world leader. Carnegie Mellon University Dana S. Scott Department of Mathematics, University of California, Berkeley

vii Preface

This handbook is intended to give an in-depth description of the many research areas that make up the expanding field of model checking. In 32 chapters, 76 of the world’s leading researchers in this domain present a thorough review of the ori- gins, theory, methods, and applications of model checking. The book is meant for researchers and graduate students who are interested in the development of for- malisms, algorithms, and software tools for the computer-aided verification of complex systems in general, and of hardware and software systems in particular. The idea for this handbook originated with Helmut Veith around 2006. It was clear to Helmut that a field as strong and useful as model checking needed a handbook to make its foundations broadly accessible. Helmut was in many ways the soul of this project. His untimely death in March 2016, with the project in its final phase, was a shock to all of us and we greatly miss him. His visionary ideas, his unbelievable energy in bringing these ideas to life, and his wonderful sense of community left a lasting mark, and this book will serve as an enduring memorial to his contributions to the field and the community. Graz, Klosterneuburg, Pittsburgh Roderick Bloem November 2016 Thomas A. Henzinger Edmund M. Clarke

ix Acknowledgements

Very special thanks go to Katarina Singer and Sasha Rubin. As project coordinators, they assisted the editors in all managerial, organizational, and technical matters nec- essary for bringing such a large project to fruition: they managed the collaboration software and the interaction with the authors, reviewers, copy editors, and the publisher throughout much of the project. They kept track of all the loose ends and spent many hours making sure that every chapter got into shape. Jessica Davies and Andrey Kupriyanov performed some of these coordination tasks for shorter periods of time towards the end of the project. Thank you! We would like to thank Alfred Hofmann of Springer for his support of the project. Ronan Nugent, our editor at Springer, was closely involved in all aspects of the book from the beginning to its completion. He deserves special credit for ﬂexibility, patience, and for keeping an unrelenting trust in the project. We also thank Philip R. Watson, Springer’s copy editor, who labored through each and every chapter and found countless mistakes. The copy editing was supported by many volunteers, who proofread individual chapters, integrated changes suggested by Springer, discussed problems with the authors, and helped create the index. The editing team was largely based at IST Austria, TU Graz and TU Vienna and con- sisted of Guy Avni, Sergiy Bogomolov, Przemysław Daca, Jessica Davies, Masoud Ebrahimi, Thomas Ferrère, Miriam García Soto, Mirco Giacobbe, Vedad Hadzic, Rinat Iusupov, Barbara Jobstmann, Anja Karl, Ayrat Khalimov, Bettina Könighofer, Hui Kong, Igor Konnov, Tomer Kotek, Bernhard Kragl, Jan Kretínský,ˇ Orna Kupferman, Andrey Kupriyanov, Alfons Laarman, Kurt Nistelberger, Jan Otop, Leo Prikler, Heinz Riener, Franz Röck, Sasha Rubin, Roopsha Samanta, Moritz Sinn, Ana Sokolova, Thorsten Tarrach, Ursula Urwanisch, and Georg Weissenbacher. Thank you! A large group of people reviewed all of the chapters, often in great detail. We are grateful to Parosh Aziz Abdulla, Rajeev Alur, Domagoj Babic, Christel Baier, Clark Barrett, David Basin, Jason Baumgartner, Saddek Bensalem, Dietmar Berwanger, Dirk Beyer, Armin Biere, Bernard Boigelot, Marius Bozga, Julian Bradﬁeld, Robert K. Brayton, Tomáš Brázdil, Randal E. Bryant, Pavol Cerny,ˇ Sagar Chaki, Krishnendu Chatterjee, Swarat Chaudhuri, Wei-Ngan Chin, Hana Chockler,

xi xii Acknowledgements

Alessandro Cimatti, Rance Cleaveland, Cas Cremers, Dennis Dams, Thao Dang, Leonardo de Moura, David Dill, Laurent Doyen, Cezara Dragoi,˘ Rolf Drechsler, Matt Dwyer, Cindy Eisner, Javier Esparza, Kousha Etessami, Marco Faella, Azadeh Farzan, Bernd Finkbeiner, Alain Finkel, Dana Fisman, Vojtechˇ Forejt, Goran Frehse, Masahiro Fujita, Sicun Gao, Patrice Godefroid, Ganesh Gopalakrishnan, Erich Grädel, Alex Groce, Radu Grosu, Orna Grumberg, Sumit Gulwani, Aarti Gupta, Arie Gurﬁnkel, Reiner Hähnle, John Hatcliff, Keijo Heljanko, David Henriques, Holger Hermanns, Kryštof Hoder, Martin Hofmann, Andreas Holzer, Gerard Holzmann, Radu Iosif, Franjo Ivanciˇ c,´ Ranjit Jhala, Barbara Jobstmann, Joost-Pieter Katoen, Shmuel Katz, Stefan Katzenbeisser, Stefan Kiefer, Johannes Kinder, Felix Klaedtke, Igor Konnov, Eric Koskinen, Laura Kovács, Daniel Kroening, Antonín Kucera,ˇ Andreas Kuehlmann, Viktor Kuncak,ˇ Orna Kupferman, Robert P. Kurshan, Marta Kwiatowska, Shuvendu Lahiri, Yassine Lakhnech, Akash Lal, Salvatore La Torre, Martin Leucker, Rupak Majumdar, Oded Maler, Tiziana Margaria, Nicolas Markey, Joao Marquez-Silva, Wilfredo Marrero, João Martins, Richard Mayr, Catherine Meadows, Tom Melham, Shin-ichi Minato, Marius Minea, David Monniaux, Kedar S. Namjoshi, Dejan Nickoviˇ c,´ Joël Ouaknine, Madhusudan Parthasarathy, Corina Pas˘ areanu,˘ Doron Peled, Ruzica Piskac, André Platzer, Stefano Quer, Jan-David Quesel, Jean-François Raskin, Tom Reps, Sasha Rubin, John Rushby, Andrey Rybalchenko, Mooly Sagiv, Sriram Sankaranarayanan, Christian Schallhart, Sven Schewe, Holger Schlingloff, David A. Schmidt, Philippe Schnoebelen, Stefan Schwoon, Helmut Seidl, Martina Seidl, Koushik Sen, Sanjit A. Seshia, Natarajan Shankar, Natasha Sharygina, Nishant Sinha, Prasad Sistla, Ana Sokolova, Dawn Song, Bernhard Steffen, Wilfried Steiner, Andreas Steininger, Mariëlle Stoelinga, Ofer Strichman, Aaron Stump, Stefan Szeider, Murali Talupur, Wolfgang Thomas, Cesare Tinelli, Ashish Tiwari, Stavros Tripakis, Rachel Tzoref, Viktor Vafeiadis, Allen Van Gelder, Martin Vechev, Thomas Wahl, Igor Walukiewicz, Bow-Yaw Wang, Farn Wang, Georg Weissenbacher, Thomas Wies, Karsten Wolf, Pierre Wolper, Eran Yahav, Karen Yorav, Greta Yorsh, Vladimir Zakharov, and Paolo Zuliani. The greatest debt of gratitude we owe to the authors, who did the most important job: writing the book. The editors were supported in the initial selection of topics and authors by an advisory board consisting of Rajeev Alur, Armin Biere, Robert K. Brayton, Randal E. Bryant, Rance Cleaveland, Cindy Eisner, Javier Esparza, Orna Grumberg, Aarti Gupta, Pei-Hsin Ho, Gerard Holzmann, Orna Kupferman, Robert P. Kurshan, Kim G. Larsen, Ken McMillan, Doron Peled, Amir Pnueli, Sriram Rajamani, Joseph Sifakis, Prasad Sistla, Scott Smolka, Wolfgang Thomas, and Pierre Wolper. Many of them contributed also to individual chapters. Finally, we thank Dana Scott for writing the Foreword, Andrei Voronkov for letting us use his EasyChair software for the initial writing and reviewing process, and Anna Petukhova for the design of the cover image, which is based on her design for the 2014 Vienna Summer of Logic. Roderick Bloem Thomas A. Henzinger Edmund M. Clarke Contents

1 Introduction to Model Checking ...... 1 Edmund M. Clarke, Thomas A. Henzinger, and Helmut Veith 1.1 TheCaseforComputer-AidedVerification...... 1 1.2 Temporal-Logic Model Checking in a Nutshell ...... 6 1.3 A Very Brief Guide Through the Chapters of the Handbook . . 13 1.4 The Future of Model Checking ...... 19 References ...... 22 2 Temporal Logic and Fair Discrete Systems ...... 27 Nir Piterman and Amir Pnueli 2.1 Introduction ...... 28 2.2 FairDiscreteSystems...... 29 2.3 Linear Temporal Logic ...... 41 2.4 ComputationTreeLogic...... 52 2.5 ExamplesforLTLandCTL...... 59 2.6 CTL∗ ...... 63 References ...... 70 3 Modeling for Verification ...... 75 Sanjit A. Seshia, Natasha Sharygina, and Stavros Tripakis 3.1 Introduction ...... 75 3.2 MajorConsiderationsinSystemModeling...... 77 3.3 ModelingBasics...... 86 3.4 Examples...... 89 3.5 KripkeStructures...... 100 3.6 Summary...... 103 References ...... 103 4 Automata Theory and Model Checking ...... 107 Orna Kupferman 4.1 Introduction ...... 107 4.2 Nondeterministic Büchi Automata on Infinite Words ...... 108

xiii xiv Contents

4.3 Additional Acceptance Conditions ...... 122 4.4 Decision Procedures ...... 132 4.5 AlternatingAutomataonInfiniteWords...... 135 4.6 Automata-BasedAlgorithms...... 141 References ...... 148 5 Explicit-State Model Checking ...... 153 Gerard J. Holzmann 5.1 Introduction ...... 153 5.2 Basic Search Algorithms ...... 155 5.3 Linear Temporal Logic ...... 158 5.4 OmegaAutomata...... 158 5.5 Nested Depth-First Search ...... 160 5.6 Abstraction...... 162 5.7 Model-Driven Verification ...... 167 5.8 Incomplete Storage ...... 168 5.9 Extensions...... 169 5.10 Synopsis ...... 170 References ...... 170 6 Partial-Order Reduction ...... 173 Doron Peled 6.1 Introduction ...... 173 6.2 Partial Order Reduction ...... 174 6.3 Reducing Edges While Preserving States ...... 182 6.4 Conclusions ...... 188 References ...... 188 7 Binary Decision Diagrams ...... 191 Randal E. Bryant 7.1 Introduction ...... 191 7.2 Terminology...... 192 7.3 A Boolean Function API ...... 193 7.4 OBDDRepresentation...... 195 7.5 ImplementingOBDDOperations...... 197 7.6 Implementation Techniques ...... 200 7.7 VariableOrderingandReordering...... 202 7.8 VariantRepresentations...... 203 7.9 Representing Non-Boolean Functions ...... 206 7.10 Scaling OBDD Capacity ...... 210 7.11 Concluding Remarks ...... 213 References ...... 214 8 BDD-Based Symbolic Model Checking ...... 219 Sagar Chaki and Arie Gurfinkel 8.1 Introduction ...... 219 8.2 Preliminaries...... 220 Contents xv

8.3 BinaryDecisionDiagrams:TheBasics...... 222 8.4 Model Checking Kripke Structures ...... 230 8.5 Push-Down Symbolic Model Checking ...... 238 8.6 Conclusion ...... 243 References ...... 244 9 Propositional SAT Solving ...... 247 Joao Marques-Silva and Sharad Malik 9.1 Introduction ...... 247 9.2 Preliminaries...... 249 9.3 CDCLSATSolvers:Organization...... 252 9.4 CDCLSATSolvers...... 253 9.5 SAT-BasedProblemSolving...... 264 9.6 ResearchDirections...... 269 References ...... 269 10 SAT-Based Model Checking ...... 277 Armin Biere and Daniel Kröning 10.1 Introduction ...... 277 10.2 Bounded Model Checking on Kripke Structures ...... 278 10.3 Bounded Model Checking for Hardware Designs ...... 281 10.4 Bounded Model Checking for Software ...... 283 10.5 Encodings into Propositional SAT ...... 287 10.6 Complete Model Checking with SAT ...... 289 10.7 Abstraction Techniques Using SAT ...... 292 10.8 Outlook and Conclusions ...... 295 References ...... 295 11 Satisﬁability Modulo Theories ...... 305 Clark Barrett and Cesare Tinelli 11.1 Introduction ...... 305 11.2 SMT in Model Checking ...... 310 11.3 The Lazy Approach to SMT ...... 312 11.4 Theory Solvers for Speciﬁc Theories ...... 317 11.5 Combining Theory Solvers ...... 324 11.6 SMT Solving Extensions and Enhancements ...... 327 11.7 Eager Encodings to SAT ...... 330 11.8 Additional Functionalities of SMT Solvers ...... 332 References ...... 335 12 Compositional Reasoning ...... 345 Dimitra Giannakopoulou, Kedar S. Namjoshi, and Corina S. Pas˘ areanu˘ 12.1 Introduction ...... 345 12.2ReasoningwithAssertions...... 348 12.3Automata-BasedAssume-GuaranteeReasoning...... 362 12.4 Related Approaches ...... 375 12.5 Conclusion ...... 378 References ...... 378 xvi Contents

13 Abstraction and Abstraction Refinement ...... 385 Dennis Dams and Orna Grumberg 13.1 Introduction ...... 385 13.2Preliminaries...... 387 13.3SimulationandBisimulationRelations...... 394 13.4AbstractionBasedonSimulation...... 399 13.5 CounterExample-Guided Abstraction Refinement (CEGAR) . . 402 13.6 Abstraction Based on Modal Simulation ...... 406 13.7 Completeness ...... 412 References ...... 414 14 Interpolation and Model Checking ...... 421 Kenneth L. McMillan 14.1 Introduction ...... 421 14.2Preliminaries...... 423 14.3 Model of Abstraction Refinement ...... 424 14.4 Refinement, Local Proofs, and Interpolants ...... 428 14.5 Refiners as Local Proof Systems ...... 434 14.6 Abstractors as Proof Generalizers ...... 441 14.7Summary...... 443 References ...... 444 15 Predicate Abstraction for Program Verification ...... 447 Ranjit Jhala, Andreas Podelski, and Andrey Rybalchenko 15.1 Introduction ...... 447 15.2Definitions...... 448 15.3 Characterizing Correctness via Reachability ...... 452 15.4 Characterizing Correctness via Inductiveness ...... 454 15.5Abstraction...... 459 15.6 Abstraction Refinement ...... 471 15.7 Solving Refinement Constraints for Predicate Abstraction . . . 481 15.8Tools...... 486 15.9 Conclusion ...... 487 References ...... 487 16 Combining Model Checking and Data-Flow Analysis ...... 493 Dirk Beyer, Sumit Gulwani, and David A. Schmidt 16.1 Introduction ...... 494 16.2 General Considerations ...... 494 16.3UnifyingFormalFramework/ComparisonofAlgorithms.... 501 16.4 Classic Examples (Component Analyses) ...... 511 16.5CombinationExamples(CompositeAnalyses)...... 520 16.6AlgorithmsforConstructingProgramInvariants...... 525 16.7CombinationsinToolImplementations...... 532 16.8 Conclusion ...... 532 References ...... 534 Contents xvii

17 Model Checking Procedural Programs ...... 541 Rajeev Alur, Ahmed Bouajjani, and Javier Esparza 17.1 Introduction ...... 541 17.2 Models of Procedural Programs ...... 543 17.3BasicVeriﬁcationAlgorithms...... 547 17.4 Specifying Requirements ...... 556 17.5 Bibliographical Remarks ...... 566 References ...... 569 18 Model Checking Concurrent Programs ...... 573 Aarti Gupta, Vineet Kahlon, Shaz Qadeer, and Tayssir Touili 18.1 Introduction ...... 574 18.2 Concurrent System Model and Notation ...... 576 18.3 PDS-Based Model Checking: Synchronization Patterns ..... 580 18.4 PDS-Based Model-Checking: Communication Patterns ..... 595 18.5 Other Models: Finite State Systems and Sequential Programs . 602 References ...... 607 19 Combining Model Checking and Testing ...... 613 Patrice Godefroid and Koushik Sen 19.1 Introduction ...... 613 19.2 Systematic Testing of Concurrent Software ...... 615 19.3 Systematic Testing of Sequential Software ...... 624 19.4 Systematic Testing of Concurrent Software with Data Inputs . . 633 19.5OtherRelatedWork...... 637 19.6 Conclusion ...... 640 References ...... 640 20 Combining Model Checking and Deduction ...... 651 Natarajan Shankar 20.1 Introduction ...... 651 20.2 Logic Background ...... 656 20.3 Deduction and Model Checking ...... 670 20.4 Conclusions ...... 680 References ...... 680 21 Model Checking Parameterized Systems ...... 685 Parosh Aziz Abdulla, A. Prasad Sistla, and Muralidhar Talupur 21.1 Introduction ...... 685 21.2PetriNets...... 687 21.3 Regular Model Checking ...... 695 21.4 Monotonic Abstraction ...... 703 21.5 Compositional Reasoning for Parameterized Veriﬁcation .... 709 21.6RelatedWork...... 719 References ...... 721 xviii Contents

22 Model Checking Security Protocols ...... 727 David Basin, Cas Cremers, and Catherine Meadows 22.1 Introduction ...... 727 22.2History...... 731 22.3 Formal Model ...... 733 22.4 Issues in Developing Model-Checking Algorithms for Security Protocols ...... 741 22.5SystemsandAlgorithms...... 748 22.6ResearchProblems...... 753 22.7 Conclusions ...... 757 References ...... 758 23 Transfer of Model Checking to Industrial Practice ...... 763 Robert P. Kurshan 23.1 Introduction ...... 763 23.2 The Technology Transfer Problem ...... 767 23.3FalseStarts...... 776 23.4 A Framework for Technology Transfer ...... 779 23.5 Formal Functional Verification in Commercial Use Today . . . 782 23.6Algorithms...... 786 23.7Future...... 788 23.8 Conclusion ...... 789 References ...... 790 24 Functional Specification of Hardware via Temporal Logic ..... 795 Cindy Eisner and Dana Fisman 24.1 Introduction ...... 795 24.2FromLTLtoRegular-Expression-BasedTemporalLogic.... 797 24.3 Clocks and Sampling ...... 805 24.4 Hardware Resets and Other Sources of Truncated Paths ..... 809 24.5 The Simple Subset ...... 817 24.6 Quantified and Local Variables ...... 818 24.7SummaryandOpenIssues...... 823 References ...... 825 25 Symbolic Trajectory Evaluation ...... 831 Tom Melham 25.1 Introduction ...... 831 25.2 Notational Preliminaries ...... 833 25.3 Sequential Circuit Models in STE ...... 834 25.4TrajectoryEvaluationLogic...... 838 25.5 The Fundamental Theorem of Trajectory Evaluation ...... 843 25.6 STE Model Checking ...... 844 25.7AbstractionandSymbolicIndexing...... 852 25.8 Compositional Reasoning ...... 857 25.9GSTEandOtherExtensions...... 862 Contents xix

25.10 Summary and Prospects ...... 865 References ...... 867 26 The mu-calculus and Model Checking ...... 871 Julian Bradﬁeld and Igor Walukiewicz 26.1 Introduction ...... 871 26.2Basics...... 872 26.3 Fundamental Properties ...... 890 26.4RelationswithOtherLogics...... 904 26.5RelatedWork...... 912 References ...... 914 27 Graph Games and Reactive Synthesis ...... 921 Roderick Bloem, Krishnendu Chatterjee, and Barbara Jobstmann 27.1 Introduction ...... 921 27.2 Theory of Graph-Based Games ...... 923 27.3 Reactive Synthesis ...... 936 27.4RelatedTopics...... 953 References ...... 954 28 Model Checking Probabilistic Systems ...... 963 Christel Baier, Luca de Alfaro, Vojtechˇ Forejt, and Marta Kwiatkowska 28.1 Introduction ...... 964 28.2 Modelling Probabilistic Concurrent Systems ...... 966 28.3 Probabilistic Computation Tree Logic ...... 975 28.4 Model-Checking Algorithms for MDPs and PCTL ...... 980 28.5 Linear Temporal Logic ...... 985 28.6 Model-Checking Algorithms for MDPs and LTL ...... 987 28.7 Tools, Applications and Model Construction ...... 990 28.8 Extensions of the Model and Speciﬁcation Notations ...... 992 28.9 Conclusion ...... 993 References ...... 993 29 Model Checking Real-Time Systems ...... 1001 Patricia Bouyer, Uli Fahrenberg, Kim Guldstrand Larsen, Nicolas Markey, Joël Ouaknine, and James Worrell 29.1 Introduction ...... 1001 29.2TimedAutomata...... 1003 29.3 Checking Reachability ...... 1007 29.4 (Bi)simulation Checking ...... 1010 29.5 Language-Theoretic Properties ...... 1012 29.6TimedTemporalLogics...... 1018 29.7SymbolicAlgorithms,DataStructures,Tools...... 1023 29.8WeightedTimedAutomata...... 1028 29.9TimedGames...... 1034 29.10 Ongoing and Future Challenges ...... 1037 References ...... 1037 xx Contents

30 Verification of Hybrid Systems ...... 1047 Laurent Doyen, Goran Frehse, George J. Pappas, and André Platzer 30.1 Introduction ...... 1048 30.2BasicDefinitions...... 1049 30.3 Decidability and Undecidability Results ...... 1055 30.4 Set-Based Reachability Analysis ...... 1058 30.5Abstraction-BasedVerification...... 1075 30.6Logic-BasedVerification...... 1084 30.7VerificationTools...... 1097 References ...... 1102 31 Symbolic Model Checking in Non-Boolean Domains ...... 1111 Rupak Majumdar and Jean-François Raskin 31.1 Introduction ...... 1111 31.2 Transition Systems and Symbolic Verification ...... 1112 31.3ExamplesofSymbolicVerification...... 1119 31.4 Games and Symbolic Synthesis ...... 1131 31.5 Probabilistic Systems ...... 1137 31.6 Conclusion ...... 1141 References ...... 1143 32 Process Algebra and Model Checking ...... 1149 Rance Cleaveland, A.W. Roscoe, and Scott A. Smolka 32.1 Introduction ...... 1149 32.2 Foundations ...... 1150 32.3 Algorithms and Methodologies ...... 1175 32.4Tools...... 1181 32.5CaseStudies...... 1185 32.6 Conclusions ...... 1190 References ...... 1191 Index ...... 1197 Contributors

Parosh Aziz Abdulla Uppsala University, Uppsala, Sweden

Rajeev Alur University of Pennsylvania, Philadelphia, PA, USA

Christel Baier Technische Universität Dresden, Dresden, Germany

Clark Barrett Stanford University, Stanford, CA, USA

David Basin ETH Zürich, Zürich, Switzerland

Dirk Beyer Ludwig-Maximilians-Universität München, Munich, Germany

Armin Biere Johannes Kepler University, Linz, Austria

Roderick Bloem Graz University of Technology, Graz, Austria

Ahmed Bouajjani Institut Universitaire de France, Paris Diderot University (Paris 7), Paris, France

Patricia Bouyer LSV, CNRS & ENS Paris-Saclay, Cachan, France

Julian Bradﬁeld University of Edinburgh, Edinburgh, UK

Randal E. Bryant Carnegie Mellon University, Pittsburgh, PA, USA

Sagar Chaki Software Engineering Institute, Carnegie Mellon University, Pitts- burgh, PA, USA

Krishnendu Chatterjee IST Austria, Klosterneuburg, Austria

Edmund M. Clarke Carnegie Mellon University, Pittsburgh, PA, USA

Rance Cleaveland University of Maryland, College Park, College Park, MD, USA

Cas Cremers University of Oxford, Oxford, UK

xxi xxii Contributors

Dennis Dams Embedded Systems Innovation by TNO, Eindhoven, The Nether- lands Luca de Alfaro University of California, Santa Cruz, Santa Cruz, CA, USA Laurent Doyen LSV, CNRS & ENS Paris-Saclay, Cachan, France Cindy Eisner IBM Research – Haifa, Haifa, Israel Javier Esparza Technical University of Munich, Munich, Germany Uli Fahrenberg École Polytechnique, Palaiseau, France Dana Fisman Ben-Gurion University, Be’er Sheva, Israel Vojtechˇ Forejt University of Oxford, Oxford, UK Goran Frehse Verimag, University Grenoble Alpes, Grenoble, France Dimitra Giannakopoulou NASA Ames Research Center, Moffett Field, CA, USA Patrice Godefroid Microsoft Research, Redmond, WA, USA Orna Grumberg Technion, Israel Institute of Technology, Haifa, Israel Sumit Gulwani Microsoft Research, Redmond, WA, USA Aarti Gupta Princeton University, Princeton, NJ, USA Arie Gurﬁnkel University of Waterloo, Waterloo, ON, Canada Thomas A. Henzinger IST Austria, Klosterneuburg, Austria Gerard J. Holzmann Nimble Research, Monrovia, CA, USA Ranjit Jhala Jacobs School of Engineering, University of California, San Diego, San Diego, CA, USA Barbara Jobstmann École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland Vineet Kahlon Google Inc., New York, NY, USA Daniel Kröning University of Oxford, Oxford, UK Orna Kupferman Hebrew University, Jerusalem, Israel Robert P. Kurshan New York, NY, USA Marta Kwiatkowska University of Oxford, Oxford, UK Kim Guldstrand Larsen Aalborg University, Aalborg, Denmark Rupak Majumdar Max Planck Institute for Software Systems, Kaiserslautern, Germany Contributors xxiii

Sharad Malik Princeton University, Princeton, NJ, USA Nicolas Markey LSV, CNRS & ENS Paris-Saclay, Cachan, France; IRISA, CNRS & INRIA & Univ. Rennes 1, Rennes, France Joao Marques-Silva University of Lisbon, Lisbon, Portugal Kenneth L. McMillan Microsoft Research, Redmond, WA, USA Catherine Meadows Naval Research Laboratory, Washington, DC, USA Tom Melham University of Oxford, Oxford, UK Kedar S. Namjoshi Bell Labs, Nokia, Murray Hill, NJ, USA Joël Ouaknine University of Oxford, Oxford, UK; Max Planck Institute for Soft- ware Systems, Saarbrücken, Germany George J. Pappas University of Pennsylvania, Philadelphia, PA, USA Corina S. Pasù areanuù NASA Ames Research Center, Moffett Field, CA, USA Doron Peled Bar Ilan University, Ramat Gan, Israel Nir Piterman University of Leicester, Leicester, UK André Platzer Carnegie Mellon University, Pittsburgh, PA, USA Amir Pnueli New York University, New York, USA; The Weizmann Institute of Science, Rehovot, Israel Andreas Podelski University of Freiburg, Freiburg, Germany Shaz Qadeer Microsoft Research, Redmond, WA, USA Jean-François Raskin Université libre de Bruxelles, Brussels, Belgium A.W. Roscoe University of Oxford, Oxford, UK Andrey Rybalchenko Microsoft Research, Cambridge, UK David A. Schmidt Kansas State University, Manhattan, KS, USA Koushik Sen University of California, Berkeley, Berkeley, CA, USA Sanjit A. Seshia University of California, Berkeley, Berkeley, CA, USA Natarajan Shankar SRI International, Menlo Park, CA, USA Natasha Sharygina Università della Svizzera italiana, Lugano, Switzerland A. Prasad Sistla University of Illinois, Chicago, IL, USA Scott A. Smolka Stony Brook University, Stony Brook, NY, USA Muralidhar Talupur Intel’s Strategic CAD Labs, Hillsboro, OR, USA Cesare Tinelli The University of Iowa, Iowa City, IA, USA xxiv Contributors

Tayssir Touili CNRS, Paris, France Stavros Tripakis University of California, Berkeley, Berkeley, CA, USA; Aalto University, Espoo, Finland Helmut Veith Technische Universität Wien, Vienna, Austria Igor Walukiewicz CNRS, University of Bordeaux, Bordeaux, France James Worrell University of Oxford, Oxford, UK Chapter 1 Introduction to Model Checking

Edmund M. Clarke, Thomas A. Henzinger, and Helmut Veith

Abstract Model checking is a computer-assisted method for the analysis of dynamical systems that can be modeled by state-transition systems. Drawing from research traditions in mathematical logic, programming languages, hardware design, and theoretical computer science, model checking is now widely used for the veriﬁcation of hardware and software in industry. This chapter is an introduction and short survey of model checking. The chapter aims to motivate and link the individual chapters of the handbook, and to provide context for readers who are not familiar with model checking.

1.1 The Case for Computer-Aided Veriﬁcation

The only effective way to raise the conﬁdence level of a program signiﬁcantly is to give a convincing proof of its correctness. [32] In the ideal world of Dijkstra’s Turing Award Lecture 1972, programs are intellectu- ally manageable, and every program grows hand in hand with a mathematical proof of the program’s correctness. The history of computer science has proven Dijkstra’s vision limited. Manual proofs, if at all, can be found only in students’ exercises, research papers on algorithms, and certain critical application areas. Although the work of McCarthy, Floyd, Hoare, and other pioneers [7, 36, 45, 62, 66] provided us with formal proof systems for program correctness, little use is made of these techniques in practice. The main challenge is scalability: real-world software systems not only include complex control and data structures, but depend on much “context” such as libraries and interfaces to other code, including lower-level systems code. As

E.M. Clarke Carnegie Mellon University, Pittsburgh, PA, USA

T.A. Henzinger (B) IST Austria, Klosterneuburg, Austria e-mail: [email protected]

H. Veith Technische Universität Wien, Vienna, Austria

© Springer International Publishing AG, part of Springer Nature 2018 1 E.M. Clarke et al. (eds.), Handbook of Model Checking, DOI 10.1007/978-3-319-10575-8_1 2 E.M. Clarke et al. a result, proving a software system correct requires much more effort, knowledge, training, and ingenuity than writing the software in trial-and-error style. This asym- metry between coding and verification is the main motivation for computer-aided verification, i.e., the use of computers for the verification of software and hardware. From a 1972 perspective, the computer industry has changed the world beyond recognition. Computer programs today have millions of lines of code, they are writ- ten and maintained by globally distributed teams over decades, and they are used in diverse and complex computing environments from micro-code to cloud computing. Computer science has become pervasive in production, transportation, infras- tructure, health care, science, finance, administration, defense, and entertainment. Programs are the most complex machines built by humans, and have huge responsi- bilities for human safety, security, health, and well-being. These developments have exacerbated the challenges and, at the same time, dramatically increased the need for correct programs and, hence, for computer-aided verification. Starting with the work of Turing, the perspectives for automated verification did not look promising. Turing’s halting problem [82] and Rice’s Theorem [79]tell us that computer-aided verification is, in general, an unsolvable problem. At face value, these theorems demonstrate the undecidability of verification even for simple properties of simple programs. Technically, all that is needed for undecidability are two integer variables that, embedded into a looping control structure, can be incremented, decremented, and checked for zero. If the values of integer variables are bounded, we obtain a system with finitely many different states, and verification becomes decidable. However, complexity theory tells us that even for finite-state systems, many verification questions require a prohibitive effort. Yet, at a time when logic in computer science was a synonym for undecidability and intractability, the invention of model checking marked a paradigm shift towards the practical use of logic for bug finding—i.e., falsification rather than verification— in the hardware and software industries. Not untypical of paradigms acquired many decades ago, the case for model checking appears simple and convincing in ret- rospect [22, 23, 72, 75]. In its basic classical form, the paradigm consists of the following insights: Modeling. Finite state-transition graphs provide an adequate formalism for the description of finite-state systems such as hardware, but also for finite-state abstractions of software and of communication protocols. Specification. Temporal logics provide a natural framework for the description of correctness properties for state-transition systems. Algorithms. There are decision procedures for determining whether a finite state- transition structure is a model of a temporal-logic formula. Moreover, the decision procedures can produce diagnostic counterexamples when the formula is not true in the structure. Taken together, these insights motivate the methodology that is shown in Fig. 1:the system under investigation is compiled into a state-transition graph (a.k.a. Kripke structure) K, the specification is expressed as a temporal-logic formula ϕ, and a decision procedure—the model checker—decides whether K | ϕ, i.e., whether the 1 Introduction to Model Checking 3

Fig. 1 Basic model-checking methodology

structure K is a model of the formula ϕ.IfK | ϕ, then the model checker outputs a counterexample that witnesses the violation of ϕ by K. The generation of counterexamples means that, in practice, falsification (the detection of bugs) can often be faster than verification (the proof of their absence). There is of course a mismatch between the early model checkers, which were essentially graph algorithms on Kripke structures, and the complexity of modern computer systems sketched above. Research in model checking spanning more than three decades has helped to close this gap in many areas that are documented throughout this handbook. We can roughly classify the advances in model checking and the chapters of this handbook in terms of two recurrent themes that have driven much of the research agenda in model checking: 1 The algorithmic challenge: Design model-checking algorithms that scale to real-life problems. The main practical problem in model checking is the com- binatorial explosion of states in the Kripke structure—the “state-explosion problem.” Since each state represents the global system status at a given time point, a state is essentially a memory snapshot of the system under investigation, and, thus, the size of the state space is exponential in the size of the memory. There- fore, even for systems of relatively modest size, it is impossible to compute and analyze the entire corresponding Kripke structure directly. In fact, in most situa- tions, the state space is not finite (e.g., unbounded memory, unbounded number of parallel processes), which leads to the modeling challenge. In practice, very large and infinite state-transition systems are approximated by effective abstractions such as finite-state abstractions, or decision procedures are approximated by so-called “semi-algorithms,” which are aimed at finding bugs but may fail to prove correctness, or both. 2 The modeling challenge: Extend the model-checking framework beyond Kripke structures and temporal logic. Kripke structures are natural representations for various flavors of communicating finite-state machines. To model and specify unbounded iteration and recursion, unbounded concurrency and distribution, process creation and reconfiguration, unbounded data types, real-time and cyber- physical systems, probabilistic computation, security aspects, etc., and to abstract these features effectively, we need to extend the modeling and specification frameworks beyond Kripke structures and classical temporal logics. Some 4 E.M. Clarke et al.

extensions maintain decidability, often through the construction of finite-state abstractions, which ties the modeling challenge back to the algorithmic challenge. Other extensions sacrifice decidability, but maintain the ability of model checking to find bugs automatically, systematically, and early in the system design process. As the two challenges are tied together closely, many chapters of this handbook address both. New models without algorithms and without experimental validation are not central to model-checking research. We believe that the main strengths of model checking are threefold. First, model checking is a systematic, algorithmic methodology which can be computerized and, ideally, fully automated. Thus, model-checking tools have the goal and promise to be used during the design process by hardware and software engineers without the assistance of verification experts. Second, model checking can be applied at different stages of the design process, on abstract models as well as on implemented code. While any one model-checking tool is usually limited to particular modeling and specification languages, the methodology as such is not restricted to any level or formalism and can be applied at different levels, to incomplete systems and partial specifications, to find different kinds of bugs. Third, and perhaps most importantly, model checking deals particularly well with concurrency. The interaction between parallel processes is one of the main sources of complexity and errors in system design. Moreover, concurrency errors are especially subtle, contingent, and therefore difficult to reproduce; they are hard to find by testing the system, and their absence is hard to prove by logical arguments or program analyses because of the extremely large number of possible interactions between parallel processes. Model checking, on the other hand, which is based on the algorithmic exploration of large state spaces, is particularly well suited for finding concurrency bugs and proving their absence. These strengths distinguish model checking from related approaches for improv- ing system quality: Testing is the fastest and simplest way to detect errors. It lends itself to easy automation and can often be handled with limited academic background. Testing covers a large spectrum from manual, ad hoc testing of code all the way to automated ef- forts and model-based testing [37, 48, 50]. Fundamentally dynamic, testing is able to detect compiler errors, hardware errors, and modeling errors that are invisible to static tools; it is integral to any comprehensive approach to safety engineering and software certification [60]. However, “program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence” [32], especially for concurrent systems. Model checking, on the other hand, provides a systematic approach to bug detection that can be applied to incomplete systems and, even for concurrent systems, offers in the limit—when no more errors are found—a certificate of correctness. In other words, while testing is only debugging, model checking is systematic debugging aimed at model verification. Chapter 19 will discuss the cross-fertilization of modern model-checking and white-box testing techniques. 1 Introduction to Model Checking 5

Abstract interpretation and other program analyses are similar to model checking in that they employ algorithms to prove program properties automatically [29, 68]. Following the tradition of programming-language semantics rather than logic, static analysis frameworks such as data-flow analysis, abstract interpretation, and rich type systems are built on lattice theory. Historically, their applications have focused mainly on fast analyses for compilers, in particular on overap- proximative analyses of numerical data types and assertion violations [6]. Thus, in comparison to model checking, the focus of program analyses is on efficiency rather than expressiveness, and on code rather than models. In recent research on software verification, abstract interpretation and model checking have converged to a large extent. Chapters 15 and 16 will explore the rich relationship between abstract interpretation and model checking in depth. Higher-order theorem proving is a powerful and proven approach for the verification of complex systems. Semi-manual theorem provers [13, 69, 85] have been used to verify critical systems ranging from floating-point arithmetic [1]tomicro- kernels [51], and even to the proof of deep mathematics such as the Kepler con- jecture [40]. In comparison to model checking, the focus of higher-order theorem proving is on expressiveness rather than efficiency: it handles data manipulation precisely and typically aims at full functional correctness. Theorem proving naturally incorporates the manual decomposition of a verification problem, which is obligatory for complex systems. But even for experts, developing proofs in higher-order logic tools is a time-consuming and often non-trivial effort worthy of a research publication. Chapter 20 will explore some connections and combinations of model checking and theorem proving. While the different verification methods have different historical starting points and different communities, ultimately they simply represent different trade-offs on the efficiency versus expressiveness (or precision) spectrum: greater expressive power tends to take us, at the cost of efficiency, from testing to abstract interpretation to model checking to theorem proving, and, over time, the differences become smaller. Through more than three decades, model checking has acquired and adapted methods from all of these research areas, but also from automata theory, process algebra, graph algorithms, game theory, hardware simulation, stochastic processes, control theory, and many other areas. At the same time, model checking has interacted with target areas for verification, most importantly computer engineering and VLSI design, software engineering and programming languages, embedded and cyber- physical systems, artificial intelligence, and even computational biology. It is thus fair to say that model checking is characterized less by purity of method than by the goal of debugging and analyzing dynamical systems that exist in the real world and can be modeled as state-transition systems. Model checking has been covered by several monographs [8, 11, 12, 26, 46, 47, 54, 55, 67, 71] and surveys [25, 28, 33]. The early history of model checking is documented in several collections of essays [38, 63]. The rest of this chapter serves as an introduction to the handbook for readers with little familiarity with the material. In Sect. 1.2 we give a minimalist introduction to the classical setting of temporal-logic model checking over Kripke structures. Section 1.3 then revisits 6 E.M. Clarke et al.

Fig. 2 American and Austrian trafﬁc lights as Kripke structures some of the fundamental challenges for model checking and puts the individual chapters of the handbook in perspective. In Sect. 1.4 we provide a brief outlook for the ﬁeld.

1.2 Temporal-Logic Model Checking in a Nutshell

We give a brief introduction to the classical view of temporal-logic model checking.

1.2.1 Kripke Structures

Kripke structures [53] are finite directed graphs whose vertices are labeled with sets of atomic propositions. The vertices and edges of the graph are called “states” and “transitions,” respectively. In our context, they are used to represent the possible configurations and configuration changes of a discrete dynamical system. For a simple illustration, consider the two Kripke structures in Fig. 2, which represent the states and state transitions of traffic lights in the USA and Austria, respectively. Formally, a Kripke structure over a set A of atomic propositions is a triple K =S,R,L where S is a finite set of states (the “state space”), R ⊆ S × S is a set of transitions (the “transition relation”), and the labeling function L: S → 2A associates each state with a set of atomic propositions. For a state s ∈ S,thesetL(s) represents the set of atomic propositions that are true when the system is in state s, and the set A \ L(s) contains the propositions that are false in state s. We assume that the transition relation R is total, i.e., that all states have non-zero outdegree. The dynamic behavior of the system represented by a Kripke structure cor- responds to a path through the graph. A path is a finite or infinite sequence π = s0,s1,s2,... of states such that (si,si+1) ∈ R for all i ≥ 0. The totality of the transition relation ensures that every finite path can be extended to an infinite path. Given an infinite path π, we write L(π) = L(s0), L(s1), L(s2),...for the corresponding infinite sequence of sets of atomic propositions. Moreover, we write i π = si,si+1,si+2,... for the infinite path that results from π by removing the first i states. Related models that are based on discrete state changes are sometimes called automata, state machines, state diagrams, labeled transition systems, etc. Throughout 1 Introduction to Model Checking 7 this book, such models are used in accordance with the traditions of the respective research areas. In most cases, algorithmic results can be easily transferred between these models. Later handbook chapters will introduce more advanced frameworks— e.g., for modeling recursive (Chap. 17), probabilistic (Chap. 28), and real-time (Chap. 29) behavior—which extend finite state-transition systems in fundamental ways and require more intricate analysis techniques.

1.2.2 The Temporal Logic CTL

CTL [34] is a propositional modal logic with path quantifiers, which are interpreted over states, and temporal operators, which are interpreted over paths. Path quantifiers: A “for every infinite path from this state” E “there exists an infinite path from this state” Temporal operators (for atomic propositions p and q): Xp “p holds at the next state” Fp “p holds at some state in the future” Gq “q holds at all states in the future” qUp “p holds at some state in the future, and q holds at all states until p holds” For instance, Fp holds on path π iff π contains a state with label p, and Aϕ holds at state s iff ϕ holds on all infinite paths that start from state s. Given a set of atomic propositions A, the syntax of CTL is defined recursively as follows: –Ifp ∈ A, then p is a formula of CTL. –Ifϕ and ψ are formulas of CTL, then ϕ ∨ ψ, ϕ ∧ ψ, ¬ϕ, Aϕ, Eϕ, Xϕ, Fϕ, Gψ, and ψUϕ are formulas of CTL. To reflect the distinction between path quantifiers and temporal operators, we distinguish a syntactic subset of CTL called state formulas. State formulas are boolean combinations of atomic propositions and CTL formulas whose outermost operator is a path quantifier, i.e., they start with A or E. As in the example of Aϕ above, the truth value of a state formula can be asserted over a state in a Kripke structure. For all other formulas of CTL, we need a path to determine the truth value. The formal semantics of CTL is based on this syntactic distinction, and presented in Table 1. An easy exercise on the semantics defined in Table 1 shows that the syntactic restriction of CTL to one of the path quantifiers and the two temporal operators X and U yields the full expressive power of CTL, i.e., all other operators can be defined from these three operators. Given a Kripke structure K, state s, and state formula f , a model-checking al- gorithm is a decision procedure for K,s | f . In case of K,s | f , many model- checking algorithms provide evidence of the violation of the satisfaction relation, 8 E.M. Clarke et al.

Table 1 Semantics of CTL. Here, K is a Kripke structure, π is a path, s is a state, p is an atomic proposition, f and g are state formulas, and ϕ and ψ are CTL formulas