User Interaction in Deductive Interactive Program Verification Zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften von der KIT-Fakult¨atf¨urInformatik des Karlsruher Instituts f¨urTechnologie (KIT) genehmigte Dissertation von Sarah Caecilia Grebing aus Mannheim Tag der m¨undlichen Pr¨ufung: 07. Februar 2019 Erster Gutachter: Prof. Dr. Bernhard Beckert Zweiter Gutachter: Assoc. Prof. Dr. Andr´ePlatzer Contents Deutsche Zusammenfassung xv 1. Introduction1 1.1. Structure and Contribution of this Thesis . .2 1.1.1. Qualitative, Explorative User Studies . .3 1.1.2. Interaction Concept for Interactive Program Verification . .5 1.2. Previously Published Material . .6 I. Foundations for this Thesis9 2. Usability of Software Systems: Background and Methods 11 2.1. Human-Computer Interaction . 12 2.2. User-Centered Process . 12 2.3. Usability . 13 2.3.1. What is Usability? . 13 2.3.2. Usability Principles . 14 2.4. Interactions . 16 2.4.1. Models of Interaction . 16 2.4.2. Interaction Styles . 18 2.5. Task Analysis . 20 2.5.1. A brief Introduction to Sequence Models . 20 2.6. Evaluation Methods . 22 2.6.1. Questionnaires . 23 2.6.2. Interviews . 24 2.6.3. Focus Groups . 24 2.6.4. Preparation and Conduction of Focus Groups and Interviews . 25 3. Interactive Deductive Program Verification 29 3.1. Introduction . 29 3.2. Logical Calculi . 30 3.3. Specification of Java Programs with JML . 32 3.3.1. Method Contracts . 33 3.3.2. Loop Invariants . 34 3.3.3. Class Invariants . 36 3.3.4. The Purpose of Specifications . 36 3.4. A Brief Introduction to Java Dynamic Logic (JavaDL) . 36 3.4.1. A Sequent Calculus for JavaDL . 38 iii Contents 3.4.2. Symbolic Execution . 41 3.4.3. Taclets . 43 3.5. Proof Process for Deductive Program Verification . 45 3.6. Interaction Styles in Interactive Program Verification Systems . 47 3.6.1. Annotation-Based Interaction . 48 3.6.2. Text-Based Interaction . 49 3.6.3. Direct Manipulation Interaction . 50 3.7. Program Verification Systems . 51 3.7.1. Direct Manipulation Interaction: KeY and KeYmaeraX . 51 3.7.2. Annotation-Based Interaction: Dafny and Why3 . 54 3.7.3. Script-Based Interaction: Isabelle/HOL and Coq . 58 II. Context of Use 61 4. User Study with Focus Groups 65 4.1. Problem Description and Research Hypothesis . 65 4.2. Study Design and Implementation . 68 4.2.1. The Script . 68 4.2.2. Participants and Setup of the Study . 71 4.2.3. Conducting the Focus Group Discussions . 72 4.3. Data Analysis . 73 4.3.1. Targets of Evaluation: KeY and Isabelle . 73 4.3.2. Strengths and Weaknesses of the Targets of Evaluation . 73 4.3.3. User Support during the Proof Process . 77 4.3.4. Mechanisms Supporting Proof State Comprehension . 80 4.3.5. The Ideal Interactive Proof System . 82 4.4. Discussion . 83 4.5. Conclusion and Future Work . 84 5. User Study: Interviews with Practical Tasks 87 5.1. Introduction . 88 5.1.1. Research Questions For This Study . 89 5.2. Prototypical History Mechanism . 90 5.2.1. Origin of Formulas . 90 5.2.2. Mocked Mechanism . 92 5.3. Methodology . 93 5.4. Script Design . 96 5.5. Running the User Study . 107 5.5.1. Moderator . 107 5.5.2. Technical Setup . 108 5.5.3. Recordings and Transcription . 108 5.5.4. Participants . 109 5.6. Results of the User Study { Proof Process . 109 5.6.1. The Proof Process . 110 iv Contents 5.6.2. Expectations if a Proof Attempt in KeY is Unfinished . 111 5.6.3. Approaches to Proceed in the Verification Process . 114 5.6.4. Improvements for User Support in the Proof Process . 118 5.6.5. Practical Task: Proof Process . 122 5.6.6. Orientation After Applying Automatic Strategies . 134 5.7. Results of the User Study { Origin of Formulas . 141 5.7.1. History Mechanism . 141 5.7.2. Origin of formulas (Practical Tasks) . 146 5.7.3. Intuition about the Origin of Formulas . 148 5.8. Conclusion and Discussion . 149 6. Summary and Conclusion 153 6.1. Summary and Conclusion . 153 6.2. Related Work . 155 III. Integration 159 7. Proof Scripting Language 163 7.1. Introduction . 163 7.2. Characteristics of Program Verification Proofs . 164 7.3. Concept for a Proof Scripting Language . 165 7.4. Prerequisites For the Proof Scripting Language . 166 7.5. Script Language Constructs . 168 7.6. An Instantiation of the Language Concept for a Proof Scripting Lan- guage for KeY . 173 7.6.1. Syntax of KPS ............................ 174 7.6.2. Configuration and Variables . 177 7.7. Formalized Semantics of KPS ........................ 178 7.7.1. Evaluation of Matching Expressions . 184 7.7.2. The keywords closes and derivable ............... 192 7.8. Conclusion and Future Work . 193 7.9. Related Work . 194 8. Proof Script Debugger 197 8.1. Debugging Proof Attempts . 198 8.1.1. Analogy between Programs and Proof Scripts . 199 8.1.2. Analogy between Debugging and Failed Proof Analysis . 200 8.1.3. Adoption of Program Debugging Methods for Proof Debugging . 201 8.2. Integrating Direct-Manipulation and Script-Based Interaction . 203 8.3. First Experiments Using the Proof Script Debugger and KPS ...... 212 8.3.1. Objectives of the Experiments . 212 8.3.2. Performing the Experiments . 213 8.3.3. Analysis of the Results and Room for Improvement . 214 v Contents 8.3.4. Experiences in Using PSDBG for the Experiments and Improve- ments . 215 8.3.5. Lessons Learned from the Evaluation . 215 8.4. Conclusion . 218 8.5. Related Work . 219 9. Proof Exploration 223 9.1. Introduction . 223 9.2. Our Concept for Proof Exploration . 224 9.2.1. Reasons and Corrective Actions for Unfinished Proof Attempts . 225 9.2.2. Exploration Actions . 226 9.3. The Exploration Mode . 228 9.4. Interaction in the Exploration Mode . 229 9.4.1. Interplay: Exploration Mode and Regular Proof Mode . 229 9.4.2. Proof Exploration in Action . 231 9.4.3. presentation of Additional Information in the Exploration Mode 234 9.5. Related Work . 236 9.6. Conclusion and Future Work . 237 10.Seamless Program Verification 239 10.1. The Structure of Verification Tasks . 241 10.2. Description of Our Concept . 242 10.2.1. Projections: Multiple Views onto the Proof Problem . 243 10.2.2. Logical and Proof Construction View . 247 10.2.3. Relations between Proof Artifacts . 249 10.3. A Concretization of the Concept . 250 10.3.1. System and Proof Overview . 251 10.3.2. The Source Code View . 253 10.3.3. The Interplay between System and Proof Overview and Source Code View .............................. 255 10.3.4. Logical and Proof Construction View . 255 10.3.5. Interplay between the Source Code View and the Logical and Proof Construction View ...................... 259 10.4. Conclusion and Future Work . 260 10.5. Related Work . 261 IV. Conclusion 265 11.Conclusions 267 V. Bibliography 271 vi Contents VI. Appendix 287 A. Appendix: Focus Groups 289 A.1. Examples for Visual Cues in the Focus Group Discussions . 289 A.1.1. Isabelle . 289 A.1.2. KeY System . 290 B. Appendix: User Study 291 B.1. Proof Process . ..