Improving Software Fault Injection Ph.D. Thesis Erik van der Kouwe Vrije Universiteit Amsterdam, 2016 This work was funded in part by European Research Council under ERC Advanced Grant 227874. Copyright © 2016 by Erik van der Kouwe. ISBN XXX-XX-XXXX-XXX-X Printed by XXX. VRIJE UNIVERSITEIT IMPROVING SOFTWARE FAULT INJECTION ACADEMISCH PROEFSCHRIFT ter verkrijging van de graad Doctor aan de Vrije Universiteit Amsterdam, op gezag van de rector magnificus prof. dr. Vinod Subramaniam, in het openbaar te verdedigen ten overstaan van de promotiecommissie van de Faculteit der Exacte Wetenschappen op X XXX 2016 om X.XX uur in de aula van de universiteit, De Boelelaan 1105 door ERIK VAN DER KOUWE geboren te Leidschendam, Nederland promotor: prof.dr. A.S. Tanenbaum “TODO add quote.” TODO add quote source. Contents Contents vii List of Figures xi List of Tables xiii Publications xv 1 Introduction 1 2 Finding fault with fault injection An Empirical Exploration of Distortion in Fault Injection Experiments 7 2.1 Introduction . 7 2.2 Related work . 12 2.3 Fidelity . 14 2.4 Approach . 16 2.5 Programs and workloads . 17 2.6 Results . 19 2.6.1 Coverage . 19 2.6.2 Execution count . 26 2.6.3 Relationship between execution count and coverage . 30 2.6.4 Relationship between faults and execution . 31 2.7 Threats to validity . 33 2.8 Recommendations . 35 2.9 Conclusion . 36 vii viii 3 On the Soundness of Silence Investigating Silent Failures Using Fault Injection Experiments 39 3.1 Introduction . 39 3.2 Approach . 41 3.2.1 Fault injection . 42 3.2.2 Program behavior . 43 3.2.3 Comparing logs . 43 3.2.4 Silent failures . 45 3.2.5 General applicability . 46 3.3 Programs and workloads . 46 3.4 Results . 48 3.4.1 Differences across programs . 50 3.4.2 Differences across fault types . 53 3.4.3 Impact of ease of reachability . 54 3.5 Threats to validity . 56 3.6 Related work . 57 3.7 Conclusion . 59 4 A Methodology to Efficiently Compare Operating System Stability 61 4.1 Introduction . 61 4.2 Related work . 63 4.3 Approach . 64 4.3.1 Fault injection . 64 4.3.2 Fault selection . 65 4.3.3 Classification of results . 66 4.3.4 Operating systems and workloads . 68 4.3.5 General applicability . 69 4.4 Results . 69 4.4.1 Coverage . 70 4.4.2 Fault activation . 70 4.4.3 Scalability . 71 4.4.4 Systems and workloads . 72 4.4.5 Operating system components . 73 4.4.6 Activation time and fault latency . 74 4.5 Threats to validity . 75 4.6 Conclusion . 76 5 HSFI: representative fault injection scalable to large code bases 77 5.1 Introduction . 77 5.1.1 Contributions . 79 5.2 Background . 79 5.3 Overview . 82 5.4 Implementation . 84 CONTENTS ix 5.4.1 Injecting faults . 85 5.4.2 Fault candidate markers . 87 5.4.3 Binary patching . 88 5.5 Evaluation . 88 5.5.1 Run-time performance . 90 5.5.2 Time taken per experiment . 91 5.5.3 Marker recognition . 95 5.5.4 Threats to validity . 96 5.6 Limitations . 97 5.7 Related work . 97 5.7.1 Use of software fault injection . 97 5.7.2 Fault representativeness . 98 5.7.3 Fault injection performance . 99 5.8 Conclusion . 99 6 Conclusion 101 References 105 Summary 117 Samenvatting 119 List of Figures 2.1 Example function to demonstrate distortion . .9 2.2 Coverage in basic blocks as a function of the number of runs with -O4 optimization . 20 2.3 Coverage per program and workload generator with -O4 optimization . 22 2.4 Coverage per program and workload generator without optimization . 23 2.5 Log-log histograms of execution count (median over 50 runs) per basic block; the x-axis shows the number of times a block was executed and the y axis how many basic blocks have been executed that often . 28 2.6 Estimation of the distribution exponent; lines indicate standard errors . 29 2.7 Geometric mean of maximum execution count per basic block depend- ing on coverage . 31 2.8 Number of faults per basic block for each fault type, distinguishing whether blocks are covered by the workload and whether the program is optimized (O4) or not (O0); the numbers are an average over all pro- grams/workloads and the lines refer to standard errors . 32 3.1 Scheduling of fork resulting in different pids . 44 3.2 Histograms of fault activation and failure ratios; frequency refers to the total number of runs for all programs/benchmarks in that bracket . 55 4.1 Phases of our approach . 67 5.1 Fault injection design; IR=intermediate representation . 83 5.2 Traditional compilation (left) and LLVM with bitcode linking (right) . 85 5.3 Code example for basic block cloning . 86 xi xii LIST OF FIGURES 5.4 Control flow graph of the code example before (left) and after (right) fault injection . 86 5.5 Unixbench performance on Linux (higher is better) . 92 5.6 Unixbench performance on MINIX 3 (higher is better) . 92 5.7 Monte Carlo simulation of rebuilds needed for Linux with HSFI . 94 5.8 Monte Carlo simulation of rebuilds needed for MINIX 3 with HSFI . 94 5.9 Time taken per experiment depending on the workload duration for Linux; ts=test set, ub=Unixbench . 95 5.10 Time taken per experiment depending on the workload duration for MINIX 3; ts=test set, ub=Unixbench . 95 List of Tables 2.1 Fault types . 10 2.2 Classification of basic blocks in bzip2 . 25 3.1 Fault types . 42 3.2 Coverage of test programs . 49 3.3 Number of failures per program/workload . 51 3.4 Number of failures per fault type . 53 3.5 Number of failures per reachability class . 56 4.1 Fault types . 65 4.2 Workloads . 68 4.3 Coverage as % of fault candidates (fc) and lines of code (loc) . 71 4.4 Runtime with and without instrumentation . 72 4.5 Stability of systems per workload . 73 4.6 Fault types . 74 4.7 Step of first fault activation . 75 5.1 Code metrics for the target programs . 90 5.2 Boot time (lower is better, std. dev. in parentheses) . 91 5.3 Run time and overhead on MINIX 3 test set (lower is better, std. dev. in parentheses) . 92 5.4 Build time to prepare experiments in seconds (std. dev. in parentheses) 93 xiii Publications This dissertation consists of the following research papers, published in peer-reviewed journals and conferences (or submitted for review to such): Erik van der Kouwe, Cristiano Giuffrida, and Andrew S. Tanenbaum. Finding fault with fault injection: an empirical exploration of distortion in fault injection experiments1. In Software Quality Journal. Pages 1–30, 2014. Erik van der Kouwe, Cristiano Giuffrida, and Andrew S. Tanenbaum. On the Soundness of Silence: Investigating Silent Failures Using Fault Injection Experiments2. In Proceedings of the Tenth European Dependable Computing Conference (EDCC ’14). May 13-16, 2014, Newcastle upon Tyne, UK. Erik van der Kouwe, Cristiano Giuffrida, Razvan Ghitulete, and Andrew S. Tanenbaum. A Methodology to Efficiently Compare Operating System Stability3. In Proceedings of the 16th IEEE International Symposium on High-Assurance Systems Engineering (HASE ’15). January 8-10, 2015, Daytona Beach, FL, USA. Erik van der Kouwe and Andrew S. Tanenbaum. HSFI: representative fault injection scalable to large code bases4. Under review. 1Appears in Chapter 2. 2Appears in Chapter 3. 3Appears in Chapter 4. 4Appears in Chapter 5. xv xvi PUBLICATIONS The following publications have not been included in the thesis: Erik van der Kouwe, Cristiano Giuffrida, and Andrew S. Tanenbaum. Evaluating Distortion in Fault Injection Experiments. In Proceedings of the 15th IEEE Sym- posium on High-Assurance Systems Engineering (HASE ’14), January 9-11, 2014, Miami, FL, USA. Awarded Best Paper. Koustuba Bhat, Ben Gras, Erik van der Kouwe, Dirk Vogt, and Cristiano Giuffrida. Taking the ‘distributed’ out of distributed recovery. Under review. Chapter 1 1 Introduction Fault injection It is unlikely that anyone reading this thesis has never experienced a computer sys- tem failing. Almost certainly, you have experienced the situation where you lost a document you were working on when your operating system suddenly stopped and required a reboot to be able to continue. It is very likely that at some point you were trying to use an online service but it was unavailable and would not respond to your computer’s requests. There is a fair chance that at some point your hard drive just stopped working and you had to recover all your important files from a backup, as- suming you had one. In all these cases, the computer system you were using was not behaving as it was designed to do because some fault caused something unexpected to happen that the system was not able to deal with transparently. There are many different types of faults possible. In case of the operating system crash, the most likely possibility is that a programmer made a mistake while writing the program code of the operating system. Due to this mistake, the operating system ended up in a state where it was no longer able to provide the services required by applica- tions, causing the.