Research Overview (2006-2011) Koushik Sen

1 Research

A primary goal of my research is to improve software reliability and quality through the development of tools and techniques based on formal methods and programming language theory. In my research I have developed specification and testing techniques that use sophisticated program analysis and formal methods internally while providing a familiar usage model that daily programmers can readily use.

Motivation behind my research Today’s software systems suffer from poor reliability. NIST estimates that $60 billion is lost annually in the US due to software errors, and such errors in transportation, medicine, and other areas can put lives at risk. This indicates that our techniques to ensure software reliability are far from the level of maturity attained by other engineering disciplines that create critical infrastructure. The situation is likely to get worse, as the complexity of software systems increases without a matching increase in the effectiveness of software quality tools and techniques. Testing is the only predominant technique used by the software industry to make software reliable. Studies show that testing accounts for more than half of the total software development cost in industry. Although testing is a widely used and a well- established technique for building reliable software, existing techniques for testing are mostly ad hoc and ineffective— serious bugs are often exposed post-deployment. Our research aims to make real-world software specification and testing more usable, effective, scalable, and rigorous.

Concolic Testing and Directed Automated Random Testing Prior to 2006 (dissertation work), I worked on several novel approaches to improving software reliability; these approaches dramatically improve the testing and runtime monitoring of software, provide a method for quantifying the correctness of software, and enable the verification of software using machine learning. We proposed Concolic Testing (also called DART: Directed Automated Random Testing), an efficient way to automatically and systematically test programs. We showed how a combination of runtime and automated theorem proving techniques could generate non-redundant and exhaustive test inputs [PLDI’05, FSE’05, CAV’06a, FASE’06, HVC’06]. The success of concolic testing in scalably and exhaustively testing real-world software was a major milestone in the ad hoc world of and has inspired the development of several industrial and academic automated testing and security tools such as PEX, SAGE, YOGI, and Vigilante at Microsoft, Apollo at IBM, and SPLAT, BitBlaze, jFuzz, Oasis, and SmartFuzz in academia. (My two key papers on the subject have been cited over 1000 times and have won 3 awards—an ACM SIGSOFT Distinguished Paper Award at ESEC/FSE’05, a NSF CAREER Award in 2008, and a Haifa Verification Conference Award in 2009.) A central reason behind the wide adoption of concolic testing is that, while concolic testing uses program analysis and automated theorem proving techniques internally, it exposes a testing usage model that is familiar to most software developers. I further proposed combining concolic testing with formal specifications by using runtime monitors [FSE’03, ICSE’04, VMCAI’04, TACAS’04]. While concolic testing and runtime monitoring are suitable for finding bugs, they cannot guarantee the correctness of software. Thus, I proposed statistical model checking [CAV’04, CAV’05, QEST’05] and learning-to-verify [ICFEM’04, FSTTCS’04, TACAS’05] to quantify and prove the correctness of software. After 2006 (at Berkeley), I worked on a number of techniques to adapt concolic testing to larger programs and a wider variety of software and testing domains. The number of test inputs that need to be generated to exhaustively test large programs is usually astronomically large, and CUTE or jCUTE fails to generate all those inputs in reasonable time budget. I have worked on a variety of techniques [ISSTA’07, ICSE’07, ASE’07b, ISSTA’08, ASE’08b, SAS’08, ICSE’09b, ICSE’11, SAT’11] to make automated test generation scalable and exhaustive. We have proposed hybrid concolic testing [ICSE’07], a combination of random testing, a fast and non-exhaustive method of testing, with con- colic testing, an exhaustive and slow testing technique. We have also developed a novel strategy in concolic test gener- ation to quickly achieve high [ASE’08b]. In the strategy, test generation is guided by the static control flow graph of the program under test. The above two heuristic techniques have enabled us to scale concolic testing significantly. We have invented predictive testing to amplify the effectiveness of testing with manually generated test

1 cases by combining static and [FSE’07, ASE’08a]. We have extended concolic testing to test “Big O” runtime complexity of programs [ICSE’09b] and to automatically test database applications [ISSTA’07].

Specifying Correctness of Parallel Programs My research focus has shifted from sequential program correctness to that of parallel programs, as the spread of multicore and manycore processors and graphic processing units (GPUs) has greatly increased the need for parallel correctness tools. Reasoning about multithreaded programs is significantly more difficult than for sequential programs due to the nondeterministic interleaving of parallel threads. We believe that the only way to tackle this complexity is to find ways to separately specify, test, and reason about the correctness of a program’s use of parallelism and a program’s functional correctness. Driven by this belief, we have developed two fundamental techniques for separating the parallelization correctness aspect of a program from the program’s sequential functional correctness. Our first technique is based on the observation that a key simplifying assumption available to the developer of se- quential code is the determinism of the execution model. In contrast, parallel programs exhibit nondeterministic behav- ior due to the interleaving of parallel threads. This nondeterminism is essential to harness the power of parallel chips. But programmers often strive to preserve the determinism of their applications in the face of this nondeterminism—that is, to always compute the same output for the same input, no matter how the parallel threads of execution are sched- uled. We argue that programmers should be provided with a framework that will allow them to express the natural, intended deterministic behavior of parallel programs directly and easily. Thus, we proposed extending programming languages with constructs for writing specifications, called bridge assertions, that focus on relating outcomes of two parallel executions differing only in thread-interleavings [CACM’10, FSE’09, ICSE’10, ASPLOS’11]. We evaluated the utility of these assertions by manually adding deterministic specifications to a number of parallel Java benchmarks. We found it to be fairly easy to specify the correct deterministic behavior of the programs using our asser- tions, despite being unable in most cases to write traditional invariants or functional correctness assertions. Our papers on this work won an ACM SIGSOFT Distinguished Paper Award at [FSE’09] and the 2010 IFIP TC2 Manfred Paul Award for Excellence in Software: Theory and Practice at [ICSE’10]. Our research has been featured in the CACM’s Research Highlights [CACM’10]. Our second technique, in the same spirit of decomposing our correctness efforts, is suggested by the observation that manually parallelized programs often employ algorithms that are different than the sequential versions. In some cases, the algorithm itself is nondeterministic. Examples of such nondeterministic algorithms include branch-and- bound algorithms, which may produce multiple valid outputs for the same input. Such algorithmic nondeterminism is often tightly coupled with the functional correctness of the code, thus violating the premise of our determinism checking technique. To address these situations, we proposed to separate the reasoning about the algorithmic and the scheduler sources of nondeterminism. For such cases, we provide a specification technique in which a nonde- terministic sequential (NDSEQ) version of a parallel program is the specification for the program’s use of paral- lelism [HotPar’10, PLDI’11]. Using this technique, the programmer expresses the intended, algorithmic nondetermin- ism in his or her program—and the program’s parallelism is correct if the nondeterministic thread scheduling adds no additional program behaviors to those of the NDSEQ program. Testing, debugging, and reasoning about functional correctness can then be performed on the NDSEQ version, with controlled nondeterminism but with no interleaving of parallel threads. We developed an automatic technique [PLDI’11] to test parallelism correctness, by checking that an observed parallel execution obeys its nondeterministic sequential specification.

Active Testing of Concurrent Programs To check parallel correctness, we have developed active testing, a new scalable automated method for testing and debugging parallel programs. Active testing combines the power of imprecise program analysis with the precision of software testing to quickly discover concurrency bugs and to reproduce discovered bugs on demand [PLDI’08, FSE’08, PLDI’09, CAV’09, ICSE’09a, FSE’10a, FSE’10b, HotPar’11, TACAS’11, ISSTA’11, SC’11]. The key idea behind active testing is to control the thread scheduler in order to force the program into a state to expose a concurrency bug, e.g. data race, deadlock, atomicity violation, or violation of sequential memory consistency. The technique starts with lightweight inexpensive dynamic analysis that identifies situations where there is suspicion that a concurrency bug may exist. This first part of the analysis is imprecise because it trades-off precision for efficiency and it tries to increase the coverage of analysis by trying to predict potential bugs in other executions by analyzing a single execution. In the second step, a directed tester executes the program under a controlled thread schedule in an attempt to bring the

2 program in the buggy state. If it succeeds, it has identified a real concurrency bug; that is, the error report is guaranteed not be a false alarm, which is a serious problem with existing dynamic analyses. The actual method of controlling the thread schedule works as follows: once a thread reaches a state that resembles the desired state, it is paused as long as possible, giving a chance to other threads to catch up and complete the candidate buggy scenario. We have implemented active testing in extensible and publicly available tools for Java, C/PThreads, and UPC, and have applied these tools to find many previously-known and unknown concurrency bugs in a number of programs, including several real-world applications containing more than 600K lines of code. Like concolic testing, active testing is an effective and practical technique that exposes a simple testing usage model to programmers while using sophisticated program analysis under the hood. A paper on this work won an ACM SIGSOFT Distinguished Paper Award at [ICSE’09a]. Active testing is one of the core technologies behind the commercial tool Zeus Virtual Machine developed by Parallocity Inc., a bay area startup on concurrency bug finding. Zeus Virtual Machine works for C/C++ PThread programs. The company has closed their “Series A” funding in 2010. Customers of this company include Cisco, Broadcom, and Brocade.

A Framework for Programmable Failure-Injection and Testing of Cloud Systems We have developed a novel technique [HotDep’10, NSDI’11, FAST/WIP’11, OOPSLA’11] to systematically test the failure recovery behavior of large-scale cloud storage systems (e.g. HDFS, ZooKeeper, and Cassandra). In the recent years, cloud computing has become one of the most important component of large-scale computing. Cloud computing requires complex distributed software to guarantee that reliable and robust services can run uninterruptedly on thou- sands of cheap and unreliable hardware. Unfortunately, testing the reliability and correctness properties of such large systems has proven to be challenging in practice. We made an important observation in this problem, which helped to make failure testing effective for large-scale cloud systems. We observed that many times specifications are violated when multiple failures happen in a system. However, testing all possible combinations of multiple failures leads to combinatorial explosion. Therefore, a brute-force systematic testing approach cannot scale to large systems. We found that many failures are equivalent with respect to the behavior of the system after each failure. Therefore, instead of testing all combinations of failures, one could only test combinations of representative failures from each equivalence class. We then proposed to use code coverage to identify equivalent failures. This key insight of equivalence helped to reduce testing time from days to hours while finding almost all bugs that could be found using the brute-force approach. The testing technique found several previously unknown critical bugs in a commercially used software. Several companies, such as Cloudera, NetApp, and Twitter, have expressed interest in using this technology.

Future Work The goal of my current and future research is not only to build tools and techniques for improving software quality, but also to better understand the challenges that programmers face in writing and verifying parallel software. This insight will support my long-term research goals of developing language constructs and disciplines that not only help programmers avoid introducing common bugs in parallel software, but also facilitate effective and systematic testing, verification, and debugging.

List of Five Most Significant Publications

[1] Patrice Godefroid, Nils Klarlund, and Koushik Sen. DART: Directed Automated Random Testing. In ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation (PLDI’05), pages 213–223. 2005. [2] Koushik Sen, Darko Marinov, and Gul Agha. CUTE: A Concolic Engine for C. In 5th joint meeting of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE’05), pages 263–272. ACM, 2005. ACM SIGSOFT Distinguished Paper Award.

[3] Koushik Sen. Race Directed Random Testing of Concurrent Programs. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’08), pages 11–21. ACM, 2008.

3 [4] Jacob Burnim and Koushik Sen. Asserting and Checking Determinism for Multithreaded Programs. Communi- cations of the ACM, 53(6):97–105, June 2010. ACM SIGSOFT Distinguished Paper Award for the FSE version of this CACM paper.

[5] Jacob Burnim, Tayfun Elmas, George Necula, and Koushik Sen. NDSeq: Runtime Checking for Nondeterminis- tic Sequential Specifications of Parallel Correctness. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’11), pages 401–414. ACM, 2011.

References

[PLDI’05] Patrice Godefroid, Nils Klarlund, and Koushik Sen. DART: Directed Automated Random Test- ing. In ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation (PLDI’05), pages 213–223. 2005. [FSE’05] Koushik Sen, Darko Marinov, and Gul Agha. CUTE: A Concolic Unit Testing Engine for C. In 5th joint meeting of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE’05), pages 263–272. ACM, 2005. ACM SIGSOFT Distinguished Paper Award. [CAV’06a] Koushik Sen and Gul Agha. CUTE and jCUTE : Concolic Unit Testing and Explicit Path Model- Checking Tools. In 18th International Conference on Computer Aided Verification (CAV’06), volume 4144 of Lecture Notes in Computer Science, pages 419–423. Springer, 2006.

[FASE’06] Koushik Sen and Gul Agha. Automated Systematic Testing of Open Distributed Programs. In In- ternational Conference on Fundamental Approaches to Software Engineering (FASE’06) (ETAPS’06 conference), volume 3922 of Lecture Notes in Computer Science, pages 339–356. Springer, 2006. [HVC’06] Koushik Sen and Gul Agha. A Race-Detection and Flipping Algorithm for Automated Testing of Multi-Threaded Programs. In Haifa verification conference 2006 (HVC’06), volume 4383 of Lecture Notes in Computer Science, pages 166–182. Springer, 2006. [FSE’03] Koushik Sen, Grigore Ros¸u, and Gul Agha. Runtime Safety Analysis of Multithreaded Programs. In 9th European Software Engineering Conference and 11th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (ESEC/FSE’03), pages 337–346. ACM, 2003.

[ICSE’04] Koushik Sen, Abhay Vardhan, Gul Agha, , and Grigore Ros¸u. Efficient Decentralized Monitoring of Safety in Distributed Systems. In 26th International Conference on Software Engineering (ICSE’04), pages 418–427. IEEE, 2004. [VMCAI’04] Howard Barringer, Allen Goldberg, Klaus Havelund, and Koushik Sen. Rule-Based Runtime Verifi- cation. In 5th International Conference on Verification, Model Checking and Abstract Interpretation (VMCAI’04), volume 2937 of Lecture Notes in Computer Science, pages 44–57. Springer, 2004. [TACAS’04] Koushik Sen, Grigore Ros¸u, and Gul Agha. Online Efficient Predictive Safety Analysis of Multi- threaded Programs. In 10th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’04), volume 2988 of Lecture Notes in Computer Science, pages 123–138. 2004.

[CAV’04] Koushik Sen, Mahesh Viswanathan, and Gul Agha. Statistical Model Checking of Black-Box Proba- bilistic Systems. In 16th International Conference on Computer Aided Verification (CAV’04), volume 3114 of Lecture Notes in Computer Science, pages 202–215. Springer, 2004. [CAV’05] Koushik Sen, Mahesh Viswanathan, and Gul Agha. On Statistical Model Checking of Stochastic Systems. In 17th International Conference on Computer Aided Verification (CAV’05), volume 3576 of Lecture Notes in Computer Science, pages 266–280. Springer, 2005.

4 [QEST’05] Koushik Sen, Mahesh Viswanathan, and Gul Agha. VESTA: A Statistical Model Checker and An- alyzer for Probabilistic Systems. In 2nd International Conference on Quantitative Evaluation of Systems (QEST’05), pages 251–252. IEEE, 2005. [ICFEM’04] Abhay Vardhan, Koushik Sen, Mahesh Viswanathan, and Gul Agha. Learning to Verify Safety Properties. In 6th International Conference on Formal Engineering Methods (ICFEM’04), volume 3308 of Lecture Notes in Computer Science, pages 274–289. Springer, 2004. [FSTTCS’04] Abhay Vardhan, Koushik Sen, Mahesh Viswanathan, and Gul Agha. Actively Learning to Verify Safety for FIFO Automata. In 24th Conference on Foundations of Software Technology and Theo- retical Computer Science (FSTTCS’04), volume 3328 of Lecture Notes in Computer Science, pages 494–505. Springer, 2004. [TACAS’05] Abhay Vardhan, Koushik Sen, Mahesh Viswanathan, and Gul Agha. Using Language Inference to Verify omega-regular Properties. In 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’05), volume 3440 of Lecture Notes in Computer Science, pages 45–60. Springer, 2005. [ISSTA’07] Michael Emmi, Rupak Majumdar, and Koushik Sen. Dynamic Test Input Generation for Database Applications. In International Symposium on Software Testing and Analysis (ISSTA’07), pages 151– 162. ACM, 2007. [ICSE’07] Rupak Majumdar and Koushik Sen. Hybrid Concolic Testing. In 29th International Conference on Software Engineering (ICSE’07), pages 416–426. IEEE, 2007. [ASE’07b] Koushik Sen. Concolic Testing. In 22nd IEEE/ACM nternational Conference on Automated Software Engineering (ASE’07), pages 571–572. IEEE, 2007. [ISSTA’08] Yamini Kannan and Koushik Sen. Universal Symbolic Execution and its Application to Likely Data Structure Invariant Generation. In International Symposium on Software Testing and Analysis (ISSTA’08), pages 283–294. ACM, 2008. [ASE’08b] Jacob Burnim and Koushik Sen. Heuristics for Scalable Dynamic Test Generation. In 23rd IEEE/ACM nternational Conference on Automated Software Engineering (ASE’08), pages 443–446. IEEE, 2008. [SAS’08] Murali Krishna Ramanathan, Koushik Sen, Ananth Grama, and Suresh Jagannathan. Protocol Infer- ence Using Static Path Profiles. In 15th International Static Analysis Symposium (SAS’08), volume 5079 of Lecture Notes in Computer Science, pages 78–92. Springer, 2008. [ICSE’09b] Jacob Burnim, Sudeep Juvekar, and Koushik Sen. WISE: Automated Test Generation for Worst-Case Complexity. In 31st International Conference on Software Engineering (ICSE’09), pages 463–473. IEEE, 2009. [ICSE’11] Cristian Cadar, Patrice Godefroid, Sarfraz Khurshid, Corina Pasareanu, Koushik Sen, Nikolai Till- mann, and Willem Visser. Symbolic Execution for Software Testing in Practice – Preliminary As- sessment. In Impact Project Focus Area in 33rd International Conference on Software Engineering (ICSE’11), pages 1066–1071. IEEE, 2011. [SAT’11] Koushik Sen. Concolic Testing and Constraint Satisfaction. In 14th International Conference on Theory and Applications of Satisfiability Testing (SAT’11). 2011. Invited Talk. [FSE’07] Pallavi Joshi, Koushik Sen, and Mark Shlimovich. Predictive Testing: Amplifying the Effective- ness of Software Testing (Poster Paper). In 6th joint meeting of the European Software Engi- neering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE’07), pages 561–564. ACM, 2007. [ASE’08a] Pallavi Joshi and Koushik Sen. Predictive Typestate Checking of Multithreaded Java Programs. In 23rd IEEE/ACM nternational Conference on Automated Software Engineering (ASE’08), pages 288–296. IEEE, 2008.

5 [CACM’10] Jacob Burnim and Koushik Sen. Asserting and Checking Determinism for Multithreaded Programs. Communications of the ACM, 53(6):97–105, June 2010. [FSE’09] Jacob Burnim and Koushik Sen. Asserting and Checking Determinism for Multithreaded Programs. In 7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE’09), pages 3–12. ACM, 2009. ACM SIGSOFT Distinguished Paper Award. Invited to CACM Research Highlights. [ICSE’10] Jacob Burnim and Koushik Sen. DETERMIN: Inferring Likely Deterministic Specifications of Mul- tithreaded Programs. In 32nd International Conference on Software Engineering (ICSE’10). IEEE, 2010. IFIP TC2 Manfred Paul Award for Excellence in Software: Theory and Practice. [ASPLOS’11] Jacob Burnim, George Necula, and Koushik Sen. Specifying and Checking Semantic Atomicity for Multithreaded Programs. In 16th International Conference on Architectural Support for Program- ming Languages and Operating Systems (ASPLOS’11), pages 79–90. 2011. [HotPar’10] Jacob Burnim, George Necula, , and Koushik Sen. Separating Functional and Parallel Correctness using Nondeterministic Sequential Specifications. In 2nd USENIX Workshop on Hot Topics in Par- allelism (HotPar’10). 2010. [PLDI’11] Jacob Burnim, Tayfun Elmas, George Necula, and Koushik Sen. NDSeq: Runtime Checking for Nondeterministic Sequential Specifications of Parallel Correctness. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’11), pages 401–414. ACM, 2011. [PLDI’08] Koushik Sen. Race Directed Random Testing of Concurrent Programs. In ACM SIGPLAN Confer- ence on Programming Language Design and Implementation (PLDI’08), pages 11–21. ACM, 2008. [FSE’08] Chang-Seo Park and Koushik Sen. Randomized Active Atomicity Violation Detection in Concurrent Programs. In 16th International Symposium on Foundations of Software Engineering (FSE’08), pages 135–145. ACM, 2008. [PLDI’09] Pallavi Joshi, Chang-Seo Park, Koushik Sen, and Mayur Naik. A Randomized Dynamic Program Analysis Technique for Detecting Real Deadlocks. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’09), pages 110–120. ACM, 2009. [CAV’09] Pallavi Joshi, Mayur Naik, Chang-Seo Park, and Koushik Sen. An Extensible Active Testing Frame- work for Concurrent Programs. In 21st International Conference on Computer Aided Verification (CAV’09), volume 5643 of Lecture Notes in Computer Science, pages 675–681. Springer, 2009. [ICSE’09a] Mayur Naik, Chang-Seo Park, Koushik Sen, and David Gay. Effective Static Deadlock Detection. In 31st International Conference on Software Engineering (ICSE’09), pages 386–396. IEEE, 2009. ACM SIGSOFT Distinguished Paper Award. [FSE’10a] Pallavi Joshi, Mayur Naik, Koushik Sen, and David Gay. An Effective Dynamic Analysis for De- tecting Generalized Deadlocks. In ACM SIGSOFT Eighteenth Symposium on the Foundations of Software Engineering (FSE’10). ACM, 2010. [FSE’10b] Nicholas Jalbert and Koushik Sen. A Trace Simplification Technique for Effective Debugging of Concurrent Programs. In ACM SIGSOFT Eighteenth Symposium on the Foundations of Software Engineering (FSE’10). ACM, 2010. [HotPar’11] Nicholas Jalbert, Cristiano Pereira, Gilles Pokam, and Koushik Sen. RADBench: A Concurrency Bug Benchmark Suite. In 3rd USENIX Workshop on Hot Topics in Parallelism (HotPar’11). 2011. [TACAS’11] Jacob Burnim, Koushik Sen, and Christos Stergiou. Sound and Complete Monitoring of Sequential Consistency for Relaxed Memory Models. In International Conference on Tools and Algorithms for the Construction ana Analysis of Systems (TACAS’11), pages 11–25. 2011. [ISSTA’11] Jacob Burnim, Koushik Sen, and Christos Stergiou. Testing Concurrent Programs on Relaxed Mem- ory Models. In International Symposium on Software Testing and Analysis (ISSTA’11). ACM, 2011.

6 [SC’11] Chang-Seo Park, Paul Hargrove, Costin Iancu, and Koushik Sen. Efficient Data Race Detection for Distributed Memory Parallel Programs. In International Conference for High Performance Comput- ing, Networking, Storage and Analysis (SC’11). 2011. [HotDep’10] Haryadi S. Gunawi, Thanh Do, Pallavi Joshi, Joseph M. Hellerstein, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Koushik Sen. Towards Automatically Checking Thousands of Fail- ures with Micro-specifications. In 6th USENIX Workshop on Hot Topics in System Dependability (HotDep’10). 2010. [NSDI’11] Haryadi S. Gunawi, Thanh Do, Pallavi Joshi, Peter Alvaro, Joseph M. Hellerstein, Andrea C. Arpaci- Dusseau, Remzi H. Arpaci-Dusseau, Koushik Sen, and Dhruba Borthakur. FATE and DESTINI: A Framework for Cloud Recovery Testing. In 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI’11). 2011.

[FAST/WIP’11] Pallavi Joshi, Haryadi S. Gunawi, and Koushik Sen. PreFail: Programmable and Efficient Failure Testing Framework. In Poster Sessions and Work-in-Progress Reports in 9th USENIX Conference on File and Storage Technologies (FAST/WIP’11). 2011. [OOPSLA’11] Pallavi Joshi, Haryadi S. Gunawi, and Koushik Sen. PREFAIL: A Programmable Tool for Multiple- Failure Injection. In Annual Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA’11). 2011.

7