Machine Learning for Software Dependability by Thibaud Lutellier A thesis presented to the University of Waterloo in fulfillment of the thesis requirement for the degree of Doctor of Philosophy in Electrical and Computer Engineering Waterloo, Ontario, Canada, 2020 © Thibaud Lutellier 2020 Examining Committee Membership The following served on the Examining Committee for this thesis. The decision of the Examining Committee is by majority vote. External Examiner: Martin Monperrus Professor, Division of Theoretical Computer Science, KTH Royal Institute of Technology Supervisor: Lin Tan Professor, Electrical & Computer Engineering, University of Waterloo Internal Member: Vijay Ganesh Professor, Electrical & Computer Engineering, University of Waterloo Derek Rayside Professor, Electrical & Computer Engineering, University of Waterloo Internal-External Member: Michael Godfrey Professor, Dept. of Computer Science, University of Waterloo ii Author’s Declaration This thesis consists of material all of which I authored or co-authored: see Statement of Contributions included in the thesis. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners. I understand that my thesis may be made electronically available to the public. iii Statement of Contributions As lead author of all the contributions of this thesis, I was responsible for contributing to conceptualizing study design, developing, implementing, and evaluating prototypes and evaluating them, carrying out data collection and analysis, drafting and submitting manuscripts, and presenting the work to conferences. My coauthors provided feedback during each step of the research and on draft manuscripts. Research presented in Chapter 2: Lawrence Pang helped implementing the context- aware model, Hung Viet Pham and Yitong Li helped with evaluating previous automatic program repair tools for C and C++. Moshi Wei helped in gathering training data. Dr. Lin Tan provided feedback and supervision in all parts of the research. Research presented in Chapter 3: Dr. Tomasz Kuchta helped to design and collect the regular expressions for the clustering. Dr. Edmund Wong and Dr. Tomasz Kuchta helped manually checking PDF files for inconsistencies. Dr. Christian Cadar and Dr. Lin Tan provided feedback and supervision in all parts of the research. Research presented in Chapter 4: Devin Chollak helped gathering the dependencies and ground truth of Java projects. Dr. Joshua Garcia provided access and support with the Arcade framework that implements most architecture recovery techniques. Dr. Robert Kroeger helped validating the ground truth of Chromium. Dr. Lin Tan, Dr. Nenad Medvidovic, and Dr. Derek Rayside provided feedback and supervision in all parts of the research. Citations: • Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan, CoCoNuT: Combining Context-Aware Neural Translation Models Using Ensemble for Program Repair, Proceeding of The ACM SIGSOFT International Symposium on Software Testing and Analysis, 2020 [157]. • Tomasz Kuchta*, Thibaud Lutellier*, Edmund Wong, Lin Tan, and Cristian Cadar, On the Correctness of Electronic Documents: Studying, Finding, and Localizing Inconsistency Bugs in PDF Readers and Files, (* The first two authors contributed equally to this paper), Empirical Software Engineering, 2018 [118]. • Thibaud Lutellier, Devin Chollak, Joshua Garcia, Lin Tan, Derek Rayside, Nenad Medvidovic, and Robert Kroeger, Measuring the Impact of Code Dependencies on Soft- ware Architecture Recovery Techniques, IEEE Transactions on Software Engineering, 2017 [156]. iv • Thibaud Lutellier, Devin Chollack, Joshua Garcia, Lin Tan, Derek Rayside, Ne- nad Medvidovic and Robert Kroeger, Comparing Software Architecture Recovery Techniques Using Accurate Dependencies, Proceeding of ACM/IEEE International Conference on Software Engineering, SEIP track, 2015 [154]. v Abstract Dependability is an important quality of modern software but is challenging to achieve. Many software dependability techniques have been proposed to help developers improve software reliability and dependability such as defect prediction [83,96,249], bug detection [6, 17, 146], program repair [51, 127, 150, 209, 261, 263], test case prioritization [152, 250], or software architecture recovery [13,42,67,111,164,240]. In this thesis we consider how machine learning (ML) and deep-learning (DL) can be used to enhanced software dependability through three examples in three different domains: automatic program repair, bug detection in electronic document readers, and software architecture recovery. In the first work we propose a new G&V technique—CoCoNuT, which uses ensemble learning on the combination of convolutional neural networks (CNNs) and a new context- aware neural machine translation (NMT) architecture to automatically fix bugs in multiple programming languages. To better represent the context of a bug, we introduce a new context-aware NMT architecture that represents the buggy source code and its surrounding context separately. CoCoNuT uses CNNs instead of recurrent neural networks (RNNs), since CNN layers can be stacked to extract hierarchical features and better model source code at different granularity levels (e.g., statements and functions). In addition, CoCoNuT takes advantage of the randomness in hyperparameter tuning to build multiple models that fix different bugs and combines these models using ensemble learning to fix more bugs. CoCoNuT fixes 493 bugs, including 307 bugs that are fixed by none of the 27 techniques with which we compare. In the second work, we present a study on the correctness of PDF documents and readers and propose an approach to detect and localize the source of such inconsistencies automatically. We evaluate our automatic approach on a large corpus of over 230K documents using 11 popular readers and our experiments have detected 30 unique bugs in these readers and files. In the third work we compare software architecture recovery techniques to understand their effectiveness and applicability. Specifically, we study the impact of leveraging accurate symbol dependencies on the accuracy of architecture recovery techniques. In addition, we evaluate other factors of the input dependencies such as the level of granularity and the dynamic-bindings graph construction. The results of our evaluation of nine architecture recovery techniques and their variants suggest that (1) using accurate symbol dependencies has a major influence on recovery quality, and (2) more accurate recovery techniques are needed. Our results show that some of the studied architecture recovery techniques scale to very large systems, whereas others do not. vi Acknowledgements I would like to particularly thank my supervisor, Dr. Lin Tan: thank you for taking me as a student, helping and supporting me during all these years. I learned and grew so much under her supervision and mentorship, I could not have wished for a better supervisor! I would like to thank my examination committee members—Dr. Martin Monperrus, Dr. Thomas Zimmermann, Dr. Sebastian Fischmeister, and Dr. Vijay Ganesh, Dr. Derek Rayside, and Dr. Michael Godfrey for their support, valuable feedback, and insightful discussion on my research. Thanks to all the students in our research team at the University of Waterloo who made going to the lab every day a pleasure: Dr. Jaechang Nam, Dr. Jinqiu Yang, Dr. Edmund Wong, Dr. Song Wang, Dr. Nasir Ali, Lei Zhang, Quinn Hanam, Sandeep Chaudhary, Devin Chollak, Ming Tan, Michael Chong, Taiyue Liu, Yuefei Liu, Alexey Zhikhartsev, Yuan Xi, Moshi Wei, Hung Pham, and Yitong Li—thank you very much for all the friendship, support, and joy. I also want to thank the team at Purdue University, while the opportunities to meet in person were rare, we had great online conversations and I learned a lot from them: Nan Jiang, Shangshu Qian, Jonathan Rosenthal, Jiannan Wang, Danning Xie, and Dr. Mijung Kim. I want to thank all my other collaborators during my Ph.D. study: Dr. Joshua Garcia, Dr. Nenad Medvidovic, Dr. Robert Kroeger, Dr. Tomasz Kuchta, Dr. Cristian Cadar, Lawrence Pang, Dr. Yaoliang Yu, Dr. Nachiappan Nagappan, Weizhen Qi, Dr. Hossain Shahriar, Dr. Komminist Weldemariam, and Dr. Mohammad Zulkernine. I really enjoyed working with them, and have learned a lot from them. vii Dedication This is dedicated to Sarah. You supported me through all these years, this work is also here thanks to you. viii Table of Contents List of Figures xiv List of Tables xvi 1 Introduction1 1.1 Project 1 – Combining Context-Aware Neural Translation Models using Ensemble for Automated Program Repair:..................2 1.2 Project 2 – On the Correctness of Electronic Documents: Studying, Finding, and Localizing Inconsistency Bugs in PDF Readers and Files:.......2 1.3 Project 3 – Measuring the Impact of Dependencies on Software Architecture Recovery Techniques:..............................3 1.4 Publications...................................4 2 Related Work6 2.1 Neural Machine Translation Models for APR.................6 2.1.1 Deep Learning for Automatic Program Repair:............6 2.1.2 G&V Program Repair:.........................7 2.1.3 Grammatical Error Correction (GEC):................7 2.1.4 Deep Learning in Software Engineering:................7 2.2 Finding, and Localizing Inconsistency Bugs in PDF Readers and Files...7 2.2.1 Cross PDF Reader Inconsistency:...................7 2.2.2 Cross Browser Inconsistency:......................8 ix 2.2.3 Vulnerabilities in PDF files:......................9