IMPROVING QUALITY OF SOFTWARE WITH FOREIGN FUNCTION INTERFACES USING STATIC ANALYSIS by Siliang Li Presented to the Graduate and Research Committee of Lehigh University in Candidacy for the Degree of Doctor of Philosophy in Computer Engineering Lehigh University May 2014 Copyright page ii Approved and recommended for acceptance as a dissertation in partial fulfillment of the requirements for the degree of Doctor of Philosophy. Date Dr. Gang Tan (Dissertation Director) Accepted Date Committee Members: Dr. Gang Tan (Committee Chair) Dr. Mooi‐Choo Chuah (Committee Member) Dr. Brian D. Davison (Committee Member) Dr. Yu David Liu (Committee Member) iii DEDICATION I dedicate this dissertation to my loving parents Wang Xiuzhi and Li Dequan, without whom I would not be where I am today. They are the bedrock of my life. I am eternally grateful for their selfless love, understanding, support and patience. Thank you for my upbringing and teachings. Additionally, I dedicate this dissertation to my extended family in China, especially my Aunts. Throughout the years, they supported me and, most important of all, helped care for my parents in my absence, allowing me worry-free focus on my studies. Finally, I dedicate this work to all my friends whom have supported me. Special thanks to Yang Youliang, whom has always been there for me, cheered me through good times and bad. Thank you all with much love. iv ACKNOWLEDGEMENTS First and foremost, I would like to express my deepest appreciation and sincerest grati- tude to my dissertation advisor and mentor Dr. Gang Tan for taking a chance in me and taking me under his wings over six years ago when I first embarked upon this exciting and challenging journey. It was under Dr. Tans superb guidance, exemplary scholarship, and tireless dedication that profoundly inspired me to be a mature thinker and indepen- dent researcher. His unwavering support, constant encouragement and limitless patience propelled me forward to make this dissertation possible. Without him, I would not have achieved the goals for my dissertation. Secondly, I would like to thank my dissertation committee of Dr. Mooi-Choo Chuah, Dr. Brian D. Davison, and Dr. David Yu Liu for their insightful consultation and in- valuable advice during the past four years from my initial research ideas to its final implementation. At Lehigh University, Dr. Chuah was the first faculty member that I came to know and respect. Throughout the years, she gave me a great deal of support and guidance that helped me overcome difficult times. Dr. Davison has always inspired me and taught me to approach research problems with broader and more practical con- siderations. It was because of his indispensable inputs that made my dissertation more encompassing. I was also fortunate enough to have collaborated with Dr. Liu from State University of New York, Binghamton. I am truly grateful for his dedication and outstanding expertise on the subject, which made this dissertation complete and com- prehensive. My research would not be possible without all the committee members support. Additionally, I would like to thank the friendly and helpful staff at the Computer Science and Engineering Department and at the P.C. Rossin College of Engineering and Applied Science at Lehigh University. Their dedication and assistance made my v experience at Lehigh pleasant and smooth. Finally, I would like to thank my friends and colleagues at the SOS lab at Lehigh University. Their collective friendship and companionship gave me a great sense of support and community, making my dissertation journey rewarding and memorable. vi TABLE OF CONTENTS List of Tables x List of Figures xi Abstract 1 1 Introduction 2 1.1 Foreign Function Interface . 2 1.2 Software composed of FFIs . 3 1.3 Issues with FFIs and software quality . 4 1.3.1 Issues with FFIs . 4 1.3.2 Software quality issues . 7 1.4 Challenges and motivations . 9 1.5 Previous work and open problems . 10 1.6 Thesis statement . 12 1.7 Static analysis . 12 1.8 Overview . 13 1.8.1 Exception analysis in the JNI . 13 1.8.2 Atomicity enforcement in the JNI . 14 1.8.3 Reference count analysis in the Python/C interface . 14 1.9 Contributions . 15 2 Exception Analysis in the Java Native Interface 18 2.1 Introduction . 18 2.2 Background: the JNI . 21 2.2.1 How JNI exceptions can be thrown . 23 2.2.2 Checked exceptions vs. unchecked exceptions . 23 2.2.3 Interface code vs. library code . 24 2.3 Defining bug patterns . 24 2.3.1 Inconsistent exception declarations . 25 2.3.2 Mishandling JNI exceptions . 26 2.4 Overview of TurboJet . 30 2.5 Scalable and precise exception analysis . 32 2.5.1 Separating interface and library code . 33 2.5.2 Fine-grained tracking of exception states . 34 2.5.3 An FSM specification of exception-state transitions . 36 2.5.4 Path-sensitive analysis . 37 2.5.5 Context-sensitive analysis . 41 2.5.6 Transfer functions for JNI functions . 46 2.5.7 Merging symbolic states . 46 vii 2.6 Finding bugs of mishandling JNI exceptions . 46 2.6.1 Determining unsafe operations . 48 2.6.2 Warning recovery . 54 2.7 Prototype implementations and evaluation . 55 2.7.1 Accuracy . 57 2.7.2 Efficiency . 61 2.7.3 Comparison with previous studies . 63 2.8 An Eclipse plug-in tool . 66 2.9 Summary . 68 3 Native Code Atomicity for Java 69 3.1 Introduction . 69 3.2 Background and assumptions . 71 3.3 The formal model . 73 3.3.1 Abstract syntax . 73 3.3.2 Constraint generation: an overview . 75 3.3.3 Intraprocedural constraint generation . 77 3.3.4 Constraint closure . 81 3.3.5 Atomicity enforcement . 82 3.4 Prototype implementation . 85 3.5 Preliminary evaluation . 87 3.6 Summary . 91 4 Reference Counting in Python/C Programs with Affine Program Analysis 92 4.1 Introduction . 92 4.2 Background: the Python/C interface and reference counting . 95 4.2.1 Python/C reference counting and its complexities . 96 4.3 Pungi overview . 99 4.4 Affine abstraction . 101 4.4.1 Bug definition with non-escaping references . 101 4.4.2 SSA transform . 104 4.4.3 Affine translation . 106 4.4.4 Escaping references . 115 4.5 Affine analysis and bug reporting . 117 4.6 Implementation and limitations . 120 4.7 Evaluation . 121 4.8 Summary . 125 5 Related Work 126 5.1 FFIs . 126 5.2 Work related to TurboJet . 127 5.3 Work related to JATO . 128 5.4 Work related to Pungi . 130 viii 6 Future Work 133 7 Concluding Remarks 135 Bibliography 136 Appendix 148 A Whitelist 148 B Interprocedural exception analysis 149 Brief Biography 154 Curriculum Vitae 155 ix LIST OF TABLES 1.1 Popular real-world JNI applications. 4 1.2 Popular Python/C packages in the Fedora LINUX operating system. 5 2.1 Accuracy evaluation of TurboJet on finding inconsistent exception dec- larations. 55 2.2 Accuracy evaluation of TurboJet for finding inconsistent exception dec- larations. 56 2.3 Experimental results for assessing effectiveness of warning recovery and static taint analysis . 60 2.4 Efficiency evaluation of TurboJet. 62 2.5 Comparing TurboJet with an alternative exception analysis on finding inconsistent exception declarations. 65 2.6 Comparison with the previous study [85] (of the 35 errors in the previ- ous study, 11 are due to explicit throws and 24 due to implicit throws). 66 4.1 Statistics about selected benchmark programs. 122 4.2 All warnings reported by Pungi, which include true reference over- and under-counting errors and false positives. 123 4.3 Comparison of errors found between Pungi and CPyChecker. 124 x LIST OF FIGURES 1.1 A conceptual definition of an FFI. 2 2.1 A simple JNI example. This example also demonstrates how a native method can violate its exception declaration. 21 2.2 An example of mishandling JNI exceptions. 22 2.3 Two more examples of mishandling JNI exceptions. 29 2.4 System architecture of TurboJet. 31 2.5 Examples for illustrating the need for path and context sensitivity. 35 2.6 An incomplete FSM specification of exception-state transitions. 37 2.7 Syntax of Java types that TurboJet tracks. 39 2.8 Example of TurboJet path sensitivity. 42 2.9 Exception analysis using ESP. 43 2.10 Exception analysis using TurboJet. 44 2.11 High-level steps of warning generation for mishandling JNI exceptions. 47 2.12 An example program and its pointer graph. The program takes a Java integer array and computes the sum of all positive elements. The nodes with shading are tainted nodes. 53 2.13 A typical example of false positives. 59 2.14 An example of TurboJet plug-in’s warning on inconsistent exception declarations. 67 2.15 An example of TurboJet plug-in’s warning on mishandling JNI excep- tions. 67 3.1 A running example . 75 3.2 Java-Side Intraprocedual Constraint Generation . 78 3.3 Native-Side Intraprocedural Constraint Generation . 79 3.4 Class-Level Constraint Generation . 80 3.5 Execution time of the benchmark programs under different locking schemes. 90 4.1 An example Python/C extension module called ntuple (its registra- tion table and module initializer code are omitted). 95 4.2 An overview of Pungi. 100 4.3 A contrived example of a buggy Python/C function. 103 4.4 Part of the control-flow graph for the code in Fig. 4.1 after SSA. 105 4.5 Syntax of affine programs. 108 4.6 Affine translation T (−) for typical C constructs. ..