This document is downloaded from DR‑NTU (https://dr.ntu.edu.sg) Nanyang Technological University, Singapore. Securing software systems via fuzz testing and verification Chen, Hongxu 2019 Chen, H. (2019). Securing software systems via fuzz testing and verification. Doctoral thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/136764 https://doi.org/10.32657/10356/136764 This work is licensed under a Creative Commons Attribution‑NonCommercial 4.0 International License (CC BY‑NC 4.0). Downloaded on 29 Sep 2021 21:45:07 SGT SECURING SOFTWARE SYSTEMS VIA FUZZ TESTING AND VERIFICATION HONGXU CHEN SCHOOL OF COMPUTER SCIENCE AND ENGINEERING 2019 SECURING SOFTWARE SYSTEMS VIA FUZZ TESTING AND VERIFICATION HONGXU CHEN School of Computer Science and Engineering A thesis submitted to Nanyang Technological University in partial fulfillment of the requirement for the degree of Doctor of Philosophy 2019 Abstract Software security has been growing in importance due to the increasing reliance on various systems such as computers, smartphones, and Internet-of-Things devices. Security weaknesseses, or vulnerabilities, range in various aspects, from the systematic levels such as designing defects, to implementation errors such as language-specific program crashes. The multifaceted nature of vulnerabilities poses significant challenges to secure different systems. In fact, multilevel approaches have to be applied to reveal vulnerabilities or solidify the systems. Generally, there are two categories of approaches to securing a system, namely testing and verification. The former tries to generate test cases to trigger erroneous behaviors in the underlying programs. The latter models the system in an abstracted description to prove the absence of certain security defects. In this thesis, we will describe our efforts in greybox fuzz testing and type checking based verification in securing different levels of software systems. Firstly, we apply greybox fuzz testing (or fuzzing) to detect implementation vulnera- bilities. These vulnerabilities are usually subtle that it is rather complicated to formally verify the absence of the corresponding defects. We target on two levels of vulnera- bilities. One is to improve the fuzzing directedness of executing towards specific tar- get program locations to reveal crash induced vulnerabilities such as buffer-overflow, use-after-free, etc. To emphasize existing challenges in directed fuzzing, we propose HAWKEYE to feature four desired properties for directedness. With the statically an- alyzed results on the target program and its target locations, HAWKEYE applies mul- tiple dynamic strategies for seed prioritization, power scheduling and mutation. The experimental results on various real-world programs showed that HAWKEYE signifi- cantly outperforms the state-of-the-art greybox fuzzers AFL and AFLGo in reaching i Abstract ii target locations and reproducing specific crashes. The other scenario is to enhance fuzzing to reveal multithreading-relevant bugs. These bugs may root from logic er- rors which exhibit abnormal behaviors in the target program. They may or may not cause immediate crashes however implicate certain security defects. We present DOU- BLADE, a novel thread-aware technique which effectively generates multithreading- relevant seeds. DOUBLADE relies on a set of thread-aware techniques, consisting of a stratified exploration-oriented instrumentation and two complementary instrumenta- tions, as well as dynamic adaptive strategies based on feedbacks. Experiments on 12 real-world programs showed that DOUBLADE significantly outperforms the state-of- the-art fuzzer AFL in generating high-quality seeds, detecting vulnerabilities, and ex- posing concurrency bugs. To integrate HAWKEYE,DOUBLADE, and other recently proposed fuzzing techniques, we have built our own general-purpose fuzzing frame- work, Fuzzing Orchestration Toolkit (FOT). Compared to other fuzzing frameworks, FOT is versatile in its functionalities and can be easily configured or extended for vari- ous fuzzing purposes. Till now, FOT has detected 200+ zero-day vulnerabilities in more than 40 world-famous projects, among which 51 CVEs have been assigned. Secondly, we apply type checking based verification to detect those vulnerabilities introduced by designing defects where testing usually fails to reveal. Particularly, we focus on the information leakage vulnerability caused by inter-app communications on an Android system. We propose a lightweight security type system that models permission mechanism on Android, where the permissions are assigned statically and facilitated to enforce access control across applications. We proved the soundness of the proposed type system in terms of non-interference, with which we are able to prove the absence of information leakage among various apps in an Android system. We also presented a deterministic type inference algorithm for the underlying type system. Finally, based on these attempts we have made so far, we briefly discuss the differ- ences between the two categories of applied approaches in terms of the capability, the practicality, and their potential combinations in securing modern software systems. Statement of Originality I hereby certify that the work embodied in this thesis is the result of original research, is free of plagiarized materials, and has not been submitted for a higher degree to any other University or Institution. Jul/22/2019 .......................................... Date Hongxu Chen iii Supervisor Declaration Statement I have reviewed the content and presentation style of this thesis and declare it is free of plagiarism and of sufficient grammatical clarity to be examined. To the best of my knowledge, the research and writing are those of the candidate except as acknowledged in the Author Attribution Statement. I confirm that the investigations were conducted in accord with the ethics policies and integrity standards of Nanyang Technological University and that the research data are presented honestly and without prejudice. Jul/22/2019 .......................................... Date Yang Liu iv Authorship Attribution Statement This thesis contains materials from 3 published and 1 submitted papers in the following peer-reviewed conferences where I am the first author. Chapter3 is published as Hongxu Chen, Yinxing Xue, Yuekang Li, Bihuan Chen, Xiaofei Xie, Xiuheng Wu, Yang Liu, “Hawkeye: Towards a Desired Directed Greybox Fuzzer”, CCS ’18 Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, Pages 2095-2108, http://doi.acm.org/10.1145/ 3243734.3243849, DOI: 10.1145/3243734.3243849 [1]. The contribution of the co-authors are as follows: • I co-designed and implemented the methodology, wrote the manuscript draft, and conducted the experiments. • Yinxing Xue co-designed the methodology and revised the major part of the draft. • Yuekang Li co-designed the methodology and helped evaluate the experimental results. • Bihuan Chen co-designed the methodology and helped revise the draft. • Xiaofei Xie helped revise the draft. • Xiuheng Wu helped evaluate the experimental results. • Yang Liu co-designed the methodology and supported the research. Chapter4 is submitted as Hongxu Chen, Shengjian Guo, Yinxing Xue, Yulei Sui, Cen Zhang, Yuekang Li, Haijun Wang, Yang Liu, “DOUBLADE: Thread-aware Grey-box Fuzzing for Effective Bug Hunting in Multithreaded Programs”. The contribution of the co-authors are as follows: v Authorship Attribution Statement vi • I co-designed and implemented the methodology, wrote the manuscript draft, and conducted the experiments. • Shengjian Guo, Yinxing Xue, and Yulei Sui co-designed the methodology and revised the major parts of the draft. • Cen Zhang helped analyze the experimental results. • Yuekang Li and Haijun Wang helped revise the paper draft. • Yang Liu supported the research and revised the paper draft. Chapter5 is published as Hongxu Chen, Yuekang Li, Bihuan Chen, Yinxing Xue, Yang Liu, “FOT: A Versatile, Configurable, Extensible Fuzzing Framework”, ESEC/FSE 2018 Proceedings of the 2018 26th ACM Joint Meeting on European Software En- gineering Conference and Symposium on the Foundations of Software Engineering, Pages 867-870, http://doi.acm.org/10.1145/3236024.3264593, DOI: 10.1145/3236024.3264593 [2]. The contributions of the co-authors are as follows: • I and Yuekang Li implemented the core logic of the FOT fuzzing framework, and wrote the manuscript draft as well as conducted all the experiments. • Bihuan Chen helped revise the draft. • Yang Liu co-designed the architecture of the framework. • Yinxing Xue co-designed the HAWKEYE extension of the framework. Chapter6 is published as Hongxu Chen, Alwen Tiu, Zhiwu Xu, Yang Liu, “A Permission- Dependent Type System for Secure Information Flow Analysis”, 2018 IEEE 31st Com- puter Security Foundations Symposium (CSF), Oxford, 2018, pp. 218-232. https: //doi.org/10.1109/CSF.2018.00023, DOI: 10.1109/CSF.2018.00023 [3]. A technical report is available at https://arxiv.org/abs/1709.09623. The contribution of the co-authors are as follows: Authorship Attribution Statement vii • I co-designed the type system, proved the soundness of the type system with Alwen Tiu, implemented a prototype of the type system, and wrote the draft. • Alwen Tiu co-designed the type system, proved the soundness of the type system, and revised the paper draft. • Zhiwu Xu co-designed the type system, and designed the type inference rules for the proposed type system. • Yang Liu organized