Vulnerability Detection on Web Browsers
Total Page:16
File Type:pdf, Size:1020Kb
This document is downloaded from DR‑NTU (https://dr.ntu.edu.sg) Nanyang Technological University, Singapore. Vulnerability detection on web browsers Yu, Haiwan 2019 Yu, H. (2019). Vulnerability detection on web browsers. Doctoral thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/143063 https://doi.org/10.32657/10356/143063 This work is licensed under a Creative Commons Attribution‑NonCommercial 4.0 International License (CC BY‑NC 4.0). Downloaded on 29 Sep 2021 06:20:20 SGT Vulnerability Detection on Web Browsers Yu Haiwan SCHOOL OF PHYSICAL AND MATHEMATICAL SCIENCES 2019 Vulnerability Detection on Web Browsers Yu Haiwan SCHOOL OF PHYSICAL AND MATHEMATICAL SCIENCES A thesis submitted to the Nanyang Technological University in partial fulfilment of the requirement for the degree of Doctor of Philosophy 2019 Statement of Originality I hereby certify that the work embodied in this thesis is the result of original research done by me except where otherwise stated in this thesis. The thesis work has not been submitted for a degree or professional qualification to any other university or institution. I declare that this thesis is written by myself and is free of plagiarism and of sufficient grammatical clarity to be examined. I confirm that the investigations were conducted in accord with the ethics policies and integrity standards of Nanyang Technological University and that the research data are presented honestly and without prejudice. 14-Aug-2019 ............................................ Date Yu Haiwan Supervisor Declaration Statement I have reviewed the content and presentation style of this thesis and declare it is free of plagiarism and of sufficient grammatical clarity to be examined. To the best of my knowledge, the research and writing are those of the candidate except as acknowledged in the Author Attribution Statement. I confirm that the investigations were conducted in accord with the ethics policies and integrity standards of Nanyang Technological University and that the research data are presented honestly and without prejudice. ............................................ Date Assoc Prof. Wu HongJun Authorship Attribution Statement This thesis does not contain any materials from papers published in peer-reviewed journals or from papers accepted at conferences in which I am listed as an author. 14-Aug-2019 ............................................ Date Yu Haiwan Abstract Web browser is the most commonly used software to access the Internet. Any vulnerability in a popular web browser will compromise the security and privacy of its users. With the increasing complexity of modern web browsers, the attack surface of web browser increased dramatically and more vulnerabilities were intro- duced. In this thesis, we developed a fuzzing framework to detect vulnerability in the web browser. Our fuzzing framework is designed for large scale fuzzing of all the popular web browsers running on virtual machines. Our fuzzing framework sup- ports fuzzing with multiple test case generation strategies in a test case generator set, and test case generation strategies can be changed when fuzzer is running. By running this fuzzing framework together with our various vulnerability detec- tion methods, many crashes were found and in total 5 CVEs were assigned to the vulnerabilities being found. In this thesis, we proposed a new type of vulnerability, namely, the memory pressure bugs. This type of vulnerability was triggered by failed memory allocation. Using the existing fuzzing methods, it is extremely hard to trigger this type of bugs. It is also extremely difficult to reproduce this type of bugs since reproducing the crashes requires the identical memory allocation to be failed, while the memory allocations in computer system are hard to predict in general. To trigger this type of bugs, we developed low memory simulation instrumentation tools to aid our fuzzer to detect memory pressure bugs in web browsers. To reproduce this type of bugs, we introduced precise memory pressure in JavaScript code. We solve the problem of premature allocation failure of memory pressure bug by leveraging on the memory fragmentation to reserve memory space for allocation before the target allocation. Three new vulnerabilities of memory pressure bugs were successfully found in the Internet explorer. ix x In this Thesis, we analyzed 5 CVEs we have found and a zero-day vulnerability in Internet explorer. We exclusively disclose the details of these 6 vulnerabilities and proof of concept (POC) to trigger them. Acknowledgements I wish to express my greatest gratitude to my supervisor Prof. Wu HongJun who guided me through four years of my PhD Candidacy. I appreciate his guidance that gives me precious experience in computer security and cryptography. I would like to thank Dr Wei Lei and Dr Wang ChenYu for their guidance on fuzzing and crash analysis. They provide helpful advice to solve problems I have encountered in my research. I will also like to thank Dr Huang Tao and Peng Lunan for help on my research projects. I would like to thank the anonymous examiners of this thesis for their valuable time spent and the comments they provide. Furthermore, I will like to thank my parents who support me both mentally and financially that helps me to overcome the difficulties in my PhD study. Lastly, I would like to thank my wife Li Jingxi, for her sacrifice when I am busy with my research. xi Contents Abstract ix Acknowledgements xi List of Figures xvii Symbols and Acronyms xix 1 Introduction1 1.1 Vulnerability..............................1 1.1.1 Stack Buffer Overflow.....................2 1.1.2 Heap vulnerabilities.......................3 1.1.2.1 Heap out of bound access..............4 1.1.2.2 Use After Free....................4 1.1.2.3 Type confusion....................5 1.1.2.4 Uninitialized memory access.............5 1.2 Mitigations...............................6 1.2.1 StackGuard...........................6 1.2.2 DEP...............................6 1.2.3 Address Space Layout Randomization.............7 1.2.4 Control-flow Integrity......................8 1.3 Exploitation...............................8 1.3.1 Info leak.............................8 1.3.2 Virtual table pointer corruption................9 1.3.3 Return oriented programming................. 10 1.3.4 Heap Spray........................... 10 1.4 Vulnerability Detection......................... 12 1.4.1 Static Analysis......................... 13 1.4.2 Dynamic Analysis........................ 13 1.4.3 Fuzzing............................. 14 1.4.4 Error detecting tools...................... 15 1.5 Thesis Organization........................... 15 1.5.1 Thesis Statement........................ 16 xiii xiv CONTENTS 2 Browsers and their vulnerabilities 17 2.1 Introduction............................... 17 2.2 Browser Structure............................ 18 2.2.1 HTML and Document Object Model............. 18 2.2.2 JavaScript Engine........................ 20 2.2.3 CSS............................... 21 2.3 Browser vulnerabilities and exploit.................. 22 2.3.1 Use-After-Free.......................... 22 2.3.2 Arbitrary read and write.................... 22 2.4 JIT.................................... 23 2.5 Browser mitigations........................... 23 2.5.1 MemGC and Isolated Heap.................. 23 2.6 Browser Vulnerability Discovery.................... 24 3 Generation-based Browser Fuzzer 27 3.1 Introduction............................... 27 3.1.1 Browser Vulnerability Detection................ 27 3.1.1.1 Mutation-based fuzzing............... 28 3.1.1.2 Generation-based fuzzing.............. 28 3.1.2 Motivation............................ 29 3.2 Fuzzing framework........................... 30 3.2.1 Test case generation...................... 30 3.2.1.1 Grammar and Vocabulary.............. 32 3.2.1.2 General purpose DOM................ 34 3.2.1.3 Internet Explorer DOM............... 35 3.2.1.4 JavaScript Engine.................. 36 3.2.2 Fuzzing Server.......................... 37 3.2.3 Error Detection and Reporting................ 38 3.2.4 Crash Archive.......................... 39 3.2.5 Crash POC Minimizer..................... 40 3.3 Implementation............................. 41 3.3.1 Internet Explorer........................ 41 3.3.2 Microsoft Edge......................... 42 3.3.3 Webkit.............................. 43 3.3.4 Chromium and Firefox..................... 44 3.4 Result.................................. 45 4 Memory Pressure Bug 47 4.1 Introduction............................... 47 4.1.1 Potential target for memory pressure bug........... 48 4.1.2 Example of memory pressure bug............... 49 4.1.3 Challenges............................ 50 4.2 Our Approach.............................. 50 CONTENTS xv 4.2.1 Memory pressure instrumentation............... 51 4.2.1.1 PIN Instrumentation................. 52 4.2.1.2 Address Sanitizer................... 53 4.2.2 Minimizing POC........................ 55 4.2.3 Pressurization of the heap................... 57 4.2.4 Impact of allocation size.................... 59 4.2.4.1 Location of target allocation............. 60 4.2.4.2 Controllable allocation................ 61 4.2.4.3 Uncontrollable allocation............... 61 4.3 Implementation and result....................... 64 4.3.1 Open source browsers...................... 64 4.3.2 Internet Explorer........................ 66 4.3.3 Memory Pressurization on Internet Explorer......... 66 4.3.4 Result and Evaluation..................... 67 4.4 Future works and conclusion.....................