“Web Protocol Fuzzing of the TLS/SSL Protocol with Focus on the OpenSSL Library” by Fouad Nouri Khalaf School of Information Technology and Electrical Engineering, University of Queensland. Submitted for the degree of Bachelor of Engineering (Honours) in the division of Software Engineering. 9th November 2020 i Fouad Nouri Khalaf [email protected] 9th November 2020 Prof Amin Abbosh Acting Head of School School of Information Technology and Electrical Engineering The University of Queensland St Lucia QLD 4072 Dear Professor Abbosh, In accordance with the requirements of the degree of Bachelor of Engineering (Honours) in the School of Information Technology and Electrical Engineering, I submit the following thesis entitled “Web Protocol Fuzzing of the TLS/SSL Protocol with Focus on the OpenSSL Library” The thesis was performed under the supervision of senior lecturer Dr Guangdong Bai. I declare that the work submitted in the thesis is my own, except as acknowledged in the text and footnotes, and that it has not previously been submitted for a degree at the University of Queensland or any other institution. Yours sincerely Fouad Nouri Khalaf ii iii iv Abstract Fuzzing is a testing mechanism that uses randomly generated data as input for functions to be tested with, detecting any unexpected behaviour in a program. To properly fuzz an application can be a challenging task as fuzzing targets must be efficient and have a high code coverage to be valuable. Inadequate fuzzing is an issue that is faced by many open-source web protocol libraries, specifically OpenSSL. The OpenSSL library contains numerous fuzzing targets that only cover a fraction of the codebase. Researching and investigating how open-source web protocol implementations are being fuzzed would then aid in producing solutions that improve the fuzzing capabilities of these implementations. Many issues were discovered that caused a lack of fuzzing, predominantly the lack of funding and education on the importance of fuzzing. It was discovered through the results gathered from investigating web protocol fuzzers that there was a significant lack of functions being fuzzed, and the majority of currently-fuzzed targets are inefficient and could be improved. Therefore, after improving some of the critical fuzzing targets currently being used by web protocol implementations, and after writing new targets that focus on covering new functions, the overall code coverage and testing efficiency of these libraries drastically increased. In the future, graphical processing units and dynamic testing will also be utilised to improve fuzzing performance and capabilities while also reducing costs. Keywords: Fuzzing, Web Protocols, Open-source, OSS-Fuzz, libFuzzer, OpenSSL, SSL, TLS, LibreSSL, BoringSSL. v Contents Abstract ...................................................................................................................................... v List of Figures ........................................................................................................................ viii List of Tables ............................................................................................................................ ix Chapter 1 Introduction ......................................................................................................... 1 1.1 Topic Definition ............................................................................................................... 1 1.2 Goals ................................................................................................................................ 2 1.3 Relevance ......................................................................................................................... 3 Chapter 2 Background and Literature Review ..................................................................... 5 2.1 TLS/SSL .......................................................................................................................... 5 2.2 OpenSSL .......................................................................................................................... 9 2.3 Heartbleed ...................................................................................................................... 11 2.3.1 After Heartbleed................................................................................................. 13 2.4 Fuzzing – A Testing Mechanism ................................................................................... 14 2.4.1 Black Box Fuzzing ............................................................................................. 16 2.4.2 White Box Fuzzing ............................................................................................ 17 2.4.3 Problems with Fuzzing OpenSSL ...................................................................... 17 2.5 Fuzzing Engines ............................................................................................................. 18 2.5.1 OSS-Fuzz ........................................................................................................... 18 2.5.2 AFL Fuzzer ........................................................................................................ 19 2.5.3 libFuzzer ............................................................................................................ 20 Chapter 3 Implementations and Experiments ..................................................................... 21 3.1 Justifying OpenSSL................................................................................................... 21 vi 3.2 Investigating Previous Implementations ................................................................... 22 3.2.1 OSS-Fuzz Project Setup ..................................................................................... 22 3.2.2 OpenSSL ............................................................................................................ 24 3.2.2.1 Target: client.c ............................................................................................... 25 3.2.2.2 Target: server.c .............................................................................................. 29 3.2.3 LibreSSL ............................................................................................................ 33 3.2.4 BoringSSL.......................................................................................................... 35 3.3 Implementing New Solutions ........................................................................................ 37 3.3.1 Improving already existing targets..................................................................... 38 3.3.2 Creating new targets .......................................................................................... 44 3.3.2.1 Unicode to ASCII ...................................................................................... 45 3.3.2.2 UTF-8 to Unicode ...................................................................................... 46 Chapter 4 Results and Discussions .................................................................................... 49 4.1 Discovered bugs and vulnerabilities ......................................................................... 49 4.2 Code coverage and fuzzing performance .................................................................. 50 Chapter 5 Conclusions ....................................................................................................... 53 5.1 Summary and conclusions ......................................................................................... 53 5.2 Possible future work .................................................................................................. 55 Appendices ............................................................................................................................... 57 A. Data analyser and plotter for libFuzzer results – code coverage ................................. 57 B. Unicode to ASCII fuzzer ............................................................................................. 58 C. UTF-8 to Unicode fuzzer ............................................................................................. 59 D. Example CSV data produced to analyse results .......................................................... 60 Bibliography ............................................................................................................................ 61 vii List of Figures Figure 1: Top-level view of how Stunnel operates (server-side) [3]. ...................................... 10 Figure 2: This is what American Fuzzy Loop looks like once it has executed [26]. ............... 19 Figure 3: An example of a fuzz target function in C [23]. ....................................................... 20 Figure 4: Code coverage of the ‘client.c’ target ...................................................................... 28 Figure 5: Snippet of the target’s logs while it is being fuzzed................................................. 29 Figure 6: Code coverage of the 'server' fuzzing target ............................................................. 32 Figure 7: Snippet of the do-while loop operation in the server target for OSSL .................... 39 Figure 8: Graph representing the code coverages with and without READ_EARLY_DATA 40 Figure 9: Graph representing the number of executions per second with and without READ_EARLY_DATA .........................................................................................................