Evaluating Approaches for Detecting and Eliminating Memory Safety Errors in Linux Kernel Programming Ilja Sidoroff Master’s Thesis June 2019 School of Technology, Communication and Transport Degree Programme in Information Technology Cyber Security Description Author(s) Type of publication Date Sidoroff, Ilja Master’s Thesis June 2019 Language of publication English Number of pages 77 Permission for web publi- cation: yes Title of publication Evaluating Approaches for Detecting and Eliminating Memory Safety Errors in Linux Kernel Programming Degree programme Information Technology Supervisor(s) Mika Rantonen Mika Karjalainen Assigned by - Abstract Memory Safety means that a program cannot access unintended memory regions. Lack of memory safety continues to be a major source of security related software errors. The problems arising from the use of memory unsafe C programming language are reviewed, both in general and Linux kernel programming context. Various methods for detecting and eliminating memory safety problems are then evaluated. Methods chosen for testing were static and dynamic analysis, and using a memory safe language as a programming language. The methods were tested during a process of writing a Linux kernel module. Unfortunately, none of the methods tested proved to be a comprehensive solution to the problem of memory unsafety. Each method had their own strengths. Static analysis is easy to include in the development process; however, it does not detect problems very efficiently. Dynamic analysis, on the other hand, is good at finding bugs; yet it requires manual testing. Memory safe languages are very promising; however, they would require significant, changes to the existing code which can be difficult to achieve in practice both due to economic and social reasons. Keywords/tags Memory Safety, Spatial and Temporal Memory Safety, C-programming Language, Rust Programming Language, Linux Kernel Programming Miscellaneous Kuvailulehti Tekijä(t) Julkaisun laji Päivämäärä Sidoroff, Ilja Opinnäytetyö, ylempi Toukokuu 2019 AMK Julkaisun kieli Englanti Sivumäärä 77 Verkkojulkaisulupa myön- netty: kyllä Työn nimi Muistiturvallisuudesta johtuvien virheiden havaitseminen ja ehkäisy Linux-kernel-ohjelmoinnissa Tutkinto-ohjelma Insinööri (YAMK), Kyberturvallisuus Työn ohjaaja(t) Rantonen, Mika (JAMK) Karjalainen, Mika (JAMK) Toimeksiantaja(t) - Tiivistelmä Muistiturvallisuus tarkoittaa sitä, että ohjelma ei voi käyttää väärää muistialuetta. Muistiturvallisuuden puute on edelleen merkittävä tietoturvaa vaarantavien ohjelmistovirheiden lähde. Opinnäytteessä tutkittiin ongelmia, jotka aiheutuvat C-ohjelmointikielen muistiturvattomuudesta, niin yleisellä tasolla kuin Linux-kernel ohjelmoinnin-yhteydessä. Lisäksi tutkittiin erilaisia menetelmiä, joilla muistiturvattomuutta voidaan havaita ja ehkäistä. Tutkittavat menetelmät olivat staattinen ja dynaaminen analyysi, sekä muistiturvallisen ohjelmointi- kielen käyttäminen. Menetelmiä testattiin Linux-kernel modulin ohjelmointi- prosessissa. Valitettavasti mikään testatuista menetelmistä ei osoittautunut kaikenkatta- vaksi ratkaisuksi muistiturvattomuuden aiheuttamiin ongelmiin. Jokaisella testatulla menetelmällä oli omat vahvat puolensa. Staattinen analyysi voidaan integroida helposti ohjelmistokehitysprosessiin, mutta se ei havaitse ongelmia kovin tehokkaasti. Dynaaminen analyysi on tehokas menetelmä virheiden löytämiseen mutta edellyttää erillistä testausta. Muistiturvalliset ohjelmointi- kielet ovat erittäin lupaavia, mutta niiden käyttö edellyttäisi suuria muutoksia olemassa olevaan koodiin, mikä voi olla käytännössä vaikeaa sekä taloudellisisten että sosiaalisisten tekijöiden vuoksi. Avainsanat Muistiturvallisuus, Paikallinen ja ajallinen muistiturvallisuus C-ohjelmointikieli, Rust-ohjelmointikieli, Linux-kernel -ohjelmointi Muut tiedot 1 Contents 1 Introduction 4 1.1 Motivation and Overview ....................... 4 1.2 Security in Linux Kernel Development ................ 5 1.3 Research Problem and Methodology ................. 6 2 Memory Safety as Security Problem 8 2.1 Definition of Memory Safety ...................... 8 2.2 Unsafe features of C Programming Language ............. 10 2.3 Stack Corruption and Arbitratry Code Execution .......... 12 2.4 Heap misuse and corruption ...................... 19 2.5 Other Types Common of Vulnerabilities ............... 23 2.6 Code Reuse Attacks .......................... 27 3 Memory Safety Issues in Linux Kernel 30 3.1 Linux Kernel Specific Considerations ................. 30 3.2 Stack and Buffer Overflows ...................... 31 3.3 Dynamic Memory Allocation ..................... 35 3.4 Data Races ............................... 36 3.5 Other Vulnerability Classes ...................... 38 4 Detecting and Eliminating Memory Safety Violations and Other Errors 39 4.1 A Faulty Kernel Module ........................ 40 4.2 Static analysis .............................. 42 4.3 Dynamic Analysis ............................ 46 4.4 Memory-Safe Languages ........................ 48 5 Conclusions 54 2 References 56 Appendices 68 Figures Figure 1 Spatial memory safety violation ............... 9 Figure 2 Temporal memory safety violation .............. 9 Figure 3 Eliminated null pointer check in Linux kernel ........ 12 Figure 4 An example of stack overflow ................. 13 Figure 5 Stack overflow ......................... 14 Figure 6 A program spawning shell ................... 14 Figure 7 Assembly code for shell-spawning program ......... 15 Figure 8 Shellcode as Cstring ...................... 16 Figure 9 Stack canary .......................... 17 Figure 10 Heap and stack layout ..................... 20 Figure 11 Heap and stack layout with guard page ........... 20 Figure 12 An example of use after free ................. 21 Figure 13 Heap with three empty bins ................. 23 Figure 14 Heap with two empty and one used bin ........... 23 Figure 15 Examples of stack layouts in format string attacks ..... 25 Figure 16 Stack in return-to-libc -attack. ................ 28 Figure 17 Partial patch for CVE-2017-1000251 ............. 31 Figure 18 Kernel process stack ...................... 33 Figure 19 Virtually mapped stack .................... 34 Figure 20 Slab allocator ......................... 36 Figure 21 A patch adding lock for CVE-2018-1000004 ......... 38 Figure 22 A Partial KASAN output for stack buffer overflow ..... 49 Figure 23 Redefinitions of C constants and functions inRust ..... 52 3 Figure 24 Stack buffer overflow in Rust ................. 53 Tables Table 1 Fault types ........................... 40 Table 2 Fault detection by basic static analysis ............ 44 Table 3 Tool versions used for static analysis ............. 44 Table 4 Fault detection by dynamic analysis .............. 48 Table 5 Faults prevented in Rust port of C module and reasons for unsafety ................................. 54 4 1 Introduction 1.1 Motivation and Overview Operating systems and their kernels are one of the central points of attacks in the computing environment. They control access to the underlying hardware, and with hardware assistance, they also enforce the separation between user space processes, and privileged kernel code. If the operating system or its kernel is compromised or subverted, the applications running on the operating system can no longer guarantee their confidentially, integrity nor availability. Current operating systems and their kernels are typically very complex. The Linux kernel currently consists of approximately 20 million (Löhner 2017) lines of C and assembly code. The complexity means that it is very difficult to ensure that there are no security critical bugs in the kernels (see below for a brief discussion on kernel security). There have been various attempts to create kernels, which can provide more secu- rity guarantees. On the one end of the spectrum, there is seL4, which a formally verified microkernel (G. Klein et al. 2009). OpenBSD, on the other hand, is a more traditional, monolithic kernel and operating system, which states explicitly its goal to be security oriented (OpenBSD Developers 2017). However, the most popular and widely used operating system kernels remain to be based on the tra- ditional monolithic and complex architecture. The most prominent examples are various versions of Microsoft Windows and Linux kernels, but even Apple’s macOS and iOS, although partly based on Mach microkernels, share the same monolithic architecture (Chisnall 2010). Since the operating systems landscape moves slowly and current operating systems are not likely to be replaced by e.g. seL4, OpenBSD or other variants, securing the current operating systems will probably have more short term utility, than creating new systems, and there is interest in exploring and implementing techniques, which can make the traditional systems more secure. The purpose of this thesis is to review and compare some of the mitigations that can be used to eliminate some of the bug classes in the Linux kernel. Linux was chosen as the target of this study for two reasons. First, following the growing number of security critical bugs in the Linux kernel, there has been an increasing interest in developing new solutions for eliminating different error classes, and not 5 just single bugs (see below for more discussion). Secondly, since the Linux kernel is free and open source software, testing these solutions is much easier than it would be in the case of Microsoft Windows or Apple macOS. 1.2 Security in Linux Kernel Development Safety and