Effectiveness of Linux Rootkit Detection Tools
Total Page:16
File Type:pdf, Size:1020Kb
Effectiveness of Linux Rootkit Detection Tools University of Oulu Faculty of Information Technology and Electrical Engineering Degree Programme in Information Processing Sciences Master’s Thesis Juho Junnila 27.3.2020 2 Abstract Rootkits – a type of software that specializes in hiding entities in computer systems while enabling continuous control or access to it – are particularly difficult to detect compared to other kinds of software. Various tools exist for detecting rootkits, utilizing a wide variety of detection techniques and mechanisms. However, the effectiveness of such tools is not well established, especially in contemporary academic research and in the context of the Linux operating system. This study carried out an empirical evaluation of the effectiveness of five tools with capabilities to detect Linux rootkits: OSSEC, AIDE, Rootkit Hunter, Chkrootkit and LKRG. The effectiveness of each tool was tested by injecting 15 publicly available rootkits in individual detection tests in virtual machines running Ubuntu 16.04, executing the detection tool and capturing its results for analysis. A total of 75 detection tests were performed. The results showed that only 37.3% of the detection tests provided any indication of a rootkit infection or suspicious system behaviour, with the rest failing to provide any signs of anomalous behaviour. However, combining the findings of multiple detection tools increased the overall detection rate to 93.3%, as all but a single rootkit were discovered by at least one tool. Variation was observed in the effectiveness of the detection tools, with detection rates ranging from 13.3% to 53.3%. Variation in detection effectiveness was also found between categories of rootkits, as the overall detection rate was 46.7% for user mode rootkits and 31.1% for kernel mode rootkits. Overall, the findings showed that while an individual detection tool‘s effectiveness can be lacking, using a combination of tools considerably increased the likelihood of a successful detection. Keywords rootkits, rootkit detection, digital forensics, computer security, operating systems, Linux Supervisor Ph.D., University lecturer, Ari Vesanen 3 Foreword This study was born of a personal curiosity to develop a deeper understanding of how rootkits work and how they can be detected using freely available tools. Its origins can be traced back to Trammell Hudson's exceptional presentation of the Thunderstrike EFI rootkit at the 31st Chaos Communication Congress (31C3) conference, which profoundly impressed me with its exotic nature, technical prowess and compelling narrative. Narrowing the research context to Linux was also due to both personal and professional interests, as it is the operating system I'm most deeply familiar with and the one I use and value the most. I would like to thank my thesis supervisor Ari Vesanen for his adept guidance and for the intriguing discussions during our meetings. I would also like to extend my sincere gratitude to my family and friends for all their support and patience during my work on this thesis, my studies and everything else. Juho Junnila Oulu, March 27, 2020 4 Abbreviations AIDE: Advanced Intrusion Detection Environment AMT: Active Management Technology APT: Advanced Package Tool EFI: Extensible Firmware Interface HIDS: Host-based Intrusion Detection System HPC: Hardware Performance Counter IDT: Interrupt Descriptor Table LKM: Loadable Kernel Module LKRG: Linux Kernel Runtime Guard ME: Intel Management Engine OSSEC: Open Source HIDS Security PAM: Pluggable Authentication Modules PID: Process ID SMM: System Management Mode SSDT: System Service Dispatch Table UID: User ID UMH: Linux Usermode Helper VM: Virtual Machine VMI: Virtual Machine Introspection VMM: Virtual Machine Monitor 5 Contents Abstract ............................................................................................................................. 2 Foreword ........................................................................................................................... 3 Abbreviations .................................................................................................................... 4 Contents ............................................................................................................................ 5 1. Introduction .................................................................................................................. 6 1.1 Motivation ............................................................................................................ 6 1.2 Research problem and questions .......................................................................... 6 1.3 Research method .................................................................................................. 7 1.4 Structure ............................................................................................................... 7 2. Background .................................................................................................................. 8 2.1 Literature search .................................................................................................. 8 2.2 Evaluation of rootkit detection effectiveness ....................................................... 9 2.3 Linux ................................................................................................................ 10 2.4 Rootkits .............................................................................................................. 12 2.5 Rootkit detection ................................................................................................ 15 3. Methodology .............................................................................................................. 18 3.1 Research method ................................................................................................ 18 3.2 Rootkits .............................................................................................................. 19 3.2.1 User mode rootkits ................................................................................. 19 3.2.2 Kernel mode rootkits .............................................................................. 21 3.3 Rootkit detection tools ....................................................................................... 23 3.4 Test environment ............................................................................................... 25 3.5 Test process ........................................................................................................ 27 3.6 Evaluation procedure ......................................................................................... 28 4. Results ........................................................................................................................ 29 4.1 Detection runs .................................................................................................... 29 4.2 OSSEC ............................................................................................................... 29 4.3 AIDE ................................................................................................................ 31 4.4 Rootkit Hunter ................................................................................................... 33 4.5 Chkrootkit .......................................................................................................... 36 4.6 LKRG ................................................................................................................ 38 4.7 Summary ............................................................................................................ 40 5. Discussion .................................................................................................................. 46 5.1 Relation to prior studies ..................................................................................... 46 5.2 RQ1: Rootkit detection effectiveness ................................................................ 46 5.3 RQ2: Differences in detection tool effectiveness .............................................. 47 5.4 RQ3: Detection of rootkit types ......................................................................... 48 5.5 Practicality of the detection tools ....................................................................... 49 6. Conclusion .................................................................................................................. 51 References ....................................................................................................................... 53 Appendix A. Literature search terms and results ........................................................... 57 Appendix B. Detection tool setup .................................................................................. 60 Appendix C. Rootkit setup ............................................................................................. 63 6 1. Introduction This chapter describes the background and motivation for this study. The research problem and research questions are also defined. Finally, the utilized research method and the structure of the thesis are described. 1.1 Motivation A rootkit is a type of software that specializes in hiding its presence in computer systems. The main purpose of a rootkit is to facilitate continuous access to the system it is installed on. While often associated with malware, the use of rootkits is not necessarily