University of Louisville ThinkIR: The University of Louisville's Institutional Repository Electronic Theses and Dissertations 12-2007 Methods for detecting kernel rootkits. Douglas Ray Wampler University of Louisville Follow this and additional works at: https://ir.library.louisville.edu/etd Recommended Citation Wampler, Douglas Ray, "Methods for detecting kernel rootkits." (2007). Electronic Theses and Dissertations. Paper 1507. https://doi.org/10.18297/etd/1507 This Doctoral Dissertation is brought to you for free and open access by ThinkIR: The University of Louisville's Institutional Repository. It has been accepted for inclusion in Electronic Theses and Dissertations by an authorized administrator of ThinkIR: The University of Louisville's Institutional Repository. This title appears here courtesy of the author, who has retained all other copyrights. For more information, please contact [email protected]. METHODS FOR DETECTING KERNEL ROOTKITS By Douglas Ray Wampler B.S., Indiana State University, 1994 M.S. Ball State University, 2003 A Dissertation Submited to the Faculty of the Graduate School of the University of Louisville In Partial Fulfillment of the Requirements For the Degree of Doctor of Philsophy Department of Computer Engineering and Computer Science University of Louisville Louisville, Kentucky December 2007 Copyright 2007 by Douglas Ray Wampler All rights reserved METHODS FOR DETECTING KERNEL ROOTKTIS By Douglas Ray Wampler B.S. Indiana State University, 1994 M.S., Ball State University, 2003 A Dissertation Approved on November 12, 2007 By the following Dissertation Committee: ________________________________________ James H. Graham, Dissertation Director ________________________________________ DarJen Chang ________________________________________ Gail W. Depuy ________________________________________ Adel S. Elmaghraby ________________________________________ Mehmed M. Kantardzic ii DEDICATION This dissertation is dedicated to my parents, Mr. Thomas R. Wampler and Mrs. Sharon S. Wampler who have given me invaluable educational opportunities, and generously funded all of my undertakings. iii ABSTRACT METHODS FOR DETECTING KERNEL ROOTKITS Douglas R. Wampler November 12, 2007 Rootkits are stealthy, malicious software that allow an attacker to gain and maintain control of a system, attack other systems, destroy evidence, and decrease the chance of detection. Existing detection methods typically rely on a priori knowledge and operate by either (a) saving the system state before infection and comparing this information post infection, or (b) installing a detection program before infection. This dissertation focuses on detection using reduced a priori knowledge in the form of general knowledge of the statistical properties of broad classes of operating system/architecture pairs. Four new approaches to rootkit detection were implemented and evaluated. A general distribution model is employed against kernel rootkits utilizing the system call table modification attack. Using approaches from the field of outlier detection, this approach successfully detected four different rootkits, with no false positives. Scalability is, however, an issue with this approach. A second, normality- based approach was investigated for use against rootkits infecting systems via the system iv call table modification attack. This approach was partially successful, but did generate false positives in 0.35% of cases. The general distribution model was then applied to rootkits infecting systems via the system call target modification attack. This dataset is dramatically larger, including disassembled memory addresses from the entire kernel. Finally, a modified version of the normality based approach proved effective in detecting kernel rootkits infecting the kernel via the system call target modification attack. This approach capitalizes on the discovery that system calls are loaded into memory sequentially, with the higher level calls, which are more likely to be infected by kernel rootkits loaded first, and the lower level calls loaded later. In the single case evaluated, the enyelkm rootkit, neither false positives nor false positives were indicated. As a final evaluation, these techniques were applied to the Microsoft Windows operating systems. The Windows equivalent of the system call table, the system service descriptor table (SSDT), appears to be almost perfectly normally distributed. A Windows rootkit employing the system call table modification attack was detected using the general distribution and ‘assumption of normality’ models. v TABLE OF CONTENTS PAGE DEDICATION……………………………………………………………………………iii ABSTRACT………………..…………………………………………………………….iv ACKNOWLEDGMENTS…………………………………………………...……………x LIST OF TABLES………………………………………………………………………..xi LIST OF FIGURES………………………………………………………………..……xiii INTRODUCTION……………………………………………………………………...…1 Dissertation Organization………………………………………………………....4 LITERATURE REVIEW………………………………………………………………....6 Rootkits……………..……………………………………………………………..6 Rootkit Classification…………………………...……….……………………….11 Rootkit Detection………………………………………………………………...12 Selected Statistical Methods…………………………………………………..…18 METHODS OF ROOTKIT OPERATION………………………………………………21 The System Call Table Modification Attack……………………………….……21 The System Call Target Modification/System Call Redirection Attack…………23 Analysis of Malicious Code……………………………………………………...28 ANALYSIS OF THE KERNEL…………………………………………………………30 Kernel Modifications………………………………………………………….…30 Memory Analysis Toolset……………….……………………………………….31 Kernel Symbols…………………………………………………………………..32 vi Linux Kernel Modules…………………………………………………………...35 Kernel Debugging: Selected Commands………………………………………..37 DETECTING SYSTEM CALL TABLE MODIFICATION ATTACKS USING GENERAL DISTRIBUTION MODELS…………….……………………………….…40 Definitions and Formal Model…………………………………………………...40 Hardware Platforms……………………………………………………………...44 Statistical Methods………………………………………………………….……45 Exerimental Results………………………………………………………..……53 Conclusions………………………………………………………………………60 DETECTING SYSTEM CALL TABLE MODIFICATION ATTACKS USING NORMAL DISTRIBUTION MODELS……………………………...……………….…62 Definitions and Formal Model…………………………………………………...62 Hardware Platforms……………………………………………………………...64 Normality of Data……………………………………………………………..…65 Statistical Methods………………………………………………………….……68 Experimental Results…………………………………………………….………69 Conclusions………………………………………………………………………76 DETECTING SYSTEM CALL TARGET MODIFICATION ATTACKS USING GENERAL DISTRIBUTION MODELS……..…………...……………………….……78 Definitions………………………………………………………………………..78 Formal Model………………………………………………………………….…82 Hardware Platforms……………………………………………………………...84 Normality of Data…………………………………………………………..……86 Statistical Methods………………………………………………………….……87 Experimental Results…………………………………………………………….88 vii Conclusions………………………………………………………………………89 DETECTING SYSTEM CALL TARGET MODIFICATION ATTACKS USING NORMAL DISTRIBUTION MODELS…………...……………………………….……90 Definitions……………………………………………………………….……….90 Formal Model………………………………………………………………….…91 Order of Appearance…………………………………………………………..…93 Normality of Data…………………………………………………………..……97 Experimental Results………………………………………………………..…...98 Conclusions……………………………………………………………………..100 DETECTING WINDOWS ROOTKITS ……..……………………………………...…102 An Overview of the Windows Architecture and Windows Rootkits………...…102 Definitions………………………………………………………………………105 Formal Model…………………………………………………………………...106 Normality of Data………………………………………………………………107 Experimental Results……………………………………………………...……108 Further Experimental Results………………………………………………..…110 Conclusions………………………………………………………………..……111 CONCLUSIONS AND FUTURE RESEARCH DIRECTIONS………………………113 Conclusions………………………………………………………………..……113 Generalizing A Priori Knowledge……………………………………………...117 Virtualized Rootkits: An Emerging Threat…………………….………………119 REFERENCES………..………..………………………………………………………123 APPENDICES……………………………………………………………………….....127 viii CURRICULUM VITAE………………………..………………………………………244 ix ACKNOWLEDGMENTS I would like to thank my dissertation director Dr. James H. Graham for his expert guidance these many long years, and especially for his uncanny ability to give remarkably sound advice at the correct times. Dr. Gail W. Depuy and Mehmed M. Kantardzic, thank both of you for providing those enduring ephiphanies that statistics and data mining could actually be useful in my research. Dr. Adel S. Elmaghraby, thank you for introducing me to what has turned out to be the very rewarding world of digital forensics. Dr. DarJen Chang, thank you for patiently helping me with my most difficult subject! I learned that I may be self sufficient and still accept assistance from others, and I wish to extend my thanks to all those who helped me when I needed it. Veronica, my faithful guide, I don't know what to say except thank you for accompanying me on what turned out to be a long, difficult, and rewarding journey. I'm afraid that, in all likelihood, I can never properly reward you. May higher education be a light for you all, in the dark places of life, when all other lights go out. x LIST OF TABLES TABLE PAGE 1. Sample System.map File Contents………………………...…………………………33 2. Distribution Fits from 32-bit Intel Machine, Kernel 2.4.27…...……………………...44 3. Distribution Fits from 64-bit SPARC Machine, Kernel 2.4.27……………...……….45 4. Results of Rkit 1.01 Experiment……………………………………………………...55 5. Results of Knark 2.4.3 Experiment……………………………...……………………55 6. Results of Sebek 2.4 Experiment………………………………………….………….57 7. System Call Table for Kernel 2.6……………………………………………….……59 8. Results