<p>To Bianca, for being who you are. </p><p>Abstract </p><p>Kernel-mode rootkits have gained a considerable momentum within the blackhat community. They&nbsp;represent a considerable threat to any computer system, as they provide an intruder with the ability to hide the presence of his malicious activity. These rootkits make changes to the operating system’s kernel, thereby providing particularly stealthy hiding techniques. </p><p>Considering the kernel rootkit threat and other threats, the collection of reliable information from a compromised system becomes a central problem within the domain of computer security.&nbsp;This thesis addresses this problem.&nbsp;It looks at the possibility of using virtualization as a means to facilitate kernel-mode rootkit detection through integrity checking. </p><p>The thesis describes several areas within the Linux kernel, which are commonly subverted by kernel-mode rootkits. It introduces the reader to the concept of virtualization and describes several technologies employing virtualization. The kernel-mode rootkit threat is then addressed through a description of their hiding methodologies. Some of the existing methods for malware detection are also addressed and analysed. </p><p>A number of general requirements, which need to be satisfied by a general model enabling kernel-mode rootkit detection, are identified.&nbsp;A model addressing these requirements is suggested, and a framework implementing the model is set-up.&nbsp;The detection capabilities of the framework are tested on a couple of rootkits. </p><p>iii </p><p>Preface </p><p>This report presents the results of my master thesis “Integrity checking of operating systems with respect to kernel level malware”. It is written as part of the Master degree (Sivilingeniør) in Computer Science at the Norwegian University of Science and Technology (NTNU), during Spring 2005. The work presented is based on a problem presented by the Computer Network Operations project (CNO857), a research project conducted at the Norwegian Defence Research Establishment (FFI).&nbsp;The work has been conducted at their facilities. </p><p>The work done during this semester has been challenging, instructive and interesting. My knowledge on the researched area was limited before I started my work. I had a limited knowledge to operating systems and Linux in particular.&nbsp;No knowledge of virtual machines, integrity checking and rootkits, and my C-programming skills where almost non existent.&nbsp;Hence, working with these new environments and technologies has thought me a lot, and I feel as though I have increased my platform as a computer scientist. </p><p>I would like to take this opportunity to thank my supervisor, Ane Daae Weng, for providing valuable guidance and feedback through all phases of this project. I would also like to thank professor Mads Nyg˚ard for useful input and support.&nbsp;Camilla Olsen and Espen Aarnes for correctional reading, and other feedback. Further, I would like to thank Tal Garfinkel for quick responses to my emails and his valuable advises on virtual machines. A special thanks is addressed to my fellow students Audun Simonsen and Oddvar Aarseth whom I have been sharing office with.&nbsp;Their support as discussion partners and frustration relieves has been invaluable. </p><p>Tobias Melcher <br>Kjeller, June 22, 2005 v</p><p>Acronyms </p><p>AIDE API </p><p>Advanced Intrusion Detection Environment Application Programmer Interface Community Public License </p><p>CPL CPU DNS ELF </p><p>Central Processing Unit Domain Name System Executable and Linking Format Norwegian Defence Research Establishment General Public License </p><p>FFI GPL HIDS IDS </p><p>Host-based Intrusion Detection System Intrusion Detection System Interrupt Descriptor Table </p><p>IDT LGPL LKM MMU NIC </p><p>Lesser General Public License Loadable Kernel Module Memory Management Unit Network Interface Card </p><p>NIDS </p><p>Network-based Intrusion Detection System <br>NTNU Norwegian University of Science and Technology </p><p>OS </p><p>Operating System User-mode Linux </p><p>UML </p><p>vii </p><p></p><ul style="display: flex;"><li style="flex:1">viii </li><li style="flex:1">Acronyms </li></ul><p></p><p>VFS VM </p><p>Virtual File System Switch Virtual Machine </p><p>VMM VMI </p><p>Virtual Machine Monitor Virtual Machine Introspection </p><p>Table of Contents </p><p></p><ul style="display: flex;"><li style="flex:1">Abstract </li><li style="flex:1">iii </li></ul><p></p><ul style="display: flex;"><li style="flex:1">v</li><li style="flex:1">Preface </li></ul><p></p><ul style="display: flex;"><li style="flex:1">Acronyms </li><li style="flex:1">vii </li></ul><p></p><ul style="display: flex;"><li style="flex:1">ix </li><li style="flex:1">Table of Contents </li></ul><p>List of Figures List of Tables xv xvii </p><ul style="display: flex;"><li style="flex:1">1 Introduction </li><li style="flex:1">1</li></ul><p></p><p>12345667<br>1.1 Motivation .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Problem&nbsp;definition .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Research&nbsp;questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Research&nbsp;methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Project&nbsp;scope and demarcation of assessment&nbsp;. . . . . . . . . . . . . . 1.6 Definitions&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Report&nbsp;outline .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . <br>1.7.1 Reader’s&nbsp;guide .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . </p><p></p><ul style="display: flex;"><li style="flex:1">2 The&nbsp;Linux kernel </li><li style="flex:1">9</li></ul><p></p><p></p><ul style="display: flex;"><li style="flex:1">2.1 Background&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . </li><li style="flex:1">9</li></ul><p>2.2 Architectural&nbsp;overview . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;11 2.3 System&nbsp;calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;12 </p><ul style="display: flex;"><li style="flex:1">2.3.1 The&nbsp;system call table and the interrupt descriptor table (IDT) </li><li style="flex:1">13 </li></ul><p>2.3.2 System&nbsp;call invocation and lifecycle .&nbsp;. . . . . . . . . . . . . . .&nbsp;13 <br>2.4 Memory&nbsp;management .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;16 <br>2.4.1 Memory&nbsp;organization .&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;16 </p><p>ix </p><p></p><ul style="display: flex;"><li style="flex:1">x</li><li style="flex:1">TABLE OF CONTENTS </li></ul><p></p><p>2.4.2 Virtual&nbsp;memory .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;17 2.4.3 Memory&nbsp;allocation . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;18 <br>2.5 File&nbsp;system management . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;19 2.6 Loadable&nbsp;Kernel Modules (LKM)&nbsp;. . . . . . . . . . . . . . . . . . . . .&nbsp;20 <br>2.6.1 Linking&nbsp;and unlinking modules&nbsp;. . . . . . . . . . . . . . . . . .&nbsp;20 <br>2.7 Discussion&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;21 </p><p></p><ul style="display: flex;"><li style="flex:1">3 The&nbsp;virtualization technology </li><li style="flex:1">23 </li></ul><p></p><p>3.1 Background&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;23 3.2 Virtualization&nbsp;techniques .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . .&nbsp;24 <br>3.2.1 The&nbsp;virtual machine monitor (VMM)&nbsp;. . . . . . . . . . . . . .&nbsp;24 3.2.2 Virtual&nbsp;environments .&nbsp;. . . . . . . . . . . . . . . . . . . . . . .&nbsp;25 <br>3.3 Virtual&nbsp;machine technologies&nbsp;. . . . . . . . . . . . . . . . . . . . . . .&nbsp;26 <br>3.3.1 Plex86&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;26 3.3.2 VMware&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;26 3.3.3 Xen&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;27 3.3.4 User-mode&nbsp;Linux (UML)&nbsp;. . . . . . . . . . . . . . . . . . . . .&nbsp;29 <br>3.4 Discussion&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;31 </p><p></p><ul style="display: flex;"><li style="flex:1">4 Rootkits </li><li style="flex:1">33 </li></ul><p></p><p>4.1 Trojan&nbsp;horse .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;33 4.2 User-mode&nbsp;rootkits . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;35 4.3 Kernel-mode&nbsp;rootkits . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;35 <br>4.3.1 Loadable&nbsp;Kernel Modules (LKM) . . . . . . . . . . . . . . . . .&nbsp;37 4.3.2 Patching&nbsp;the running kernel . . . . . . . . . . . . . . . . . . . .&nbsp;39 4.3.3 Patching&nbsp;the kernel binary image . . . . . . . . . . . . . . . . .&nbsp;40 4.3.4 Create&nbsp;a fraudulent virtual system&nbsp;. . . . . . . . . . . . . . . .&nbsp;41 4.3.5 Running&nbsp;programs in kernel-mode&nbsp;. . . . . . . . . . . . . . . .&nbsp;41 <br>4.4 Discussion&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;41 </p><p></p><ul style="display: flex;"><li style="flex:1">5 Defending&nbsp;the Linux kernel </li><li style="flex:1">43 </li></ul><p></p><p>5.1 Protecting&nbsp;a Linux system . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;43 5.2 Intrusion&nbsp;detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;44 5.3 Anomality&nbsp;detection .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;45 <br>5.3.1 File&nbsp;integrity checking&nbsp;. . . . . . . . . . . . . . . . . . . . . . .&nbsp;46 5.3.2 Kernel&nbsp;integrity checking&nbsp;. . . . . . . . . . . . . . . . . . . . .&nbsp;47 5.3.3 Detecting&nbsp;anomalities in program execution&nbsp;. . . . . . . . . . .&nbsp;48 5.3.4 Network&nbsp;connections and activity .&nbsp;. . . . . . . . . . . . . . . .&nbsp;49 </p><p></p><ul style="display: flex;"><li style="flex:1">TABLE OF CONTENTS </li><li style="flex:1">xi </li></ul><p></p><p>5.4 Signature&nbsp;detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;50 <br>5.4.1 Fingerprinting&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;50 5.4.2 Characterising&nbsp;kernel-mode rootkits&nbsp;. . . . . . . . . . . . . . .&nbsp;50 <br>5.5 Hardware-based&nbsp;detection .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . .&nbsp;50 5.6 Sandboxing&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;51 <br>5.6.1 Virtual&nbsp;machine introspection (VMI) .&nbsp;. . . . . . . . . . . . . .&nbsp;51 5.6.2 Intrusion&nbsp;detection in virtual environments&nbsp;. . . . . . . . . . .&nbsp;51 <br>5.7 Discussion&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;52 </p><p></p><ul style="display: flex;"><li style="flex:1">6 A&nbsp;model supporting kernel integrity checking </li><li style="flex:1">53 </li></ul><p></p><p>6.1 Motivating&nbsp;scenario .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;53 <br>6.1.1 The&nbsp;MyOil scenario&nbsp;. . . . . . . . . . . . . . . . . . . . . . . .&nbsp;54 6.1.2 Summarising&nbsp;and analysing the scenario&nbsp;. . . . . . . . . . . . .&nbsp;56 <br>6.2 Model&nbsp;requirements .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;58 <br>6.2.1 Properties&nbsp;of kernel level malware&nbsp;. . . . . . . . . . . . . . . .&nbsp;58 6.2.2 Behaviour&nbsp;and support .&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;59 6.2.3 Provided&nbsp;services . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;60 <br>6.3 Overall&nbsp;description of the model&nbsp;. . . . . . . . . . . . . . . . . . . . .&nbsp;60 <br>6.3.1 Component&nbsp;functionalities and responsibilities&nbsp;. . . . . . . . .&nbsp;62 6.3.2 Main&nbsp;features . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;65 6.3.3 Comparison&nbsp;of the VMI architecture and the described model .&nbsp;65 <br>6.4 Employing&nbsp;the model on the MyOil scenario .&nbsp;. . . . . . . . . . . . . .&nbsp;66 </p><p></p><ul style="display: flex;"><li style="flex:1">7 Building&nbsp;a framework </li><li style="flex:1">69 </li></ul><p></p><p>7.1 Selecting&nbsp;a virtualization technology&nbsp;. . . . . . . . . . . . . . . . . . .&nbsp;69 <br>7.1.1 Requirement&nbsp;coverage .&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;70 7.1.2 Other&nbsp;considerations . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;71 7.1.3 Open&nbsp;source evaluation&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;72 7.1.4 Summarizing&nbsp;the virtualization technology evaluation&nbsp;. . . . .&nbsp;72 <br>7.2 Selecting&nbsp;an integrity checker&nbsp;. . . . . . . . . . . . . . . . . . . . . . .&nbsp;73 <br>7.2.1 Requirement&nbsp;coverage .&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;73 7.2.2 Other&nbsp;considerations . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;75 7.2.3 Open&nbsp;source evaluation&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;75 7.2.4 Summarizing&nbsp;the integrity checker evaluation&nbsp;. . . . . . . . . .&nbsp;75 <br>7.3 Framework&nbsp;setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;76 <br>7.3.1 Installing&nbsp;Xen .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;77 7.3.2 Running&nbsp;Xen . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;78 7.3.3 Installing&nbsp;and initialising Afick&nbsp;. . . . . . . . . . . . . . . . . .&nbsp;79 </p><p></p><ul style="display: flex;"><li style="flex:1">xii </li><li style="flex:1">TABLE OF CONTENTS </li></ul><p></p><p>7.4 Summary&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;80 </p><p></p><ul style="display: flex;"><li style="flex:1">8 Applying&nbsp;the framework </li><li style="flex:1">83 </li></ul><p></p><p>8.1 Test&nbsp;setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;83 <br>8.1.1 Preparing&nbsp;the guest system&nbsp;. . . . . . . . . . . . . . . . . . . .&nbsp;84 8.1.2 Installing&nbsp;and running Adore&nbsp;. . . . . . . . . . . . . . . . . . .&nbsp;84 8.1.3 Installing&nbsp;and running Adore-ng&nbsp;. . . . . . . . . . . . . . . . .&nbsp;84 8.1.4 Installing&nbsp;and running SucKIT&nbsp;. . . . . . . . . . . . . . . . . .&nbsp;85 8.1.5 Final&nbsp;test setup .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;86 <br>8.2 Test&nbsp;results .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;87 <br>8.2.1 Test&nbsp;1 .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;87 8.2.2 Test&nbsp;2 .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;88 8.2.3 Result&nbsp;summary .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;89 </p><p></p><ul style="display: flex;"><li style="flex:1">9 Discussion&nbsp;and evaluation </li><li style="flex:1">91 </li></ul><p></p><p>9.1 Discussion&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;91 <br>9.1.1 Discussion&nbsp;of results&nbsp;. . . . . . . . . . . . . . . . . . . . . . . .&nbsp;91 9.1.2 Discussion&nbsp;of the framework .&nbsp;. . . . . . . . . . . . . . . . . . .&nbsp;92 9.1.3 Discussion&nbsp;of the model&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;94 <br>9.2 Evaluation&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;96 </p><p></p><ul style="display: flex;"><li style="flex:1">10 Conclusion and further work </li><li style="flex:1">99 </li></ul><p></p><p>10.1 Conclusion&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;99 <br>10.1.1 Important&nbsp;themes .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . .&nbsp;99 10.1.2 Contributions&nbsp;of this thesis&nbsp;. . . . . . . . . . . . . . . . . . . .&nbsp;100 10.1.3 The&nbsp;future .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;101 <br>10.2 Further&nbsp;work .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;101 <br>10.2.1 Implementing&nbsp;the model . . . . . . . . . . . . . . . . . . . . . .&nbsp;101 10.2.2 Further&nbsp;research .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;101 </p><p></p><ul style="display: flex;"><li style="flex:1">Bibliography </li><li style="flex:1">103 </li></ul><p>109 113 <br>A Glossary B Rootkit&nbsp;examples </p><p>B.1 Subverting&nbsp;the VFS - Adore-ng&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;113 B.2 Patching&nbsp;/dev/kmem - SucKIT&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;115 </p><p></p><ul style="display: flex;"><li style="flex:1">C Rootkit&nbsp;detection tools </li><li style="flex:1">117 </li></ul><p></p><ul style="display: flex;"><li style="flex:1">TABLE OF CONTENTS </li><li style="flex:1">xiii </li></ul><p></p><ul style="display: flex;"><li style="flex:1">D Creating&nbsp;and running Loadable Kernel Modules (LKM) </li><li style="flex:1">119 </li></ul><p></p><p>D.1 LKM&nbsp;programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;119 D.2 LKM&nbsp;compilation .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;120 </p><p></p><ul style="display: flex;"><li style="flex:1">E Notation </li><li style="flex:1">121 </li></ul><p></p><p>E.1 Notation&nbsp;used in the evaluation table .&nbsp;. . . . . . . . . . . . . . . . . .&nbsp;121 E.2 False-positives&nbsp;and false-negatives .&nbsp;. . . . . . . . . . . . . . . . . . . .&nbsp;123 </p><p>List of Figures </p><p></p><ul style="display: flex;"><li style="flex:1">1.1 Report&nbsp;overview .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . </li><li style="flex:1">8</li></ul><p>2.1 Intel&nbsp;x86 protection rings&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;10 2.2 Conceptual&nbsp;Linux architecture&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;11 2.3 System&nbsp;calls as an interface to hardware&nbsp;. . . . . . . . . . . . . . . . .&nbsp;12 2.4 The&nbsp;Linux system call table&nbsp;. . . . . . . . . . . . . . . . . . . . . . . .&nbsp;14 2.5 System&nbsp;call invocation in Linux&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;15 2.6 Reserved&nbsp;memory for the Linux kernel&nbsp;. . . . . . . . . . . . . . . . . .&nbsp;17 2.7 The&nbsp;Linux virtual memory .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . .&nbsp;18 2.8 The&nbsp;Linux file system&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;19 2.9 Loadable&nbsp;kernel module&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;21 </p><p>3.1 Virtual&nbsp;machine environments [30]&nbsp;. . . . . . . . . . . . . . . . . . . .&nbsp;25 3.2 VMware&nbsp;architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;27 3.3 Xen&nbsp;architecture .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;29 3.4 User-mode&nbsp;Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;31 </p><p>4.1 Comparison&nbsp;of user-mode and kernel-mode rootkits .&nbsp;. . . . . . . . . .&nbsp;36 4.2 Loading&nbsp;an evil Loadable Kernel Module (LKM)&nbsp;. . . . . . . . . . . .&nbsp;38 4.3 Runtime&nbsp;kernel patching . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;40 </p><p>6.1 Scenario:&nbsp;MyOil’s network .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . .&nbsp;54 6.2 Scenario:&nbsp;MyOil’s network under attack&nbsp;. . . . . . . . . . . . . . . . .&nbsp;56 6.3 Overall&nbsp;system description . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;61 6.4 Architectural&nbsp;overview . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;61 6.5 Employing&nbsp;the model on the scenario .&nbsp;. . . . . . . . . . . . . . . . . .&nbsp;66 </p><p>7.1 Detection&nbsp;framework . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;76 7.2 The&nbsp;configuration file for the Fedora2 domain&nbsp;. . . . . . . . . . . . . .&nbsp;79 7.3 Running&nbsp;Xen .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;80 </p><p>xv </p><p></p><ul style="display: flex;"><li style="flex:1">xvi </li><li style="flex:1">LIST OF FIGURES </li></ul><p></p><p>8.1 Running&nbsp;SucKIT on the Fedora2 domain .&nbsp;. . . . . . . . . . . . . . . .&nbsp;86 8.2 Running&nbsp;a modified SucKIT on the Fedora2 domain&nbsp;. . . . . . . . . .&nbsp;86 8.3 The&nbsp;framework test setup&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;87 8.4 Detecting&nbsp;Adore by comparison . . . . . . . . . . . . . . . . . . . . . .&nbsp;88 8.5 Detecting&nbsp;the Adore rootkit with Afick .&nbsp;. . . . . . . . . . . . . . . . .&nbsp;89 </p><p>B.1 The&nbsp;Adore user interface, Ava.&nbsp;. . . . . . . . . . . . . . . . . . . . . .&nbsp;114 </p><p>List of Tables </p><p>3.1 Para-virtualizing&nbsp;the x86 architecture in Xen [5]&nbsp;. . . . . . . . . . . .&nbsp;28 3.2 Running&nbsp;Linux in user-mode [17, 59]&nbsp;. . . . . . . . . . . . . . . . . . .&nbsp;30 </p><p>5.1 Linux&nbsp;commands for anomality detection&nbsp;. . . . . . . . . . . . . . . .&nbsp;45 6.1 General&nbsp;requirements .&nbsp;. . . . . . . . . . . . . . . . . . . . . . . . . . .&nbsp;58 7.1 Open&nbsp;source software evaluation&nbsp;. . . . . . . . . . . . . . . . . . . . .&nbsp;72 7.2 Open&nbsp;source software evaluation&nbsp;. . . . . . . . . . . . . . . . . . . . .&nbsp;75 7.3 Mapping&nbsp;the framework to the model of Chapter 6&nbsp;. . . . . . . . . . .&nbsp;81 </p><p>8.1 Mapping&nbsp;the test setup to the model of Chapter 6&nbsp;. . . . . . . . . . .&nbsp;86 9.1 Covering&nbsp;the general requirements&nbsp;. . . . . . . . . . . . . . . . . . . .&nbsp;94 </p><p>xvii </p><p>Chapter 1 </p><p>Introduction </p><p>The theme of this master thesis is the use of integrity checking of the operating systems kernel to discover anomalities implying kernel level exploits.&nbsp;The focus will be on the Linux kernel and the detection of kernel-mode rootkits. </p><p>Section 1.1 outlines the motivation and importance of discovering malicious activity. Section&nbsp;1.2 defines the problem addressed.&nbsp;Lastly Section 1.7 describes the structure of this thesis, serving as a roadmap for the reader. </p><p>1.1 Motivation </p><p>Interconnected computers are subject to constant attacks from people wanting to exploit the processing power, gain access to information, or to have fun.&nbsp;Preventing and detecting such attacks is important to assure a certain level of integrity and privacy for computing systems.&nbsp;Since methods employed by attackers become more and more sophisticated, we have a continuous arms race between defenders and attackers. </p><p>Common goals of the attacker include; to make sure that the legitimate users or system administrators are unaware of their system being compromised, and when administrator privileges are obtained, to maintain and keep this privileged access in the foreseeable future. A commonly used method achieving these goals is the use of a rootkit. A&nbsp;rootkit is a collection of tools, which allows an intruder to hide his presence and maintain his access. </p><p>Two types of rootkits exist; user-mode rootkits and kernel-mode rootkits. A usermode rootkit modifies critical system level binaries and programs, while a kernel-mode </p><p>1</p><p></p><ul style="display: flex;"><li style="flex:1">2</li><li style="flex:1">Chapter 1. Introduction </li></ul><p></p><p>rootkit replaces or modifies the operating systems kernel.&nbsp;Kernel-mode rootkits are harder to detect than user-mode rootkits.&nbsp;They operate on a low level and hence user-mode inspection tools are unable to detect such rootkits. This makes the kernelmode rootkit a very powerful tool. Further, rootkits have become more user-friendly as blackhat hackers<sup style="top: -0.3298em;">1 </sup>have applied user interfaces allowing not so well-informed intruders, or script kiddies, to apply the rootkit easily.&nbsp;These advantages have made kernel-mode rootkits very popular, and they have become common in a high percentage of the intrusions reported, implying administrator level access [39]. </p>