Operating System Stability and Security Through Process Homeostasis
Total Page：16
File Type：pdf, Size：1020Kb
Operating System Stability and Security through Process Homeostasis by Anil Buntwal Somayaji B.S., Massachusetts Institute of Technology, 1994 DISSERTATION Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Computer Science The University of New Mexico Albuquerque, New Mexico July 2002 c 2002, Anil Buntwal Somayaji iii iv Dedication To all those system administrators who spend their days nursemaiding our computers. v vi Acknowledgments Graduate school has turned out to be a long, difficult, but rewarding journey for me, and I would not have made it this far without the help of many, many people. I cannot hope to completely document how these influences have helped me grow; nevertheless, I must try and acknowledge those who have accompanied me. First, I would like to thank my committee: David Ackley, Rodney Brooks, Barney Maccabe, Margo Seltzer, and especially my advisor, Stephanie Forrest. They all had to read through a rougher manuscript than I would have liked, and had to do so under significant time pressures. They provided invaluable feedback on my ideas and experiments. Most importantly, they have been my teachers. I'm also grateful to have been part of the Adaptive Computation Group at the University of New Mexico and to have been a member of the complex adaptive systems community here in New Mexico. Fellow graduate students at UNM and colleagues at the Santa Fe Institute have shaped my thoughts and have inspired me to do my best work. I will miss all of you. The Computer Science Department at UNM has been my home for the past several years. Its faculty members have instructed me and have treated me as a colleague; its administrative staff have helped me with the practical details of being a graduate student. The department has been a supportive environment, and it will be hard to leave. Also, the Systems Support Group of the UNM Computer Science department helped me collect some of my most important datasets. In the process, they sometimes had to deal with crises that I inadvertently created. They have been both helpful and understanding. Over the past several years many people in the computer security community have helped us understand important past work, yet have also encouraged us to pursue our own vision. They have been valuable colleagues and friends. The Massachusetts Institute of Technology was my home during my undergrad- uate years, and I was also a visiting graduate student in its Artificial Intelligence laboratory during the 1996-97 academic year. This year was a valuable part of my graduate school education. With the help of my former physics teacher, Bill Rodriguez, the University School of Nashville's web server provided the best data on the behavior of pH. Finally, I must thank my many friends for supporting me, Dana for being there during the last stages of my graduate career, and my parents and my sister Shalini for providing the love and support to get me this far. This work was funded by NSF, DARPA, ONR, Intel, and IBM. vii viii Operating System Stability and Security through Process Homeostasis by Anil Buntwal Somayaji ABSTRACT OF DISSERTATION Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Computer Science The University of New Mexico Albuquerque, New Mexico July 2002 Operating System Stability and Security through Process Homeostasis by Anil Buntwal Somayaji B.S., Massachusetts Institute of Technology, 1994 Ph.D., Computer Science, University of New Mexico, 2002 Abstract Modern computer systems are plagued with stability and security problems: ap- plications lose data, web servers are hacked, and systems crash under heavy load. Many of these problems arise from rare program behaviors. pH (process Homeosta- sis) is a Linux 2.2 kernel extension which detects unusual program behavior and responds by slowing down that behavior. Inspired by the homeostatic mechanisms organisms use to stabilize their internal environment, pH detects changes in pro- gram behavior by observing changes in short sequences of system calls. When pH determines that a process is behaving unusually, it responds by slowing down that process's system calls. If the anomaly corresponds to a security violation, delays often stop attacks before they can do damage. Delays also give users time to decide whether further actions are warranted. xi My dissertation describes the rationale, design, and behavior of pH. Experimental results are reported which show that pH effectively captures the normal behavior of a variety of programs under normal use conditions. This captured behavior allows it to detect anomalies with a low rate of false positives (as low as 1 user interven- tion every five days). Data are presented that show pH responds effectively and autonomously to buffer overflows, trojan code, and kernel security flaws. pH can also help administrators by detecting newly-introduced configuration errors. At the same time, pH is extremely lightweight: it incurs a general performance penalty of only a few percent, a slowdown that is imperceptible in practice. The pH prototype is licensed under the GNU General Public License and is available for download at http://www.cs.unm.edu/∼soma/pH/. xii Contents List of Figures xix List of Tables xxi 1 Introduction 1 1.1 Motivation . 2 1.2 Computer Homeostasis . 3 1.3 System-call Monitoring and Response . 6 1.4 Contributions . 8 1.5 Overview . 9 2 Background 11 2.1 Computer Security . 11 2.1.1 Intrusion Detection . 13 2.1.2 Automated Response . 17 2.1.3 Kernel-level Security Projects . 19 xiii Contents 2.2 Operating Systems . 20 2.3 Compilers . 21 2.4 Fault Tolerance . 22 2.5 System Administration . 24 2.6 Artificial Life & Robotics . 25 3 Homeostasis 29 3.1 Biological Homeostasis . 30 3.1.1 Temperature Control . 30 3.1.2 Immune System . 31 3.2 Process Homeostasis . 34 3.2.1 Requirements . 35 3.2.2 Abstractions . 36 3.2.3 Enclosed System . 38 3.2.4 System Property . 38 3.2.5 Detector . 39 3.2.6 Effector . 41 3.2.7 The User's View of pH . 42 4 Analyzing System Call Sequences 45 4.1 Requirements . 45 xiv Contents 4.2 Description . 47 4.3 Analysis . 51 4.3.1 Speed . 52 4.3.2 Profile Size . 53 4.3.3 Convergence . 54 4.3.4 Anomaly Sensitivity . 56 4.4 Summary . 57 5 Implementing pH 59 5.1 Why a kernel implementation? . 59 5.2 Implementation Overview . 61 5.3 Classifying Normal Profiles . 65 5.3.1 Requirements . 65 5.3.2 Profile Heuristics . 66 5.4 Delaying System Calls . 68 5.5 Tolerization and Sensitization . 71 5.6 Data Structures . 72 5.7 Kernel Integration . 76 5.8 Interacting with pH . 78 5.9 Parameters . 81 5.9.1 System Call Window Sizes . 82 xv Contents 5.9.2 Logging . 82 5.9.3 Classifying Normal Profiles . 83 5.9.4 Automated Response . 84 5.10 Performance . 86 6 Normal Program Behavior in Practice 91 6.1 What is Normal Behavior? . 92 6.2 A Day in Detail . 94 6.3 Choosing a Method & Window Size . 98 6.4 A Few Weeks on a Personal Workstation . 101 6.5 Normal Monitoring . 102 6.6 False Positives . 107 6.7 Normal Behavior on Other Hosts . 111 6.8 Profile Diversity and Complexity . 114 6.9 suspend execve Issues & Longer-Term Data . 119 6.10 Normal in Practice . 122 7 Anomalous Behavior 125 7.1 What Changes Are Detectable? . 126 7.2 Inetd Perturbations . 133 7.2.1 Daytime . 135 xvi Contents 7.2.2 Chargen . 136 7.2.3 Unknown External Program . 139 7.2.4 Unknown Service . 141 7.2.5 Summary . 144 7.3 Intrusions . 145 7.3.1 A Buffer Overflow . 147 7.3.2 Trojan Code . 149 7.3.3 A Kernel Vulnerability . 151 7.4 Other Attacks . 153 7.4.1 Earlier pH Experiments . 153 7.4.2 Linux capability exploit . 156 7.4.3 System-call Monitoring Experiments . 157 7.5 Intelligent Adversaries . 160 7.6 Summary . 162 8 Discussion 165 8.1 Contributions . 165 8.2 Limitations . 167 8.2.1 Capturing Normal Behavior . 168 8.2.2 Denial of Service . 169 8.2.3 Portability . 171 xvii Contents 8.3 Computer Homeostasis . 173 References 175 xviii List of Figures 1.1 A schematic diagram of classes of program behavior. 4 1.2 The process of making a system call. 7 3.1 An MHC-peptide complex . 33 5.1 Basic flow of control and data in a pH-modified Linux kernel. 63 5.2 Training and testing datasets . 64 5.3 pHmon screenshot . 64 5.4 pH's normal profile heuristics. 67 5.5 Schematic delay graph . ..