VM-based Security Overkill: A Lament for Applied Systems Security Research Sergey Bratus Michael E. Locasto Ashwin Ramaswamy Dartmouth College University of Calgary Dartmouth College Sean W. Smith Dartmouth College ABSTRACT open-source cousins like VirtualBox). Virtual machine manage- Virtualization has seen a rebirth for a wide variety of uses; in our ment infrastructure has — notably and recently — increased the field, systems security researchers routinely use it as a standard tool practicality of large scale, commodity distributed computing (i.e., for providing isolation and introspection. Researchers’ use of vir- the “cloud”). tual machines has reached a level of orthodoxy that makes it dif- In addition to arguments involving reduced management, admin- ficult for the collective wisdom to consider alternative approaches istration, and hardware costs, virtualization technology routinely to protecting computation. We suggest that many scenarios exist sees service in a security context, primarily driven by the assump- where virtual machines do not provide a suitable tool or appropri- tion that virtualization provides the best approach (for some value ate security properties. We analyze the use of virtual machines in of “best”, whether this means (1) “most practical”; (2) more com- jail chroot(2) the systems security space and we highlight other work that ques- prehensive than BSD’s , , Janus, or systrace [35]; tions the current (ab)uses of virtualization. or (3) something else) to providing isolation between execution The takeaway message of this paper is that “self-protection” mech- containers. At an increased cost (typically), a virtual environment anisms still represent an interesting and viable path of research. At can provide inspection capabilities in addition to basic isolation. some point, hypervisors (or whatever the lowest layer of software, This kind of isolation and trapping of sensitive operations presents firmware, or programmable hardware is) must rely on detection and a temptation too sweet for system security researchers to resist (un- protection mechanisms embedded within themselves. derstandably, since pre-existing environments reduce the workload of setting up an execution container). While we agree that virtualization can be quite useful in many Categories and Subject Descriptors scenarios, we observe that its level of use has approached ortho- H.1.1 [Models and Principles]: Systems and Information The- doxy in terms of the appropriate technology for composing secu- ory—Value of Information rity systems involving isolation, inspection, introspection, or be- havioral analysis of execution. In short, as a field, the community General Terms seems to have found the perfect implementation form of a practical reference monitor. Security, Measurement 1.1 Contribution Keywords This paper attempts to question that orthodoxy. Are VM-based virtualization, isolation, VM solutions scalable or even economically feasible to enterprise and SCADA network with respect to management and administration 1. INTRODUCTION overhead they require? Are non-VM approaches still a viable and practical means of achieving isolation, inspection, and other forms Virtualization is clearly an enabling technology: providing exe- of program behavior analysis? Specifically, in situations where cution containers can make computing cheaper, more mobile, eas- sliding another security-enforcing layer such as a hypervisor/VMM ier to back up, share, and archive entire OS environments. Many might prove too costly for the platform, software might inevitably variations on the theme of pure virtualization exist, from open- fall back to examining itself. We suggest that (1) such a fallback source CPU emulators like Bochs and QEMU through container- is inevitable for certain scenarios and so (2) it has to be done with based systems like User-mode Linux, OpenVZ, and Zap [31] to appropriate engineering principles and care to make it least ad hoc more commercial offerings like VMWare and VirtualPC (and their as possible. 1.2 Motivation: Detecting Malicious Compu- Permission to make digital or hard copies of all or part of this work for tation With Little Performance Impact personal or classroom use is granted without fee provided that copies are Our motivation to examine the possibility of non-VM approaches not made or distributed for profit or commercial advantage and that copies to software supervision, introspection, mediation, and isolation be- bear this notice and the full citation on the first page. To copy otherwise, to gan with a consideration of how to implement anti-rootkit detection republish, to post on servers or to redistribute to lists, requires prior specific capabilities on a range of low-power and low-resource embedded permission and/or a fee. NSPW’10, September 21–23, 2010, Concord, MA, USA. platforms (e.g., mobile phones, 802.11 access points, SCADA con- Copyright 2010 ACM 978-1-4503-0415-3/10/09 ...$10.00. trollers). Such devices typically do not have the resources to run an entire VM infrastructure. Furthermore, as a practical matter, using so-called “personal firewalls”), despite their history of vulnerabili- currently available commercial or F/OSS VM technology entails si- ties, continue to be adopted. multaneously running and maintaining two operating systems, with Second, the study of instrumentation for certain classes of unde- all their attendant services, libraries, and software packages, the sirable events, even when undertaken in the presence of theoreti- locking down and constant patching of which presents a dramat- cally cleaner designs that would obviate the need for such instru- ically increased administrative burden compared with our design mentation, often shows the way toward more efficient and econom- (a relatively simple, self-contained kernel module that has but one ical designs. Practical OS security engineering offers an an im- task: monitoring a set of hooks, which we examine in more detail pressive gallery of ideas that started as “hacks” (OpenWall, PaX, in our technical reports [38, 37]). As explained below, in that sys- LIDS, BSD Jails and Linux VServer) and ended up informing in- tem we focus on computation rather than code. In particular, we dustrial solutions like ExecShield, GrSecurity hardening patches, focus on detecting violations of an observed, known invariant of LSM, SELinux, and AppArmour. the correct Linux kernel operation. Finally, although one could argue that the components necessary Rootkit programming Historically, rootkits and exploits tended to support VMMs now exist in commodity hardware, using these to be mistakenly associated with malicious, foreign code intro- capabilities entails warping legacy systems around them, i.e., either duced into the system; consequently, anti-rootkit efforts focused explicitly rewriting kernels (paravirtualization), or writing device on detecting a foreign body of code one way or another. This mis- emulation layers: a heavier proposition than instrumentation-based taken assumption has been challenged by original hacker research self-analysis. We next consider some additional shortcomings of publications [14, 51, 29, 16] that propose flexible exploit program- the VMM-based monitoring philosophy. ming techniques that involved no foreign code, and has hopefully been defeated for good by [44, 13], which made it clear to the aca- demic community that they were dealing with a Turing-complete 2. WHAT EXACTLY IS WRONG WITH VIR- programming and execution model rather than an ad-hoc collection TUALIZATION? of obscure hacks. As organizations increase their adoption of virtualization envi- Hund et al. [22] have further generalized and applied this model ronments, and with the current industry focus on information se- to rootkit programming. They stress that the essence of a rootkit curity, it is natural to wonder just how a virtualization framework was not malicious code but rather malicious computation effected might pull double duty by improving a security posture as well as either exclusively or largely with the unanticipated use of existing, easing management burden and infrastructure costs. Of course, trusted code integral to the exploited system. merely running a virtualized environment does not automatically We note that the art of co-opting and re-using existing (and there- entail a guarantee of increased security. As we discuss below, even fore inherently more stable) code has been an established pattern of 1 the basic isolation properties of a VM framework are questionable; rootkit programming since its early days. In fact, we believe that it remains entirely unclear whether a VMM can get the job of iso- it would fair to speak of Rootkit Design Patterns, such as “context- lation (a job that OS designers and microkernel researchers have based hot patching”, “multi-layer interception”, and others [33, 48, grappled with for years) correct, especially in the absence of hard- 32], all dedicated to runtime manipulation of both live code and ware primitives for this purpose.2 We also agree with the sentiment live data while lowering the probability of a crash. expressed in recent work [6, 11, 24, 40, 17] (some of it our own) that current VM implementations actually use a flawed event trap- 1.3 Philosophy on Self-Monitoring and Same- ping framework that fails to capture events of natural and signifi- layer Attacks cant interest to security policies. Essentially, our previous work [38, 37] proposes an alternative to VM-based monitoring that uses a ring 0 technique for catching