Non-intrusive, Out-of-band and Out-of-the-box Systems Monitoring in the Cloud Sahil Suneja Canturk Isci Vasanth Bala University of Toronto IBM T.J. Watson Research IBM T.J. Watson Research [email protected] [email protected] [email protected] Eyal de Lara Todd Mummer t University of Toronto IBM T.J. Watson Research [email protected] [email protected] ABSTRACT Categories and Subject Descriptors The dramatic proliferation of virtual machines (VMs) in dat- K.6.4 [Management of Computing and Information acenters and the highly-dynamic and transient nature of VM Systems]: System Management—Centralization/ decentral- provisioning has revolutionized datacenter operations. How- ization; D.4.7 [Operating Systems]: Organization and ever, the management of these environments is still carried Design—Distributed systems;C.5.0[Computer System out using re-purposed versions of traditional agents, origi- Implementation]: General; C.4 [Performance of Sys- nally developed for managing physical systems, or most re- tems]: Design studies cently via newer virtualization-aware alternatives that re- quire guest cooperation and accessibility. We show that these existing approaches are a poor match for monitoring Keywords and managing (virtual) systems in the cloud due to their de- Virtualization; Virtual Machine; Cloud; Data Center; Mon- pendence on guest cooperation and operational health, and itoring; Analytics; Agentless; VMI their growing lifecycle management overheads in the cloud. In this work, we first present Near Field Monitoring (NFM), our non-intrusive, out-of-band cloud monitoring and analyt- 1. INTRODUCTION ics approach that is designed based on cloud operation prin- Cloud computing and virtualization technologies are dra- ciples and to address the limitations of existing techniques. matically changing how IT systems operate. What used NFM decouples system execution from monitoring and ana- to be a relatively static environment, with fixed physical lytics functions by pushing monitoring out of the targets sys- nodes, has quickly transformed into a highly-dynamic en- tems’ scope. By leveraging and extending VM introspection vironment, where (clusters of) virtual machines (VMs) are techniques, our framework provides simple, standard inter- programmatically provisioned, started, replicated, stopped faces to monitor running systems in the cloud that require no and deprovisioned with cloud APIs. VMs have become the guest cooperation or modification, and have minimal effect processes of the cloud OS, with short lifetimes and a rapid on guest execution. By decoupling monitoring and analyt- proliferation trend [24]. ics from target system context, NFM provides “always-on” While the nature of data center operations has changed, monitoring, even when the target system is unresponsive. the management methodology of these (virtual) machines NFM also works “out-of-the-box” for any cloud instance as has not adapted appropriately. Tasks, such as performance it eliminates any need for installing and maintaining agents monitoring, compliance and security scans, and product dis- or hooks in the monitored systems. We describe the end-to- covery amongst others are carried out using re-purposed ver- end implementation of our framework with two real-system sions of tools originally developed for managing physical sys- prototypes based on two virtualization platforms. We dis- tems or via newer virtualization-aware alternatives that re- cuss the new cloud analytics opportunities enabled by our quire guest cooperation and accessibility. These approaches decoupled execution, monitoring and analytics architecture. require a communication channel, i.e., a hook,intotherun- We present four applications that are built on top of our ning system, or the introduction of a software component, framework and show their use for across-time and across- i.e., an agent,withinthesystemruntime.Therearetwokey system analytics. problems with the existing approaches: First, proliferation of VMs and their ephemeral, short- lived nature, makes the cost of provisioning and maintaining hooks and agents a major pain point. Moreover, the mon- Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed itoring extensions “pollute” the end-user system, intervene for profit or commercial advantage and that copies bear this notice and the full cita- with guest execution, and potentially open up new points of tion on the first page. Copyrights for components of this work owned by others than vulnerability. A recent observation from Amazon Web Ser- ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- vices [3] highlights how agent operation and maintenance publish, to post on servers or to redistribute to lists, requires prior specific permission issues can impact managed systems. In this case, an incom- and/or a fee. Request permissions from [email protected]. plete maintenance update for DNS configuration in some of SIGMETRICS’14, June 16–20, 2014, Austin, Texas, USA. Copyright 2014 ACM 978-1-4503-2789-3/14/06 ...$15.00. the agents, coupled with a memory leak issue led to a per- http://dx.doi.org/10.1145/2591971.2592009 . formance degradation for part of Amazon storage services. This observation drives the first research question we address VM console-like interface enabling administrators to query in our work: How can we perform IT operations (monitor- system state without having to log into guest systems, as ing, compliance, etc.) without relying on guest cooperation well as a handy “time travel” capability for forensic analy- or in-VM hooks? sis of systems, and (iv) a hypervisor-paging aware out-VM Second, existing techniques work only so long as the mon- virus scanner that demonstrates how across-stack knowledge itored systems function properly, and they fail once a system of system state can dramatically improve the operational becomes unresponsive—exactly when such monitoring infor- efficiency of common management applications like virus mation is the most valuable. A recent Google outage [12] scan. Finally, we present a quantitative evaluation showcas- presents a prime example to the effect of this limitation, ing NFM’s high accuracy, monitoring frequency, reliability where a significant portion of Google’s production systems and efficiency, as well as low impact on monitored systems. became unresponsive due to a dynamic loader misconfigu- The rest of the paper is organized as follows. Section ration. As a result of this, none of the in-system agents 2summarizesthetechniquesemployedtodayforenterprise could publish data outside, neither was it possible to log in virtual systems monitoring. Section 3 gives a high level view to the impacted systems for manual diagnosis. Thus,it was of our solution architecture and discusses its benefits over extremely difficult to get system information when it was existing alternatives. Section 4 gives the implementation most crucial. This observation drives our second research details. Section 5 describes the applications we have built to question: How can we monitor and manage systems even demonstrate NFM’s capability. Section 6 evaluates NFM’s when they become unresponsive or are compromised? performance. Section 7 presents related work and Section 8 To address these research challenges, this paper intro- offers our conclusions. duces Near Field Monitoring (NFM), a new approach for system monitoring that leverages virtualization technology 2. EXISTING TECHNIQUES to decouple system monitoring from system context. NFM System monitoring has been a major part of enterprise IT extends VM introspection (VMI) techniques, and combines operations. We categorize the various techniques employed these with a backend cloud analytics platform to perform today as follows: monitoring and management functions without requiring ac- cess into, or cooperation of the target systems. NFM crawls 1. Application-specific in-VM agents for monitoring guest VM memory and disk state in an out-of-band manner from systems. outside the guest’s context, to collect system state which is 2. Application specific hooks for remotely accessing and mon- then fed to the analytics backend. The monitoring functions itoring systems. simply query this systems data, instead of accessing and in- 3. Limited black-box metrics collected by the virtualization truding each running system. In stark contrast with existing layer without guest cooperation. techniques, NFM seamlessly works even when a system be- comes unresponsive (always-on monitoring), and does not 4. General purpose agents or hooks that provide generic in- require the installation and configuration of any hooks or VM information through the virtualization layer. agents in the target system (it works out-of-the-box). Unlike Existing cloud monitoring and management solutions em- the in-VM solutions that run within the guest context and ploy one or more of the above methods to deliver their ser- compete for resources allocated to the guest VMs, NFM is vices. For example, Amazon’s CloudWatch [2] service falls non-intrusive and does not steal guests’ cycles or interfere under the third category in its base operation, while it can be with their actual operation. We believe our approach lays extended by the end users with in-VM data providers (as in the foundation for the right way of systems monitoring in the first category) to provide deeper VM-level information. the cloud; very much like how we monitor processes