GQ: Practical Containment for Measuring Modern Malware Systems Christian Kreibich,† Nicholas Weaver,† Chris Kanich,* Weidong Cui,§ and Vern Paxson†‡ TR-11-002 May 2011 Abstract Measurement and analysis of modern malware systems such as botnets relies crucially on execution of specimens in a setting that enables them to communicate with other systems across the Internet. Ethical, legal, and technical constraints, however, demand containment of resulting network activity in order to prevent the malware from harming others while still ensuring that it exhibits its inherent behavior. Current best practices in this space are sorely lacking: measurement researchers often treat containment superficially, sometimes ignoring it altogether. In this paper we present GQ, a malware execution “farm” that uses explicit containment primitives to enable analysts to develop containment policies naturally, iteratively, and safely. We discuss GQ’s architecture and implementation, our methodology for developing containment policies, and our experiences gathered from six years of development and operation of the system. † ICSI, 1947 Center St., Ste. 600, Berkeley, CA 94704 * UC San Diego, San Diego, CA § Microsoft Research, Redmond, Washington ‡ UC Berkeley, Berkeley, CA 94704 This work was partially supported by funding provided to ICSI through National Science Foundation grant CNS- 0433702 (“CCIED: Collaborative Center for Internet Epidemiology and Defenses”). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors or originators and do not necessarily reflect the views of the National Science Foundation. 1 Introduction First, we develop an architecture for a malware exe- cution platform that provides a natural way to configure Despite strenuous efforts of security vendors and re- and enforce containment policies by means of contain- searchers, the Internet is not becoming a safer place. Re- ment servers, establishing first-order primitives for ef- cent industry reports state that 60,000 new samples of fecting containment on a per-flow granularity. We began malware—software created with the intent to conduct our work on GQ during the peak of the “worm era” [9], malicious activity—appear every day [19]. In this en- and initially focused on capturing self-propagating net- vironment, malware analysts face an increasingly chal- work worms. Accordingly, we highlight both the sys- lenging task of separating chaff—uninteresting rehashes tem’s original architectural features that have “stood the of known malware strains—from truly novel behavior, test of time,” as well as the shortcomings the emerged as in order to enable close study of the novelty to under- our usage of GQ began to shift towards broader forms of stand its goals and determine how to combat it. At malware. the same time, analysts must also follow the activity Second, we describe a principled—albeit largely in known, major outbreaks—akin to infiltrating criminal manual—approach to the development of containment organizations—to stay abreast of recent developments. policies to ensure prevention of harm to others. Our goal These tasks rely crucially on execution: “letting loose” here is to draw attention to the issue, highlight poten- malware samples in an execution environment to study tial avenues for future research, and point out that far their behavior, sometimes only for seconds at a time from being a chore, containment development can ac- (e.g., to understand the bootstrapping behavior of a bi- tively support the process of studying malware behavior. nary, perhaps in tandem with static analysis), but poten- Third, we present insights from our extensive oper- tially also for weeks on end (e.g., to conduct long-term ational experience [4,9, 13, 17, 18, 24] in developing botnet monitoring via “infiltration” [13]). containment policies that serve to illustrate the effective- This need to execute malware samples in a laboratory ness, importance, and utility of a principled approach to setting exposes a dilemma. On the one hand, uncon- containment. strained execution of the malware under study will likely We begin by reviewing the approaches to contain- enable it to operate fully as intended, including embark- ment framed in previous work (x 2), which provides fur- ing on a large array of possible malicious activities, such ther context for our discussion of the issue of contain- as pumping out spam via email or the web, contribut- ment (x 3). We then present GQ’s design goals (x 4), ing to denial-of-service floods, conducting click fraud, architecture (x 5), and implementation (x 6), illustrating or obscuring other attacks by proxying malicious traffic. both initial as well as eventual features as a consequence On the other hand, if executed in complete isolation, the of our continued desire to separate policy from mech- malware will almost surely fail to operate as intended, anism and increase general applicability of the system. since it cannot contact external servers via its command- Finally, we report on our operational experiences over and-control (C&C) channel in order to obtain input data GQ’s lifetime (x 7), discuss our current methodology for or execution instructions. developing containment policies and containment’s gen- Thus, industrial-strength malware analysis requires eral feasibility (x 8), and offer concluding thoughts (x 9). containment: execution environments that on the one hand allow malware to engage with the external world 2 Related Work to the degree required to manifest their core behavior, but doing so in a highly controlled fashion to prevent the A large body of prior work focuses on malware execu- malware from inflicting harm on others. Despite the crit- tion for the purpose of understanding the modi operandi ical importance of proper containment, researchers of- of binary specimens. Historically, “honeyfarms” were ten treat the matter at best superficially, sometimes ig- the first platforms supporting large-scale malware execu- noring it altogether. We perceive the root cause for this tion. These systems focused in particular on worm ex- shortcoming as arising largely from a lack of technical ecution, with the “honey” facet of the name referring to tools to realize sufficiently rich containment, along with a honeypot-style external appearance of presenting a set a perception of containment as a chore rather than an of infectible systems in order to trap specimens of global opportunity. To improve the state of affairs, we present worm outbreaks early in the worm’s propagation. Vrable GQ, a malware execution “farm” we have developed and et. al designed the Potemkin honeyfarm as a (highly) operated regularly for six years. GQ’s design explic- purpose-specific prototype of a worm honeyfarm that ex- itly focuses on enabling the development of precise-yet- plored the scalability constraints present when simulat- flexible containment policies for a wide range of mal- ing hundreds of victim machines on a single physical ma- ware. Drawing upon GQ’s development and operation chine [28]. Their work also framed the core issue of con- over this period, we make three contributions: tainment, but leveraged a major simplification that (sim- 1 ple) worms can potentially provide, namely that one can They also recognize the importance of executing mal- observe worm propagation even when employing a very ware safely. In contrast to GQ, the authors designed Bot- conservative containment policy of redirecting outbound lab specifically to study email spam, for which they em- connections to additional analysis machines in the hon- ployed a static containment policy: “[T]raffic destined to eyfarm. Jian and Xu’s Collapsar, a virtualization-based privileged ports, or ports associated with known vulnera- honeyfarm, targeted the transparent integration of hon- bilities, is automatically dropped, and limits are enforced eypots into a distributed set of production networks [11]. on connections rates, data transmission, and the total The authors focused on the functional aspects of the sys- window of time in which we allow a binary to execute.” tem, and while Collapsar’s design supports the notion of In addition, the authors ultimately concluded that con- assurance modules to implement containment policies, tainment poses an intractable problem, and ceased pur- in practice these simply relied on throttling and use of suing their study: “Moreover, even transmitting a “be- “Inline Snort” to block known attacks. nign” C&C message could cause other, non-Botlab bot With the end of the “worm era,” researchers shifted fo- nodes to transmit harmful traffic. Given these concerns, cus from worm-like characteristics to understanding mal- we have disabled the crawling and network fingerprint- ware activity more broadly. Bayer et al. recognized the ing aspects of Botlab, and therefore are no longer ana- significance of the containment problem in a paper intro- lyzing or incorporating new binaries.” ducing the Anubis platform [2], which has served as the Researchers have also explored malware execution in basis for numerous follow-up studies: “Of course, run- the presence of complete containment (no external con- ning malware directly on the analyst’s computer, which nectivity). These efforts rely on mechanisms that aim is probably connected to the Internet, could be disas- to mimic the fidelity of communication between bots trous as the malicious code could easily escape and in- and external machines. The SLINGbot system emulates fect other machines.” The remainder of the paper, how- bot behavior rather than allowing the traffic of live mal- ever, focuses on technical aspects of monitoring the spec- ware on the commodity Internet [10]; clearly, such an imen’s interaction with the OS; they do not address the approach can impose significant limits on the obtainable safety aspect further. depth of analysis. The DETER testbed [21] relies upon The CWSandbox environment shares the dynamic experimenters and testbed operators to come to a con- analysis goals of Anubis, using virtualization or po- sensus on the specific containment policy for a given ex- tentially “raw iron” (non-virtualized) execution instead periment.