Addressing SMTP-based Mass-Mailing Activity Within Enterprise Networks David Whyte, P. C. van Oorschot, Evangelos Kranakis School of Computer Science Carleton University, Ottawa, Canada dlwhyte, paulv, [email protected] Abstract ple, the MyDoom mass-mailing worm at its peak was responsible for one in every twelve Internet email mes- Malicious mass-mailing activity on the Internet is a sages [6]. The majority of mass-mailing worms employ serious and continuing threat that includes mass-mailing the same infection delivery mechanism: a Simple Mail worms, spam, and phishing. A mechanism commonly Transfer Protocol engine (SMTP-engine), which turns an used to deliver such malicious mass mail is an SMTP- infected system into a malicious mail server. As mail engine, which turns an infected system into a malicious server filtering techniques become more effective, spam- mail server. We present a technique that enables, within mers and phishers are resorting to hijacking ordinary PCs a single mailing attempt in many popular network envi- (thereafter zombies) and using built in SMTP-engines or ronments, detection and containment of (even zero-day) mail proxy programs to send malicious mail without the SMTP-engine based mass-mailing activity. Contrary to owner's knowledge [3, 12]. In fact, it has been estimated other mass-mailing detection techniques our approach is that 80% of spam is sent by spam zombies [12]. content independent and requires no attachment process- In this paper, we exploit the interaction between ing, network traffic correlation, statistical measures, or SMTP-engines and DNS servers to provide a new method system behavioral analysis. It relies instead on the obser- to detect malicious mass-mailing activity within an en- vation of DNS MX queries within the enterprise network. terprise network. In short, SMTP-engine infected clients This stateless detection technique requires minimal com- typically request Mail Exchanger (MX) records from a putational resources making it ideally suited for real-time DNS server (either their local DNS server or DNS servers wire-speed deployment. outside the network boundary) in order to locate the mail servers that can deliver the malicious mail to their in- tended victims. While some legitimate client systems 1 Introduction run their own email servers locally, most enterprise envi- ronments use perimeter mail servers to send and receive email.2 In this scenario, only the corporate mail servers Internet users are inundated by a steady stream of within the enterprise network are generally expected to emails infected with malicious code, unwanted product query DNS servers for MX records (see further discus- advertisements, and requests for personal information sion, including exceptions in Section 4.2). from criminals masquerading as legitimate entities to en- able the commission of fraudulent activity. The use of Our Contributions. We present a technique, imple- gateway anti-virus (and per client) software and spam fil- mented and tested with a software prototype, to detect ters offers some measure of protection. However, these and quarantine SMTP-engine mass-mailing based solely perimeter defences often fail to detect zero-day worms on the observation of a DNS MX record request from and viruses, often quarantine legitimate emails misidenti- client systems. No modeling or statistical measurement fied as spam, and do not address perhaps the most preva- of user or network behavior is required. Furthermore, it lent infection method: users unwittingly opening mali- does not rely on attachment scanning, allowing detection of malicious text-based emails with hypertext embedded cious attachments. A strong argument can be made that 3 the best chance to detect and quarantine malicious email links to malicious websites. To validate these claims, occurs before it is sent outside of the enterprise network. including spam). To date, the use of mass-mailing worms has been the 2This allows for gateway anti-virus software at the network perime- fastest way to propagate malicious mail.1 For exam- ter and lower cost (e.g. maintenance, support, policy enforcement) cor- porate email. 1We define malicious mail as unwanted email unwittingly sent by 3These websites infect a system by sending malicious code through a compromised system whether or not it contains malicious code (i.e. website content retrieved by the client system. 1 we performed tests in an isolated test network with a live every email for dangerous attachments. They employ vir- mass-mailing worm. tual machine clusters, host-based intrusion detection, and Our anomaly-based approach is appealing for a num- email-worm vaccine aware Mail Transfer Agents. ber of reasons: Hu et al. [10] present an application of the PAIDS (ProActive Intrusion Detection System) detec- Speed 1. : in certain network environments the possi- tion paradigm using a prototype system called BESIDES bility to detect and contain an SMTP-engine before which detects mass-mailing viruses. PAIDS employs a single malicious email message can be sent. two general techniques: comparing a system's behavior 2. Detection and containment of zero-day mass- against its security policy (behavior skewing) and iso- mailing worms: possible because the approach does lating illegal system behaviors in a virtual environment not rely on existing worm signatures. (cordoning). Their prototype detected a number of real 3. Impact to quarantined system: once identified as mass-mailing worms with a low false positive rate. How- a malicious mass-mailer, only SMTP activity (port ever, their implementation is deployed at SMTP servers 25) will be blocked on the system allowing all other which would fail to detect SMTP-engine activity. SMTP- user activity to proceed unhindered. engines bypass network mail servers (and even in some 4. Low-false positive rate: empirical analysis (see Sec- cases local DNS servers) making network-based detec- tion 4.2) suggests that client MX record requests are tion techniques necessary. rare for most users.4 Gupta et al. [9] use specification-based anomaly de- 5. Ease of deployment: the approach is network-based, tection to detect email viruses. Their approach looks for runs on commodity hardware, and relies on the ob- increases in mail traffic from clients to mail servers over servation of a protocol found in all networks (i.e. a threshold determined during a training period. Specif- DNS). ically, the statistics of send and deliver transitions in a state machine are maintained for both individual clients Organization. Section 2 discusses related work. Sec- and the entire collection of clients within the network. tion 3 outlines the basic approach. Section 4 presents an Using a series of simulated experiments they detected empirical analysis of client MX record request activity. stealthy (e.g. polymorphic) viruses with a low false posi- Section 5 discusses our prototype and its performance in tive rate. an isolated worm test network. Section 6 contrasts our Wong et al. [20] performed an empirical study on technique with others. We conclude in Section 7. mass-mailing worm behavior using network traffic traces from a college campus. The characteristics of two mass- 2 Related Work mailing worms with respect to DNS activity and TCP traffic flows were studied. They found that changes in Zou et al. [23] developed a mass-mailing worm model network activity from infected hosts allowed for interest- by profiling the user behavior of email checking times ing detection possibilities. They propose that a more in- and email attachment opening probabilities. They ana- depth investigation of monitoring and containing mass- lyzed the impact of selective immunization defense, that mailing worms using DNS servers should be performed entails making the most connected email users' systems as it holds promise as a way to slow down propaga- immune to an email worm. Their results reveal that al- tion. One important observation was that defences de- though a power law topology enables a worm to spread signed for monitoring SMTP servers will not work well more quickly, it also allows for faster containment. Their for mass-mailing worms as they have their own SMTP- work provides an email worm model that incorporates engines. user behavior and offers some insight into worm prop- Ishibashi et al. [11] employ a technique that uses agation on a number of network topologies. The same a Bayesian inference method to calculate and assign authors propose [22] a multi-step feedback email defence a value to the suspiciousness of specific domain name mechanism to detect malicious email within an enterprise queries from individual hosts. This method assumes that network; and suggest the use of a honeypot to detect out- there is partial prior information about the normal char- going viruses. acteristic domain name queries from the network. Sig- Sidiroglou et al. [17] propose an architecture to detect natures are manually derived from the query content of zero-day worms and viruses, which intercepts and scans suspected worm infected hosts. Hosts that send domain requests that match the signature query content are as- 4In a university network of about 300 users over one week, we found sumed to be infected with a mass-mailing worm. Their only 5 anomalous MX record queries from client systems. While in technique is not suitable for detecting zero-day worms in most corporate environments the deployed software application base- line differs substantially from a university network, the greater software real-time as it requires both manual analysis and a prede- diversity in the latter makes it a good test environment. termined signature to identify suspected worm activity. 2 Whyte et al. [18] used DNS activity to detect the pres- ence of scanning worms within an enterprise network. Internet The observation of connections outside the network not Router preceded by a DNS query was considered anomalous and Enterprise Network a strong indicator of scanning worm activity. They hy- pothesized that MX queries from client systems could in- Mail DNS dicate mass-mailing worm infection, but recognized that the detection and containment of mass-mailing worms 2 MX Query would require the collection of different network data 3 Email (i.e.