An Anomaly-based Botnet Detection Approach for Identifying Stealthy Botnets Sajjad Arshad1, Maghsoud Abbaspour1, Mehdi Kharrazi2, Hooman Sanatkar1 1Electrical and Computer Engineering Department, Shahid Beheshti University; G.C; Tehran, Iran 2 Department of Computer Engineering, Sharif University of Technology; Tehran, Iran [email protected], [email protected], [email protected], [email protected] Abstract—Botnets (networks of compromised computers) resistant to the changes of the C&C mechanisms, we are often used for malicious activities such as spam, click should study inherent characteristics of botnet fraud, identity theft, phishing, and distributed denial of behaviors. Bots connect to the C&C channel and service (DDoS) attacks. Most of previous researches have execute the received commands. Bots belonging to the introduced fully or partially signature-based botnet same botnet receive the same commands that causes detection approaches. In this paper, we propose a fully anomaly-based approach that requires no a priori them having similar netflows characteristics and knowledge of bot signatures, botnet C&C protocols, and performing same attacks. There are netflows that C&C server addresses. We start from inherent present communication between bots and C&C servers characteristics of botnets. Bots connect to the C&C such as binary downloading and sending spam. Also, channel and execute the received commands. Bots there are several types of attacks like scanning and belonging to the same botnet receive the same commands distributed denial of service (DDoS). Our method is that causes them having similar netflows characteristics based on detecting these anomalous behaviors and and performing same attacks. Our method clusters bots finding meaningful relation between these activities to with similar netflows and attacks in different time detect set of bots inside the monitored network. This windows and perform correlation to identify bot infected hosts. We have developed a prototype system and paper makes the following main contributions: evaluated it with real-world traces including normal • We propose an anomaly-based method that traffic and several real-world botnet traces. The results requires no a priori knowledge of bot show that our approach has high detection accuracy and signatures, botnet C&C protocols, and the C&C low false positive. server addresses. • Keywords—Botnet; Netflow; Clustering; Anomaly-based We illustrate a method that can detect bots in Detection the monitored network in real-time. • In addition to detect bots with malicious I. INTRODUCTION activities (e.g. scanning, DDoS), our method Botnet is a collection of compromised hosts (bots) can detect bots that do not perform malicious that are under control of an attacker (botmaster). Bots activities. But, our approach detects bots with are used to send spam emails, host phishing web sites, malicious activities so fast. • cooperate in distributed denial of service (DDoS) We have developed a prototype system based attacks, and other kinds of malicious activities. on our method and evaluated it with real-world Botmaster needs a command and control (C&C) network traces including normal traffic and channel to command the bots and coordinate malicious several real-world botnet traces. activities. Most of botnets C&C channels are using IRC The rest of the paper is organized as follows. We (Internet Relay Chat) protocol. In this protocol, review the related works in section 2. Section 3 botmaster has a real-time communication with the bots. describes the architecture and implementation of the There are also a few botnets that use the HTTP protocol approach. Section 4 evaluates the effectiveness of the for C&C channels [1][2]. In HTTP-based C&C, the approach on various network traces and finally the botmaster does not communicate directly with the bots. paper is concluded in section 5. Instead, the bots periodically contact the C&C server to obtain their commands. These two protocols provide a II. RELATED WORKS centralized C&C mechanism. The main disadvantage of There are several researches that propose different centralized C&C mechanism is the single-point-of- botnet detection approaches. Binkley and Singh [5] failure problem. For example, if IRC or HTTP server is combines IRC statistics and TCP work weight to detect taken down, the botmaster will not be able to IRC-based botnets. Karasaridis [6] used IRC netflows communicate with the bots anymore. Thus, botmasters and scanning activities to detect IRC botnet controllers. began using peer-to-peer (P2P) protocols for C&C Livadas [7] proposed a machine learning based channels. Currently, Storm Worm [3] and Nugache [4] approach which uses network-level traffic features of are the most popular P2P botnets. chat protocols for botnet detection. Rishi [8] is a For designing a botnet detection approach that is signature-based IRC botnet detection approach that Netflows (C&C Communication, Spam Emails, Binary Updating) Attacks (Scanning, DDoS, …) Scanning DDoS Spam Binary Updating C&C Scanning DDoS Time Spam Binary Updating Bot C&C TW1 TW2 TW3 TW4 TW5 Time Bot Figure 1. Similarity in bots behaviors in consecutive time windows finds suspicious nicknames by analyzing IRC-based by using database produced by Domain-IP Mapping botnets traffic. The above approaches are used for only component. Then Netflow Clustering and Alert detecting IRC-based botnets; whereas we do not have Clustering components cluster non-filtered netflows and assumption about botnet C&C protocols. BotHunter [9] alerts. Finally at the end of each time window uses a user-defined bot infection model in order to correlation engine correlates the generated alert clusters correlate alerts (e.g. scanning). BotSniffer [10] is a and netflow clusters in order to detect bot infected centralized botnet C&C detection approach that hosts. The following sections illustrate architecture of performs group analysis across multiple hosts through our approach in detail. Our botnet detection approach different time windows. But, it is used for detecting operates on online traffic and detects bots in real-time. centralized botnets. BotMiner [11] clusters hosts based The following sections will illustrate each component in on their netflow-level statistics (e.g., the byte count, the details. packet count, etc.) and their malicious activities (e.g., A. Traffic Dispatching scanning) and then correlates clusters to find bots. Nevertheless, it works offline and cannot detect bots This component has been designed for separating quickly. BotProbe [12] proposes an active botnet different types of traffic of Domain-IP Mapping, probing technique that detects bots by distinguishing Netflow Generating and Alert Generating components. botnet C&C dialogs from human-human conversations. It delivers DNS traffic to Domain-IP Mapping and TCP The aforementioned approaches are mostly signature- traffic to Netflow Generating component. Also TCP, based and cannot find unknown botnets. UDP and ICMP traffic is delivered to Alert Generating component. Irrelevant traffics like DHCP and ARP will III. BOTNET DETECTION APPROACH be filtered out in order to increase performance. In this section, we illustrate a new botnet detection B. Domain-IP Mapping approach whose goal is to detect groups of bot infected As discussed above, the bots can use the IPs or DNS hosts which belong to the same botnet in a monitored names hard coded in their binary file in order to connect network. Bots belonging to the same botnet have to the C&C servers. This component has been designed similar behaviors during a time window and this to map the domains and IPs to each other. This similarity continues during consecutive time windows component processes DNS queries and responses in as shown in Fig. 1. Our approach finds behavior order to store in database the IP addresses in similarity of hosts in different properties such as correspondence with the queried domain names. This netflow information through a predefined time window database only is used to filter netflow to the sites in the and tries to detect bots by correlating these similar white list as described in Netflow Filtering section. behaviors between different time windows. Fig. 2 shows the architecture of our botnet detection C. Netflow Generating approach, which consists of nine interconnected This component processes TCP traffic and generates the components that analyze traffic online. Traffic netflows between hosts. Most of the routers like Cisco dispatching component delivers traffic to Domain-IP and Juniper have the capability of generating netflows. Mapping, Netflow Generating and Alert Generating Also open source tools like ARGUS exist. We components. Domain-IP Mapping component maps the developed our own efficient tool to generate netflows. DNS domains to corresponded IPs for filtering In addition to simple netflows, our tool is able to purposes. Netflow Generating component generates generate partial netflows (netflows that is not finished TCP netflows between hosts. Alert Generating inside a time windows, is considered as a completed component reports the malicious activities of the hosts netflow, the remained part of the netflow is considered like scanning. The following five components perform as another netflow in next time windows). The most at the end of each time window. Alert Filtering important property of our tool is its efficiency and component filters useless alerts generated by Alert customized netflow record generating. Currently,