DOCSLIB.ORG

Botnet Detection a Numerical and Heuristic Analysis

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Botnet Detection a Numerical and Heuristic Analysis University of Minho School of Engineering Luís Miguel Ferreira Costa Mendonça Botnet Detection A Numerical and Heuristic Analysis January 2012 University of Minho School of Engineering Luís Miguel Ferreira Costa Mendonça Botnet Detection A Numerical and Heuristic Analysis Master Thesis Master in Communication Networks and Services Engineering Research oriented by: Henrique Manuel Dinis dos Santos January 2012 “Information is not knowledge” Albert Einstein Botnet Detection: A Numerical and Heuristic Analysis i Table of Contents 1. Introduction .............................................................................................. 1 1.1 Goals .................................................................................................................... 2 1.2 Research Methodology ........................................................................................... 3 1.3 Document Organization and Notes of Attention ....................................................... 4 1.4 Research Calendar ................................................................................................ 5 2. Botnet Characterization ............................................................................. 7 2.1 Definition ............................................................................................................... 7 2.2 Propagation and Recruitment ................................................................................. 7 2.3 Command and Control ........................................................................................ 11 2.4 Maintenance and Availability ................................................................................ 20 2.5 Attacks ................................................................................................................ 23 2.6 Botnet History: The Malware Chronicles ............................................................... 27 2.7 Conclusions ......................................................................................................... 29 3. Botnet Detection and Analysis .................................................................31 3.1 Honeynets, Honeypots and SPAM Traps .............................................................. 31 3.2 Signature-based Methods ..................................................................................... 31 3.3 DNS-based Methods ............................................................................................ 32 3.4 Anomaly-based Methods ...................................................................................... 34 3.5 Related Work ....................................................................................................... 34 3.6 Conclusions ......................................................................................................... 36 4. Botnet Anomaly-Based Detection .............................................................39 4.1 Bot Behavior ........................................................................................................ 41 4.2 Botnet Behavior ................................................................................................... 42 4.3 Useful Traffic Data for Anomalous-Based Detection .............................................. 42 Botnet Detection: A Numerical and Heuristic Analysis ii 4.4 Conclusions ......................................................................................................... 44 5. Proposed Model and Prototype ...............................................................47 5.1 Initial Research .................................................................................................... 47 5.2 Netflow as the Traffic Feature Source ................................................................... 55 5.3 Traffic Attributes Heuristics .................................................................................. 57 5.4 Communications Fingerprints .............................................................................. 61 5.5 Detection Framework Implementation .................................................................. 63 5.6 Prototype Implementation .................................................................................... 70 5.7 Conclusions ......................................................................................................... 72 6. Experimental Setup, Results and Analysis ................................................73 6.1 Test Datasets Characterization ............................................................................. 73 6.2 Prototype Test Environment ................................................................................. 76 6.3 Testing Methodology ............................................................................................ 78 6.4 Detection Sensitivity and Specificity Analysis ........................................................ 82 6.5 Performance and Storage: Real-Time? .................................................................. 89 6.6 Final Results and Conclusions.............................................................................. 91 7. Discussion, Future Work and Conclusions ...............................................95 8. References .............................................................................................99 9. Apendix A: Botnet History ..................................................................... 107 Botnet Detection: A Numerical and Heuristic Analysis iii List of Figures Figure 1: IRC Botnet Topology ............................................................................................ 12 Figure 2: HTTP Botnet Topology ......................................................................................... 13 Figure 3: HTTP Botnet Hierarchic Topology ........................................................................ 14 Figure 4: P2P Botnet Topology ........................................................................................... 15 Figure 5: Number of distinct hosts with scan-like activity ..................................................... 49 Figure 6: Distinct source IPs per number of bytes and packets ........................................... 51 Figure 7: Number of distinct Source IPs with crowd-like behavior per ten-minute period ...... 52 Figure 8: Number of distinct hosts connecting to SQL Server ports per bytes per packet ..... 53 Figure 9: Communication Fingerprints (example) ................................................................ 62 Figure 10: Scanner Communication Fingerprints ................................................................ 62 Figure 11: Malicious Host Fingerprints ............................................................................... 63 Figure 12: HTTP Server Fingerprints .................................................................................. 63 Figure 13: Clustering Database Tables ............................................................................... 69 Figure 14: Padre Integrated Development Environment ...................................................... 71 Figure 15: SQL Server Scripts Test Examples ..................................................................... 71 Figure 16: Typical University of Minho 24-hour traffic shown in Nfsen (12th January 2011) . 74 Figure 17: Netflow Capture System .................................................................................... 74 Figure 18: Distinct active hosts for each hourly period (16th November 2011) .................... 76 Figure 19: Distinct active hosts for each hourly period present in blacklists (16th November 2011) ................................................................................................................................ 76 Figure 20: nfsen "testplugin" command-line tool ................................................................ 77 Figure 21: ROC Plot for CAST variation ............................................................................... 83 Figure 22: First-Pass Detection - Sensitivity versus Specificity .............................................. 84 Figure 23: MNACT ROC plots with (left) and without (right) clustering analysis .................... 85 Figure 24: ROC Plot for detection without clustering analysis for MNACT=70 and HAST=950 ......................................................................................................................................... 86 Figure 25: Hosts detected without clustering analysis for MNACT=70 and HAST=950 ......... 87 Figure 26: Hosts detected using clustering analysis for MNACT=70 and HAST=950 ............ 88 Figure 27: ROC Plot for detection with clustering analysis for MNACT=70 and HAST=990 ... 88 Figure 28: Hosts detected using clustering analysis for MNACT=70 and HAST=990 ............ 89 Botnet Detection: A Numerical and Heuristic Analysis iv Figure 29: Performance Evaluation Chart ........................................................................... 90 List of Tables Table 1: Network Traffic Data Source Classification ............................................................ 46 Table 2: Top 5 bytes per packet clusters with higher SQL Server scan activity ..................... 53 Table 3: Netflow Traffic Attributes Selected ......................................................................... 56 Table 4: Sample Communication Fingerpint Values ............................................................ 61 Table 5: Connection Anomaly
Load more

Recommended publications
  • Botnets, Cybercrime, and Cyberterrorism: Vulnerabilities and Policy Issues for Congress
    Order Code RL32114 Botnets, Cybercrime, and Cyberterrorism: Vulnerabilities and Policy Issues for Congress Updated January 29, 2008 Clay Wilson Specialist in Technology and National Security Foreign Affairs, Defense, and Trade Division Botnets, Cybercrime, and Cyberterrorism: Vulnerabilities and Policy Issues for Congress Summary Cybercrime is becoming more organized and established as a transnational business. High technology online skills are now available for rent to a variety of customers, possibly including nation states, or individuals and groups that could secretly represent terrorist groups. The increased use of automated attack tools by cybercriminals has overwhelmed some current methodologies used for tracking Internet cyberattacks, and vulnerabilities of the U.S. critical infrastructure, which are acknowledged openly in publications, could possibly attract cyberattacks to extort money, or damage the U.S. economy to affect national security. In April and May 2007, NATO and the United States sent computer security experts to Estonia to help that nation recover from cyberattacks directed against government computer systems, and to analyze the methods used and determine the source of the attacks.1 Some security experts suspect that political protestors may have rented the services of cybercriminals, possibly a large network of infected PCs, called a “botnet,” to help disrupt the computer systems of the Estonian government. DOD officials have also indicated that similar cyberattacks from individuals and countries targeting economic,
    [Show full text]
  • A the Hacker
    A The Hacker Madame Curie once said “En science, nous devons nous int´eresser aux choses, non aux personnes [In science, we should be interested in things, not in people].” Things, however, have since changed, and today we have to be interested not just in the facts of computer security and crime, but in the people who perpetrate these acts. Hence this discussion of hackers. Over the centuries, the term “hacker” has referred to various activities. We are familiar with usages such as “a carpenter hacking wood with an ax” and “a butcher hacking meat with a cleaver,” but it seems that the modern, computer-related form of this term originated in the many pranks and practi- cal jokes perpetrated by students at MIT in the 1960s. As an example of the many meanings assigned to this term, see [Schneier 04] which, among much other information, explains why Galileo was a hacker but Aristotle wasn’t. A hack is a person lacking talent or ability, as in a “hack writer.” Hack as a verb is used in contexts such as “hack the media,” “hack your brain,” and “hack your reputation.” Recently, it has also come to mean either a kludge, or the opposite of a kludge, as in a clever or elegant solution to a difficult problem. A hack also means a simple but often inelegant solution or technique. The following tentative definitions are quoted from the jargon file ([jargon 04], edited by Eric S. Raymond): 1. A person who enjoys exploring the details of programmable systems and how to stretch their capabilities, as opposed to most users, who prefer to learn only the minimum necessary.
    [Show full text]
  • Fortinet's March Threatscape Report Shows Domination of Ransomware and Troublesome Zero-Day
    Fortinet's March Threatscape Report Shows Domination of Ransomware and Troublesome Zero-Day Rise of Ransomware Is Primarily Driven by Bredolab and Pushdo Botnets SUNNYVALE, CA, Apr 01, 2010 (MARKETWIRE via COMTEX News Network) -- Fortinet(R) (NASDAQ: FTNT) -- a leading network security provider and worldwide leader of unified threat management (UTM) solutions -- today announced its March 2010 Threatscape report showed domination of ransomware threats with nine of the detections in the malware top ten list resulting in either scareware or ransomware infesting the victim's PC. Fortinet observed the primary drivers behind these threats to be two of the most notorious botnet "loaders" -- Bredolab and Pushdo. Another important finding is the aggressive entrance of a new zero-day threat in FortiGuard's top ten attack list, MS.IE.Userdata.Behavior.Code.Execution, which accounted for 25 percent of the detected activity last month. Key threat activities for the month of March include: -- SMS-based Ransomware High Activity: A new ransomware threat -- W32/DigiPog.EP -- appeared in Fortinet's top ten malware list. DigiPog is an SMS blocker using Russian language, locking out a system and aggressively killing off popular applications like Internet Explorer and Firefox until an appropriate code is entered into a field provided to the user. To obtain the code, a user must send an SMS message to the provided number, receiving a code in return. Upon execution, DigiPog registers the user's MAC address with its server. It is the first time that SMS-based ransomware enters Fortinet's top ten list, showing that the rise of ransomware is well on its way.
    [Show full text]
  • The Downadup Codex a Comprehensive Guide to the Threat’S Mechanics
    Security Response The Downadup Codex A comprehensive guide to the threat’s mechanics. Edition 2.0 Introduction Contents Introduction.............................................................1 Since its appearance in late-2008, the Downadup worm has become Editor’s Note............................................................5 one of the most wide-spread threats to hit the Internet for a number of Increase in exploit attempts against MS08-067.....6 years. A complex piece of malicious code, this threat was able to jump W32.Downadup infection statistics.........................8 certain network hurdles, hide in the shadows of network traffic, and New variants of W32.Downadup.B find new ways to propagate.........................................10 defend itself against attack with a deftness not often seen in today’s W32.Downadup and W32.Downadup.B threat landscape. Yet it contained few previously unseen features. What statistics................................................................12 set it apart was the sheer number of tricks it held up its sleeve. Peer-to-peer payload distribution...........................15 Geo-location, fingerprinting, and piracy...............17 It all started in late-October of 2008, we began to receive reports of A lock with no key..................................................19 Small improvements yield big returns..................21 targeted attacks taking advantage of an as-yet unknown vulnerability Attempts at smart network scanning...................23 in Window’s remote procedure call (RPC) service. Microsoft quickly Playing with Universal Plug and Play...................24 released an out-of-band security patch (MS08-067), going so far as to Locking itself out.................................................27 classify the update as “critical” for some operating systems—the high- A new Downadup variant?......................................29 Advanced crypto protection.................................30 est designation for a Microsoft Security Bulletin.
    [Show full text]
  • Post-Mortem of a Zombie: Conficker Cleanup After Six Years Hadi Asghari, Michael Ciere, and Michel J.G
    Post-Mortem of a Zombie: Conficker Cleanup After Six Years Hadi Asghari, Michael Ciere, and Michel J.G. van Eeten, Delft University of Technology https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/asghari This paper is included in the Proceedings of the 24th USENIX Security Symposium August 12–14, 2015 • Washington, D.C. ISBN 978-1-939133-11-3 Open access to the Proceedings of the 24th USENIX Security Symposium is sponsored by USENIX Post-Mortem of a Zombie: Conficker Cleanup After Six Years Hadi Asghari, Michael Ciere and Michel J.G. van Eeten Delft University of Technology Abstract more sophisticated C&C mechanisms that are increas- ingly resilient against takeover attempts [30]. Research on botnet mitigation has focused predomi- In pale contrast to this wealth of work stands the lim- nantly on methods to technically disrupt the command- ited research into the other side of botnet mitigation: and-control infrastructure. Much less is known about the cleanup of the infected machines of end users. Af- effectiveness of large-scale efforts to clean up infected ter a botnet is successfully sinkholed, the bots or zom- machines. We analyze longitudinal data from the sink- bies basically remain waiting for the attackers to find hole of Conficker, one the largest botnets ever seen, to as- a way to reconnect to them, update their binaries and sess the impact of what has been emerging as a best prac- move the machines out of the sinkhole. This happens tice: national anti-botnet initiatives that support large- with some regularity. The recent sinkholing attempt of scale cleanup of end user machines.
    [Show full text]
  • Undergraduate Report
    UNDERGRADUATE REPORT Attack Evolution: Identifying Attack Evolution Characteristics to Predict Future Attacks by MaryTheresa Monahan-Pendergast Advisor: UG 2006-6 IINSTITUTE FOR SYSTEMSR RESEARCH ISR develops, applies and teaches advanced methodologies of design and analysis to solve complex, hierarchical, heterogeneous and dynamic problems of engineering technology and systems for industry and government. ISR is a permanent institute of the University of Maryland, within the Glenn L. Martin Institute of Technol- ogy/A. James Clark School of Engineering. It is a National Science Foundation Engineering Research Center. Web site http://www.isr.umd.edu Attack Evolution 1 Attack Evolution: Identifying Attack Evolution Characteristics To Predict Future Attacks MaryTheresa Monahan-Pendergast Dr. Michel Cukier Dr. Linda C. Schmidt Dr. Paige Smith Institute of Systems Research University of Maryland Attack Evolution 2 ABSTRACT Several approaches can be considered to predict the evolution of computer security attacks, such as statistical approaches and “Red Teams.” This research proposes a third and completely novel approach for predicting the evolution of an attack threat. Our goal is to move from the destructive nature and malicious intent associated with an attack to the root of what an attack creation is: having successfully solved a complex problem. By approaching attacks from the perspective of the creator, we will chart the way in which attacks are developed over time and attempt to extract evolutionary patterns. These patterns will eventually
    [Show full text]
  • PC Anti-Virus Protection 2011
    PC Anti-Virus Protection 2011 12 POPULAR ANTI-VIRUS PROGRAMS COMPARED FOR EFFECTIVENESS Dennis Technology Labs, 03/08/2010 www.DennisTechnologyLabs.com This test aims to compare the effectiveness of the most recent releases of popular anti-virus software1. The products include those from Kaspersky, McAfee, Microsoft, Norton (Symantec) and Trend Micro, as well as free versions from Avast, AVG and Avira. Other products include those from BitDefender, ESET, G-Data and K7. The tests were conducted between 07/07/2010 and 22/07/2010 using the most up to date versions of the software available. A total of 12 products were exposed to genuine internet threats that real customers could have encountered during the test period. Crucially, this exposure was carried out in a realistic way, reflecting a customer’s experience as closely as possible. For example, each test system visited real, infected websites that significant numbers of internet users were encountering at the time of the test. These results reflect what would have happened if those users were using one of the seven products tested. EXECUTIVE SUMMARY Q Products that block attacks early tended to protect the system more fully The nature of web-based attacks means that the longer malware has access to a system, the more chances it has of downloading and installing further threats. Products that blocked the malicious and infected websites from the start reduced the risk of compromise by secondary and further downloads. Q 100 per cent protection is rare This test recorded an average protection rate of 87.5 per cent. New threats appear online frequently and it is inevitable that there will be times when specific security products are unable to protect from some of these threats.
    [Show full text]
  • The Botnet Chronicles a Journey to Infamy
    The Botnet Chronicles A Journey to Infamy Trend Micro, Incorporated Rik Ferguson Senior Security Advisor A Trend Micro White Paper I November 2010 The Botnet Chronicles A Journey to Infamy CONTENTS A Prelude to Evolution ....................................................................................................................4 The Botnet Saga Begins .................................................................................................................5 The Birth of Organized Crime .........................................................................................................7 The Security War Rages On ........................................................................................................... 8 Lost in the White Noise................................................................................................................. 10 Where Do We Go from Here? .......................................................................................................... 11 References ...................................................................................................................................... 12 2 WHITE PAPER I THE BOTNET CHRONICLES: A JOURNEY TO INFAMY The Botnet Chronicles A Journey to Infamy The botnet time line below shows a rundown of the botnets discussed in this white paper. Clicking each botnet’s name in blue will bring you to the page where it is described in more detail. To go back to the time line below from each page, click the ~ at the end of the section. 3 WHITE
    [Show full text]
  • MODELING the PROPAGATION of WORMS in NETWORKS: a SURVEY 943 in Section 2, Which Set the Stage for Later Sections
    942 IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 16, NO. 2, SECOND QUARTER 2014 Modeling the Propagation of Worms in Networks: ASurvey Yini Wang, Sheng Wen, Yang Xiang, Senior Member, IEEE, and Wanlei Zhou, Senior Member, IEEE, Abstract—There are the two common means for propagating attacks account for 1/4 of the total threats in 2009 and nearly worms: scanning vulnerable computers in the network and 1/5 of the total threats in 2010. In order to prevent worms from spreading through topological neighbors. Modeling the propa- spreading into a large scale, researchers focus on modeling gation of worms can help us understand how worms spread and devise effective defense strategies. However, most previous their propagation and then, on the basis of it, investigate the researches either focus on their proposed work or pay attention optimized countermeasures. Similar to the research of some to exploring detection and defense system. Few of them gives a nature disasters, like earthquake and tsunami, the modeling comprehensive analysis in modeling the propagation of worms can help us understand and characterize the key properties of which is helpful for developing defense mechanism against their spreading. In this field, it is mandatory to guarantee the worms’ spreading. This paper presents a survey and comparison of worms’ propagation models according to two different spread- accuracy of the modeling before the derived countermeasures ing methods of worms. We first identify worms characteristics can be considered credible. In recent years, although a variety through their spreading behavior, and then classify various of models and algorithms have been proposed for modeling target discover techniques employed by them.
    [Show full text]
  • Diapositiva 1
    Feliciano Intini Responsabile dei programmi di Sicurezza e Privacy di Microsoft Italia • NonSoloSecurity Blog: http://blogs.technet.com/feliciano_intini • Twitter: http://twitter.com/felicianointini 1. Introduction - Microsoft Security Intelligence Report (SIR) 2. Today‘s Threats - SIR v.8 New Findings – Italy view 3. Advancements in Software Protection and Development 4. What the Users and Industry Can Do The 8th volume of the Security Intelligence Report contains data and intelligence from the past several years, but focuses on the second half of 2009 (2H09) Full document covers Malicious Software & Potentially Unwanted Software Email, Spam & Phishing Threats Focus sections on: Malware and signed code Threat combinations Malicious Web sites Software Vulnerability Exploits Browser-based exploits Office document exploits Drive-by download attacks Security and privacy breaches Software Vulnerability Disclosures Microsoft Security Bulletins Exploitability Index Usage trends for Windows Update and Microsoft Update Microsoft Malware Protection Center (MMPC) Microsoft Security Response Center (MSRC) Microsoft Security Engineering Center (MSEC) Guidance, advice and strategies Detailed strategies, mitigations and countermeasures Fully revised and updated Guidance on protecting networks, systems and people Microsoft IT ‗real world‘ experience How Microsoft IT secures Microsoft Malware patterns around the world with deep-dive content on 26 countries and regions Data sources Malicious Software and Potentially Unwanted Software MSRT has a user base
    [Show full text]
  • Proposal of N-Gram Based Algorithm for Malware Classification
    SECURWARE 2011 : The Fifth International Conference on Emerging Security Information, Systems and Technologies Proposal of n-gram Based Algorithm for Malware Classification Abdurrahman Pektaş Mehmet Eriş Tankut Acarman The Scientific and Technological The Scientific and Technological Computer Engineering Dept. Research Council of Turkey Research Council of Turkey Galatasaray University National Research Institute of National Research Institute of Istanbul, Turkey Electronics and Cryptology Electronics and Cryptology e-mail:[email protected] Gebze, Turkey Gebze, Turkey e-mail:[email protected] e-mail:[email protected] Abstract— Obfuscation techniques degrade the n-gram malware used by the previous work. Similar methodologies features of binary form of the malware. In this study, have been used in source authorship, information retrieval methodology to classify malware instances by using n-gram and natural language processing [5], [6]. features of its disassembled code is presented. The presented The first known use of machine learning in malware statistical method uses the n-gram features of the malware to detection is presented by the work of Tesauro et al. in [7]. classify its instance with respect to their families. n-gram is a fixed size sliding window of byte array, where n is the size of This detection algorithm was successfully implemented in the window. The contribution of the presented method is IBM’s antivirus scanner. They used 3-grams as a feature set capability of using only one vector to represent malware and neural networks as a classification model. When the 3- subfamily which is called subfamily centroid. Using only one grams parameter is selected, the number of all n-gram vector for classification simply reduces the dimension of the n- features becomes 2563, which leads to some spacing gram space.
    [Show full text]
  • Security Chapter
    Barbarians at the Gateway (and just about everywhere else): A Brief Managerial Introduction to Information Security Issues1 a gallaugher.com case provided free to faculty & students for non-commercial use © Copyright 1997-2009, John M. Gallaugher, Ph.D. – for more info see: http://www.gallaugher.com/chapters.html Draft version last modified: Dec. 7 , 2009 – comments welcome [email protected] Note: this is an earlier version of the chapter. All chapters updated Dec. 2009 are now hosted (and still free) at http://www.flatworldknowledge.com. For details see the ‘Courseware’ section of http://gallaugher.com INTRODUCTION LEARNING OBJECTIVES: After studying this section you should be able to: 1. Recognize that information security breaches are on the rise. 2. Understand the potentially damaging impact of security breaches. 3. Recognize that information security must be made a top organizational priority. Sitting in the parking lot of a Minneapolis Marshalls, a hacker armed with a laptop and a telescope‐shaped antenna infiltrated the store’s network via an insecure Wi‐Fi base station. The attack launched what would become a billion‐dollar plus nightmare scenario for TJX, the parent of retail chains that include Marshalls, Home Goods, and T.J. Maxx. Over a period of several months, the hacker and his gang stole at least 45.7 million credit and debit card numbers, and pilfered driver’s license and other private information from an additional 450,000 customers2. TJX, at the time a $17.5 billion, Fortune 500 firm, was left reeling from the incident. The attack deeply damaged the firm’s reputation.
    [Show full text]