DOCSLIB.ORG

Clustering Spam Domains and Hosts: Anti-Spam Forensics with Data Mining

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Clustering Spam Domains and Hosts: Anti-Spam Forensics with Data Mining CLUSTERING SPAM DOMAINS AND HOSTS: ANTI-SPAM FORENSICS WITH DATA MINING by CHUN WEI ALAN P. SPRAGUE, COMMITTEE CHAIR ANTHONY SKJELLUM CHENGCUI ZHANG KENT R. KERLEY RANDAL VAUGHN A DISSERTATION Submitted to the graduate faculty of The University of Alabama at Birmingham, in partial fulfillment of the requirements for the degree of Doctor of Philosophy BIRMINGHAM, ALABAMA 2010 Copyright by Chun Wei 2010 TABLE OF CONTENTS Page ABSTRACT ....................................................................................................................... iv LIST OF TABLES ...............................................................................................................v LIST OF FIGURES ........................................................................................................... vi LIST OF ABBREVIATIONS .......................................................................................... viii CHAPTER 1 INTRODUCTION ...........................................................................................................1 1.1 Current Spam Trend....................................................................................................1 1.2 Protective Mechanisms of Spammers ........................................................................2 1.2.1 Word Obfuscation ...............................................................................................2 1.2.2 Botnet ..................................................................................................................3 1.2.3 Spam Hosting Infrastructure................................................................................4 1.2.4 Fast-Flux Service Networks ................................................................................6 1.3 Research Problem, Goal and Impact ..........................................................................7 2 LITERATURE REVIEW ..............................................................................................12 2.1 Anti-Spam Research .................................................................................................12 2.1.1 Spam Filtering ...................................................................................................13 2.1.2 Message Obfuscation.........................................................................................14 2.1.3 Research on Botnet Detection ...........................................................................17 2.1.4 Research on URLs and Spam Hosts ..................................................................21 2.1.5 Scam vs. Spam Campaign .................................................................................25 2.2 Research on Data Clustering ....................................................................................25 2.2.1 Linkage Based Clustering..................................................................................25 2.2.2 Connected Components .....................................................................................27 2.2.3 Research on Data Streams .................................................................................30 3 HIERARCHICAL CLUSTERING ................................................................................33 3.1 Attribute Extraction ..................................................................................................33 3.2 Clustering Methods ..................................................................................................34 i 3.2.1 Agglomerative Hierarchical Clustering Based on Common Attributes ............35 3.2.2 Connected Components with Weighted Edges .................................................37 3.3 Experimental Results ................................................................................................38 3.3.1 Data Collection ..................................................................................................38 3.3.2 Results of Agglomerative Hierarchical Clustering............................................39 3.3.3 Validation of Results .........................................................................................39 3.3.4 Results of Weighted Edges ................................................................................42 3.4 Discussion .................................................................................................................44 4 FUZZY STRING MATCHING ......................................................................................46 4.1 String Similarity ........................................................................................................46 4.1.1 Inverse Levenshtein Distance ............................................................................46 4.1.2 String Similarity ................................................................................................47 4.2 Subject Similarity .....................................................................................................48 4.2.1 Subject Similarity Score Based on Partial Token Matching .............................48 4.2.2 Adjustable Similarity Score Based on Subject Length......................................49 4.3 Subject Clustering Algorithms .................................................................................50 4.3.1 Simple Algorithm ..............................................................................................50 4.3.2 Recursive Seed Selection Algorithm .................................................................51 4.4 Experimental Results ................................................................................................52 5 CLUSTERING SPAM DOMAINS ................................................................................54 5.1 Retrieval of Spam Domain Data ...............................................................................54 5.1.1 Wildcard DNS Record .......................................................................................56 5.1.2 Retrieval of Hosting IP Addresses ....................................................................57 5.2 Daily Clustering Methods .........................................................................................57 5.2.1 Hosting IP Similarity between Two Domains ...................................................60 5.2.2 Subject Similarity between Two Domains ........................................................61 5.2.3 Overall Similarity between Two Domains ........................................................62 5.2.4 Bi-connected Component Algorithm ................................................................63 5.2.5 Labeling Emails Based on Domain Clusters .....................................................64 5.3 Day to Day Clustering Method .................................................................................64 5.3.1 Similarity between Two Clusters ......................................................................66 5.3.2 Linking Two Clusters ........................................................................................68 5.4 Experimental Results ................................................................................................69 5.4.1 Daily Clustering Results ....................................................................................69 5.4.2 Tracing Clusters over the Experiment Period of Time ......................................74 5.5 Discussion .................................................................................................................80 6 TRACKING CLUSTERS USING HISTORICAL DATA .............................................83 6.1 Historical Cluster Repository....................................................................................84 6.2 Experiment on IP Tracing .........................................................................................85 6.2.1 Canadian Pharmacy Scam .................................................................................86 ii 6.2.2 Ultimate Replica Watches Scam .......................................................................89 6.2.3 Tracing a Phishing Campaign ...........................................................................91 6.2.4 Other Scams and IP Addresses ..........................................................................92 6.3 Discussion ................................................................................................................93 7 CONCLUSION AND FUTURE WORK .......................................................................95 7.1 Benefits and Impact ..................................................................................................97 7.1.1 Improving Domain Black Listing ......................................................................97 7.1.2 Forensic Applications ........................................................................................98 7.1.3 Contributions to Data Mining ..........................................................................100 7.2 Future Work ............................................................................................................101 LIST OF REFERENCES .................................................................................................106 APPENDIX A Spam Database Description ......................................................................................113 B Recursive Seed Selection Algorithm (Pseudo Code) ................................................117 C Bi-connected Component Algorithm
Load more

Recommended publications
  • In-Depth Evaluation of Redirect Tracking and Link Usage
    Proceedings on Privacy Enhancing Technologies ; 2020 (4):394–413 Martin Koop*, Erik Tews, and Stefan Katzenbeisser In-Depth Evaluation of Redirect Tracking and Link Usage Abstract: In today’s web, information gathering on 1 Introduction users’ online behavior takes a major role. Advertisers use different tracking techniques that invade users’ privacy It is common practice to use different tracking tech- by collecting data on their browsing activities and inter- niques on websites. This covers the web advertisement ests. To preventing this threat, various privacy tools are infrastructure like banners, so-called web beacons1 or available that try to block third-party elements. How- social media buttons to gather data on the users’ on- ever, there exist various tracking techniques that are line behavior as well as privacy sensible information not covered by those tools, such as redirect link track- [52, 69, 73]. Among others, those include information on ing. Here, tracking is hidden in ordinary website links the user’s real name, address, gender, shopping-behavior pointing to further content. By clicking those links, or or location [4, 19]. Connecting this data with informa- by automatic URL redirects, the user is being redirected tion gathered from search queries, mobile devices [17] through a chain of potential tracking servers not visible or content published in online social networks [5, 79] al- to the user. In this scenario, the tracker collects valuable lows revealing further privacy sensitive information [62]. data about the content, topic, or user interests of the This includes personal interests, problems or desires of website. Additionally, the tracker sets not only third- users, political or religious views, as well as the finan- party but also first-party tracking cookies which are far cial status.
    [Show full text]
  • Spamming Botnets: Signatures and Characteristics
    Spamming Botnets: Signatures and Characteristics Yinglian Xie, Fang Yu, Kannan Achan, Rina Panigrahy, Geoff Hulten+,IvanOsipkov+ Microsoft Research, Silicon Valley +Microsoft Corporation {yxie,fangyu,kachan,rina,ghulten,ivano}@microsoft.com ABSTRACT botnet infection and their associated control process [4, 17, 6], little In this paper, we focus on characterizing spamming botnets by effort has been devoted to understanding the aggregate behaviors of leveraging both spam payload and spam server traffic properties. botnets from the perspective of large email servers that are popular Towards this goal, we developed a spam signature generation frame- targets of botnet spam attacks. work called AutoRE to detect botnet-based spam emails and botnet An important goal of this paper is to perform a large scale analy- membership. AutoRE does not require pre-classified training data sis of spamming botnet characteristics and identify trends that can or white lists. Moreover, it outputs high quality regular expression benefit future botnet detection and defense mechanisms. In our signatures that can detect botnet spam with a low false positive rate. analysis, we make use of an email dataset collected from a large Using a three-month sample of emails from Hotmail, AutoRE suc- email service provider, namely, MSN Hotmail. Our study not only cessfully identified 7,721 botnet-based spam campaigns together detects botnet membership across the Internet, but also tracks the with 340,050 unique botnet host IP addresses. sending behavior and the associated email content patterns that are Our in-depth analysis of the identified botnets revealed several directly observable from an email service provider. Information interesting findings regarding the degree of email obfuscation, prop- pertaining to botnet membership can be used to prevent future ne- erties of botnet IP addresses, sending patterns, and their correlation farious activities such as phishing and DDoS attacks.
    [Show full text]
  • Redirect URL
    July 2012 Redirect URL User Guide Welcome to AT&T Website Solutions SM We are focused on providing you the very best web hosting service including all the tools necessary to establish and maintain a successful website. This document contains information that will help you to redirect pages from your site to other locations. You can use it to create a new Domain Redirection as well. © 2012 AT&T Intellectual Property. All rights reserved. AT&T products and services are provided or offered by subsidiaries and affiliates of AT&T Inc. under the AT&T brand and not by AT&T Inc. AT&T, AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. All other trademarks are the property of their owners. This document is not an offer, commitment, representation or warranty by AT&T and is subject to change. Your Web Hosting service is subject to the Terms and Conditions (T&Cs), which may be found at http://webhosting.att.com/Terms-Conditions.aspx . Service terms and Fees are subject to change without notice. Please read the T&Cs for additional information. © 2010 AT&T Intellectual Property. All rights re served. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. Table of ContenContentstststs Introduction ........................................................................................................................................................ 3 Create a New Redirect ...................................................................................................................................
    [Show full text]
  • A Rule Based Approach for Spam Detection
    A RULE BASED APPROACH FOR SPAM DETECTION Thesis submitted in partial fulfillment of the requirements for the award of degree of Master of Engineering In Computer Science & Engineering By: Ravinder Kamboj (Roll No. 800832030) Under the supervision of: Dr. V.P Singh Mrs. Sanmeet Bhatia Assistant Professor Assistant Professor Computer Science & Engineering Department of SMCA COMPUTER SCIENCE AND ENGINEERING DEPARTMENT THAPAR UNIVERSITY PATIALA – 147004 JULY- 2010 i ii Abstract Spam is defined as a junk Email or unsolicited Email. Spam has increased tremendously in the last few years. Today more than 85% of e-mails that are received by e-mail users are spam. The cost of spam can be measured in lost human time, lost server time and loss of valuable mail. Spammers use various techniques like spam via botnet, localization of spam and image spam. According to the mail delivery process anti-spam measures for Email Spam can be divided in to two parts, based on Emails envelop and Email data. Black listing, grey listing and white listing techniques can be applied on the Email envelop to detect spam. Techniques based on the data part of Email like heuristic techniques and Statistical techniques can be used to combat spam. Bayesian filters as part of statistical technique divides the income message in to words called tokens and checks their probability of occurrence in spam e-mails and ham e-mails. Two types of approaches can be followed for the detection of spam e-mails one is learning approach other is rule based approach. Learning approach required a large dataset of spam e-mails and ham e-mails is required for the training of spam filter; this approach has good time characteristics filter can be retrained quickly for new Spam.
    [Show full text]
  • Detection of Malicious Urls by Correlating the Chains of Redirection in an Online Social Network (Twitter)
    International Journal of Research Studies in Computer Science and Engineering (IJRSCSE) Volume 1, Issue 3, July 2014, PP 33-38 ISSN 2349-4840 (Print) & ISSN 2349-4859 (Online) www.arcjournals.org Detection of Malicious URLs by Correlating the Chains of Redirection in an Online Social Network (Twitter) 1MD.Sabeeha, SK.Karimullah, 2P.Babu 1PG Schalor,CSE, Quba college of engineering and technology 2, Associate professor, QCET, NELLORE ABSTRACT: Twitter is prone to malicious tweets containing URLs for spam, phishing, and malware distribution. Conventional Twitter spam detection schemes utilize account features such as the ratio of tweets containing URLs and the account creation date, or relation features in the Twitter graph. These detection schemes are ineffective against feature fabrications or consume much time and resources. Conventional suspicious URL detection schemes utilize several features including lexical features of URLs, URL redirection, HTML content, and dynamic behavior. However, evading techniques such as time-based evasion and crawler evasion exist. In this paper, we propose WARNINGBIRD, a suspicious URL detection system for Twitter. Our system investigates correlations of URL redirect chains extracted from several tweets. Because attackers have limited resources and usually reuse them, their URL redirect chains frequently share the same URLs. We develop methods to discover correlated URL redirect chains using the frequently shared URLs and to determine their suspiciousness. We collect numerous tweets from the Twitter public timeline and build a statistical classifier using them. Evaluation results show that our classifier accurately and efficiently detects suspicious URLs. We also present WARNINGBIRD as a near real-time system for classifying suspicious URLs in the Twitter stream.
    [Show full text]
  • OWASP Asia 2008 Rsnake.Pdf
    1 Robert Hansen - CEO SecTheory LLC Bespoke Boutique Internet Security Web Application/Browser Security Network/OS Security http://www.sectheory.com/ Advisory capacity to VCs/start-ups Founded the web application security lab http://ha.ckers.org/ - the lab http://sla.ckers.org/ - the forum 2 iHumble I want to explain the history… Only a few know the whole story. Sit back and relax, it’s story time. 3 We’ve all heard these sentiments: “If you find a vulnerability, we ask that you share it with us. If you share it with us, we will respond to you with a time we will fix that hole.” Scott Petry – Director @ Google (We’ll be coming back to this!) 4 It all started four years ago… We found that redirection vulnerabilities were being used by phishers in a number of sites, Visa, Doubleclick, eBay and of course, Google to confuse consumers. Timeframes for fixes: Visa closed their hole down within hours Double Click within days (partially) eBay within weeks Google still hasn’t closed them (~4 years later) Every company agrees it’s a hole. Everyone 5 Word gets out – fast! http://blogs.geekdojo.net/brian/archive/2004/10/14/googlephishing.aspx http://lists.virus.org/dshield-0602/msg00156.html http://blog.eweek.com/blogs/larry_seltzer/archive/2006/03/05/8240.aspx http://thespamdiaries.blogspot.com/2006/03/google-used-as-url-cloaking-device- in.html http://www.docuverse.com/blog/donpark/EntryViewPage.aspx?guid=e08af74b- 8b86-418c-94e0-7d29a7cb91e2 http://email.about.com/od/outlooktips/qt/et043005.htm http://listserv.educause.edu/cgi- bin/wa.exe?A2=ind0511&L=security&T=0&F=&S=&P=15599 http://www.zataz.com/news/13296/google-corrige-une-faille.html http://google.blognewschannel.com/archives/2007/02/22/google-changes-redirects- adds-nofollow-to-blogger/ http://googlesystem.blogspot.com/2007/02/google-redirect-notice.html And others… 6 Everyone has vulns.
    [Show full text]
  • Real-Time Detection System for Suspicious Urls Krishna Prasad Chouty Governors State University
    Governors State University OPUS Open Portal to University Scholarship All Capstone Projects Student Capstone Projects Fall 2015 Real-Time Detection System for Suspicious URLs Krishna Prasad Chouty Governors State University Anup Chandra Thogiti Governors State University Kranthi Sudha Vudatha Governors State University Follow this and additional works at: http://opus.govst.edu/capstones Part of the Digital Communications and Networking Commons, and the Information Security Commons Recommended Citation Chouty, Krishna Prasad; Thogiti, Anup Chandra; and Vudatha, Kranthi Sudha, "Real-Time Detection System for Suspicious URLs" (2015). All Capstone Projects. 164. http://opus.govst.edu/capstones/164 For more information about the academic degree, extended learning, and certificate programs of Governors State University, go to http://www.govst.edu/Academics/Degree_Programs_and_Certifications/ Visit the Governors State Computer Science Department This Project Summary is brought to you for free and open access by the Student Capstone Projects at OPUS Open Portal to University Scholarship. It has been accepted for inclusion in All Capstone Projects by an authorized administrator of OPUS Open Portal to University Scholarship. For more information, please contact [email protected]. REAL-TIMEDETECTION SYSTEM FOR SUSPICIOUS URL’S By Krishna Prasad, Chouty B.Tech, JAWAHARLAL NEHURU TECHNOLOGICAL UNIVERSITY, 2012 AnupChandra, Thogiti B.Tech, JAWAHARLAL NEHURU TECHNOLOGICAL UNIVERSITY, 2011 Kranthi Sudha, Vudatha B.Tech, JAWAHARLAL NEHURU TECHNOLOGICAL UNIVERSITY,
    [Show full text]
  • Towards Electronic Persistence Using ARK Identifiers
    Towards Electronic Persistence Using ARK Identifiers John A. Kunze California Digital Library University of California, Office of the President, Oakland, CA 94612, USA [email protected] Abstract. The ARK (Archival Resource Key) is the only persistent naming scheme for internet information objects that squarely faces the service issues underlying practical electronic permanence. The ARK is a specially constructed, actionable URL encapsulating a globally unique identity that is independent of the current service provider. Each ARK is by definition bound to three things: (a) object access, (b) object meta- data, and (c) a faceted commitment statement. These service bindings provide the best way for service providers to communicate their varied commitments to objects. It is for these reasons that the California Digital Library is pursuing the use of ARKs. 1 Introduction Web users know the irritation of broken URLs [2]. Returning to a web address only to find the desired object missing is more than an irritation for scholars and catalogers, whose trades rely on persistent reference. If there is one thing that distinguishes a digital library from a mere web site, it is that libraries do their best to provide reliable, persistent access through durable links. To that end much discussion and development activity has focused on persistent identifiers, in other words, on links that continue to provide access into the indefinite fu- ture. The popular vision of ubiquitous, technologically-hardened identifiers has a deceptively simple problem statement: 1. The goal: long-term identifiers. 2. The problem: URLs break. 3. Why URLs break: because objects are moved, removed, and replaced.
    [Show full text]
  • Review on Email Spam Filtering Techniques
    International Journal of Scientific Engineering and Applied Science (IJSEAS) – Volume-2, Issue-3, March 2016 ISSN: 2395-3470 www.ijseas.com Review on Email Spam Filtering Techniques Shiva Sharma1, U. Dutta 2 Computer Science and Engineering Department, Maharana Pratap College of technology Putli Ghar Road, Near Collectorate, Gwalior-474006, Madhya Pradesh, India ABSTRACT- In this paper we present email online messages, blogs, forums, whatsapp and spam filtering and email authorship instant messaging services. Among all CMC, identification. Electronic mail is used by email has remained a key source of written millions of people to communicate around the communication, especially in the last few years. world daily and is a mission-critical Due to its salient features, it is the preferred application for many businesses. Over the last source of written communication for almost 10 years, unsolicited bulk email has become a every population (a part from illiterate) major problem for email users called senders connected to the Internet. It is a very quick, and receivers. And the recent years spam asynchronous written communication channel became as a big problem of internet and that is used for various purposes ranging from electronic communication known as users. formal to informal communication. Email There is developed lot of techniques to fight messages can be sent to a single receiver or them. We presents the overview of existing e- broadcasted to groups known as users. An email mail spam filtering methods is given. The message can reach to a number of receivers classification, valuation and juxtaposition of simultaneously and instantly at singe time. traditional and learning based methods are These days, the majority of individuals even provided.
    [Show full text]
  • Detection of Malicious Url Redirection and Distribution
    Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology ISSN 2320–088X IJCSMC, Vol. 2, Issue. 5, May 2013, pg.1 – 5 RESEARCH ARTICLE DETECTION OF MALICIOUS URL REDIRECTION AND DISTRIBUTION T. Manjula Devi 1, R. Krishnaveni 2 1School of Computing Sciences, Hindustan University, India 2School of Computing Sciences, Hindustan University, India 1 [email protected]; 2 [email protected] Abstract— Web-based malicious software (malware) has been increasing over the Internet .It poses threats to computer users through web sites. Computers are infected with Web-based malware by drive-by-download attacks. Drive-by-download attacks force users to download and install the Web-based malware without being aware of it .these attacks evade detection by using automatic redirections to various websites. It is difficult to detect these attacks because each redirection uses the obfuscation technique. This paper analyzes the HTTP communication data of drive-by-download attacks. The results show significant features of the malicious redirections that are used effectively when we detect malware. Key Terms: - Web-based malware; drive-by-download attacks; packet capturing I. INTRODUCTION Damage resulting from Web-based malware has been increasing. Web-based malware uses a drive-by- download technique as its attack methods. Drive-by-download attacks force computer users to download and install malware without being aware of it by exploiting the vulnerabilities in a Web browser or some external components [1]. Figure 1 illustrates a typical drive-by- download attack. A user accessing an entrance site is redirected to malicious Web sites in sequence.
    [Show full text]
  • CHUENCHUJIT-THESIS-2016.Pdf
    c 2016 Thasphon Chuenchujit A TAXONOMY OF PHISHING RESEARCH BY THASPHON CHUENCHUJIT THESIS Submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science in the Graduate College of the University of Illinois at Urbana-Champaign, 2016 Urbana, Illinois Adviser: Associate Professor Michael Bailey ABSTRACT Phishing is a widespread threat that has attracted a lot of attention from the security community. A significant amount of research has focused on designing automated mitigation techniques. However, these techniques have largely only proven successful at catching previously witnessed phishing cam- paigns. Characteristics of phishing emails and web pages were thoroughly analyzed, but not enough emphasis was put on exploring alternate attack vectors. Novel education approaches were shown to be effective at teach- ing users to recognize phishing attacks and are adaptable to other kinds of threats. In this thesis, we explore a large amount of existing literature on phishing and present a comprehensive taxonomy of the current state of phish- ing research. With our extensive literature review, we will illuminate both areas of phishing research we believe will prove fruitful and areas that seem to be oversaturated. ii In memory of Nunta Hotrakitya. iii ACKNOWLEDGMENTS I would like to express my deepest gratitute to Professor Michael Bailey for guiding this work from start to finish. I also greatly appreciate the essential assistance given by Joshua Mason and Zane Ma. Finally, I wish to thank my parents for their love and support throughout my life. iv TABLE OF CONTENTS CHAPTER 1 INTRODUCTION . 1 CHAPTER 2 RELATED WORK . 3 CHAPTER 3 ATTACK CHARACTERISTICS .
    [Show full text]
  • The History of Digital Spam
    The History of Digital Spam Emilio Ferrara University of Southern California Information Sciences Institute Marina Del Rey, CA [email protected] ACM Reference Format: This broad definition will allow me to track, in an inclusive Emilio Ferrara. 2019. The History of Digital Spam. In Communications of manner, the evolution of digital spam across its most popular appli- the ACM, August 2019, Vol. 62 No. 8, Pages 82-91. ACM, New York, NY, USA, cations, starting from spam emails to modern-days spam. For each 9 pages. https://doi.org/10.1145/3299768 highlighted application domain, I will dive deep to understand the nuances of different digital spam strategies, including their intents Spam!: that’s what Lorrie Faith Cranor and Brian LaMacchia ex- and catalysts and, from a technical standpoint, how they are carried claimed in the title of a popular call-to-action article that appeared out and how they can be detected. twenty years ago on Communications of the ACM [10]. And yet, Wikipedia provides an extensive list of domains of application: despite the tremendous efforts of the research community over the last two decades to mitigate this problem, the sense of urgency ``While the most widely recognized form of spam is email spam, the term is applied to similar abuses in other media: instant remains unchanged, as emerging technologies have brought new messaging spam, Usenet newsgroup spam, Web search engine spam, dangerous forms of digital spam under the spotlight. Furthermore, spam in blogs, wiki spam, online classified ads spam, mobile when spam is carried out with the intent to deceive or influence phone messaging spam, Internet forum spam, junk fax at scale, it can alter the very fabric of society and our behavior.
    [Show full text]