Towards Detection and Prevention of Malicious Activities Against Web Applications and Internet Services Dissertation zur Erlangung des Grades eines Doktor-Ingenieurs der Fakult¨at fur¨ Elektrotechnik und Informationstechnik an der Ruhr-Universit¨at Bochum vorgelegt von Apostolos Zarras aus Athen, Griechenland Bochum, August 2015 Tag der mundlichen¨ Prufung:¨ 27.08.2015 Gutachter: Prof. Dr. Thorsten Holz, Ruhr-Universit¨at Bochum Zweitgutachter: Prof. Dr. Herbert Bos, Vrije Universiteit Amsterdam Abstract The Internet has witnessed a tremendous growth the last years. Undoubtedly, its services and mostly the World Wide Web have become an integral part in the lives of hundreds of millions of people, who use it in daily basis. Unfortunately, as the Internet's popularity increases, so does the interest of attackers who seek to exploit vulnerabilities in users' machines. What was primary meant to be mainly a competition among computer experts to test and improve their technical skills has turned into a multi-billion dollar business. Nowadays, attackers try to take under their control vulnerable computers that will allow them to perform their nefarious tasks, such as sending spam emails, launching distributed denial-of-service (DDoS) attacks, generate revenue from online advertisements by performing click-frauds, or stealing personal data like email accounts and banking credentials. In this dissertation, we address the security issues online users face every day from two points of view. First, we investigate how infected computers that constitu- te a botnet|network of compromised machines which are remotely controlled by an entity, known as botmaster|perform their malicious activities and we propose countermeasures against them. We study two of the most fundamental Internet pro- tocols, SMTP and HTTP, and leverage the fact that numerous entities, including cybercriminals, implement these protocols with subtle but perceivable differences, which we can accurately detect. We then develop novel mitigation techniques that utilize these discrepancies to block the compromised computers from successfully executing the commands issued by a botmaster. Second, we examine different ways in which attackers exploit the web infrastructure to infect new victims. We initially study the formed alliances among web spammers that aim to boost the page rank of their websites. As a result, these websites gain more popularity and can potentially host exploits. We then move forward and in- vestigate the extent to which attackers can abuse the logic of search engine crawlers in order to perform various attacks. Next, we evaluate the ecosystem of online ad- vertisements and analyze the risks related to abuses. Finally, we propose a system that can protect users from online threats while surfing the web. i Zusammenfassung Das Internet hat in den letzten Jahren ein enormes Wachstum erfahren. Seine Anwendungen, allen voran das World Wide Web, sind ein wesentlicher Bestand- teil des t¨aglichen Lebens fur¨ Millionen von Menschen geworden. Mit der steigen- den Popularit¨at des Internets wuchs auch das Interesse von Angreifern, die nach Schwachstellen in den Computern suchen, um sich diese zu Eigen zu machen. Was als eine Art Wettkampf bzw. Kr¨aftemessen unter Computerexperten begann, um technische F¨ahigkeiten zu testen und weiterzuentwickeln, hat sich mittlerweile in ein Multi-Milliarden-Dollar-Gesch¨aft gewandelt. Heutzutage versuchen Angreifer, anf¨allige Computer unter ihre Kontrolle zu bekommen, um Spam-Emails zu ver- senden, \distributed denial-of-service" (DDoS) Attacken durchzufuhren,¨ Profit aus Online-Anzeigen durch sogenannte Click-Frauds zu erzielen oder aber sensible Da- ten aus Email- und Bankkonten zu stehlen. In dieser Dissertation wenden wir uns von zwei Seiten den Sicherheitsproblemen zu, mit denen Internet Benutzer t¨aglich konfrontiert sind. Erstens untersuchen wir wie infizierte Computer in einem Botnetz|ein Netzwerk aus infiltrierten und von einem Botmaster ferngesteuerten Computern|ihre Angriffe ausuben¨ und pr¨asentieren ent- sprechende Gegenmaßnahmen. Dazu untersuchen wir zwei integrale Protokolle des Internets, SMTP und HTTP, und machen uns zunutze, dass diese Protokolle von unterschiedlichen Akteuren wie z.B. Cyberkriminellen in leicht unterschiedlicher Weise genutzt werden. Durch die Analyse dieser Unterschiede k¨onnen wir neue Me- thoden zur Schadensbegrenzung entwicklen, die infizierte Computer davon abhalten die Befehle eines Botmasters auszufuhren.¨ Zweitens untersuchen wir verschiedene Methoden mit denen die Angreifer uber¨ das Web neue Computer infizieren. Hierbei untersuchen wir zum einen Allianzen von Web-Spammern deren Ziel die Steigerung der so genannten PageRanks b¨osartiger Internetseiten ist. Ein h¨oherer PageRank fuhrt¨ zu einer erh¨ohten Sichtbarkeit einer Webseite und typischerweise daher auch zu h¨oheren Infektionsraten. Zum anderen ermittelten wir in welchem Ausmaß Angreifer die Logik von Web-Crawlern von Suchmaschinen ausnutzen, um verschiedene Angriffe durchzufuhren.¨ Daneben be- trachten wir wie Kriminelle Online-Werbung fur¨ ihre Zwecke missbrauchen. Schließ- lich schlagen wir aufbauend auf unseren Untersuchungen ein System vor das An- wendern helfen kann sicher im Web zu surfen. iii Acknowledgements It is a great pleasure to acknowledge everyone that have significantly influenced and inspired my life during my graduate studies. Thus, in this part, I would like to take the time to thank every single one of them. First and foremost, I would like to express my special appreciation to my advisor Thorsten Holz for the valuable guidance and support he showed me over the past four years. I would like to thank him for giving me the opportunity to focus on topics that I was interested in, encouraging my research and allowing me to grow as a research scientist. Next, I would like to thank all former and current members of the Chair for Systems Security of Ruhr-University Bochum that contributed for a pleasant and productive environment. I recall countless meetings in which we had to switch to English because of me. Thank you guys and sorry for the inconvenience. During the time of my Ph.D. studies I had the privilege of collaborating with re- searchers from different universities and countries. This work would have not been possible without all these folks. Apart from this remote collaboration, I also had the chance of visiting the security labs of University of California, Santa Barbara (UCSB) and Institut Eur´ecom. There, I met and worked together with some really intelligent people. I could not help but thank all my old friends along with the new friends I made here in Bochum. The time we spent together was a welcome break in my daily routine. I would also like to express my deepest gratitude to my parents Nikos and Maria, my beloved little sister Nantia, and my grandparents for their support, patience, encouragement and wise advice during my whole life. I never would have made it through, without their love and support. Last, but certainly not least, I would like to thank my wife Eva who has been a bright light in my life. I am truly grateful for her unconditional love, support, and patience. Without her, I would not have been able to balance my research with everything else. I cannot wait until the day we will meet our baby for the first time. This dissertation is dedicated to all these people who believed in me from the time of my birth until now. v Contents 1 Introduction1 1.1 Motivation................................2 1.2 Dissertation Scope............................3 1.3 Contributions...............................4 1.4 Publications................................6 1.5 Outline..................................8 I Botnet Detection and Mitigation 11 2 Spam Mitigation 15 2.1 Introduction................................ 15 2.2 SMTP Protocol.............................. 18 2.3 SMTP Dialects.............................. 20 2.4 Learning Dialects............................. 22 2.4.1 Learning Algorithm....................... 22 2.4.2 Collecting SMTP Conversations................ 24 2.5 Matching Conversations to Dialects................... 27 2.5.1 Building the Decision State Machine.............. 27 2.5.2 Making a Decision........................ 29 2.5.3 Applying the Decision...................... 30 2.6 Botnet Feedback Mechanism...................... 31 2.7 System Overview............................. 33 2.8 Evaluation................................. 34 2.8.1 Evaluating the Dialects..................... 35 vii Contents 2.8.2 Evaluating the Feedback Manipulation............. 39 2.9 Discussion................................. 40 2.9.1 Evading Dialects Detection................... 40 2.9.2 Mitigating Feedback Manipulation............... 41 2.10 Related Work............................... 41 2.11 Summary................................. 43 3 HTTP-Based Malware Mitigation 45 3.1 Introduction................................ 46 3.2 HTTP Protocol.............................. 48 3.3 HTTP-Based Malware.......................... 49 3.4 HTTP-Level Detection.......................... 50 3.4.1 Header Chains.......................... 51 3.4.2 HTTP Templates......................... 52 3.5 System Overview............................. 53 3.5.1 Virtual Machine Zoo....................... 54 3.5.2 Learner.............................. 55 3.5.3 Decision Maker.......................... 56 3.6 Evaluation................................. 57 3.6.1 Establishing the Ground Truth................. 57 3.6.2 Model Generation in Various Web Clients..........