Identification and Recognition of Remote-Controlled Malware

Inauguraldissertation zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften der Universität Mannheim

vorgelegt von

Christian Jörn Dietrich aus Düsseldorf, Deutschland

Mannheim, 2012 Dekan: Prof. Dr. Heinz Jürgen Müller, Universität Mannheim Referent: Prof. Dr. Felix Christoph Freiling, Universität Erlangen-Nürnberg Korreferent: Prof. Dr. Christopher Kruegel, University of California, Santa Barbara

Tag der mündlichen Prüfung: 28. März 2013

2 Abstract

Remote-controlled malware, organized in so-called botnets, have emerged as one of the most prolific kinds of malicious software. Although numbers vary, in ex- treme cases such as Conficker, Bredolab and Mariposa, one botnet can span up to several million infected computers. This way, attackers draw substantial revenue by monetizing their bot-infected computers. This thesis encapsulates research on the detection of botnets – a required step towards the mitigation of botnets. First, we design and implement Sandnet, an observation and monitoring infrastructure to study the botnet phenomenon. Using the results of Sandnet, we evaluate detection approaches based on traffic analysis and rogue visual monetization. While traditionally, malware authors designed their botnet command and con- trol channels to be based on plaintext protocols such as IRC, nowadays, botnets leverage obfuscation and encryption of their C&C messages. This renders meth- ods which use characteristic recurring payload bytes ineffective. In addition, we observe a trend towards distributed C&C architectures and nomadic behavior of C&C servers in botnets with a centralized C&C architecture, rendering blacklists infeasible. Therefore, we identify and recognize botnet C&C channels by help of traffic analysis. To a large degree, our clustering and classification leverage the sequence of message lengths per flow. As a result, our implementation, called CoCoSpot, proves to reliably detect active C&C communication of a variety of botnet families, even in face of fully encrypted C&C messages. Furthermore, we observe that botmasters design their C&C channels in a more stealthy manner so that the identification of C&C channels becomes even more difficult. Indeed, with Feederbot we found a botnet that uses DNS as carrier protocol for its command and control channel. By help of statistical entropy as well as behavioral features, we design and implement a classifier that detects DNS- based C&C, even in mixed network traffic of benign users. Using our classifier, we even detect another botnet family which uses DNS as carrier protocol for its command and control. Finally, we show that a recent trend of botnets consists in rogue visual mon- etization. Perceptual clustering of Sandnet screenshots enables us to group malware into rogue visual monetization campaigns and study their localization as well as monetization properties.

i

Zusammenfassung

Fernsteuerbare Schadsoftware, zusammengeschaltet in sog. Botnetzen, hat sich mittlerweile zu einer sehr verbreiteten Art an Schadsoftware entwickelt. Obwohl die genauen Zahlen mitunter schwanken, so zeigt sich in Extremfällen wie etwa bei Conficker, Bredolab und Mariposa, dass ein einzelnes Botnetz aus infizierten Computern mit bis zu zweistelliger Millionenanzahl besteht. Die Angreifer er- wirtschaften somit erhebliche Einkommen, indem sie die infizierten Computer monetarisieren. Diese Arbeit umfasst Forschungsarbeiten zur Erkennung von Botnetzen – ein notwendiger Schritt, um Botnetze zu entschärfen. Zunächst entwerfen und imple- mentieren wir die Beobachtungsumgebung Sandnet, um das Botnetz-Phänomen detailliert untersuchen zu können. Mit Hilfe der Ergebnisse des Sandnet ent- werfen und bewerten wir Erkennungsmechanismen, die sowohl auf Verkehrsflus- sanalyse des Netzwerkverkehrs als auch auf dem vis