Masaryk University Faculty of Informatics Analysis of SSL/TLS handshakes in malware encrypted traffic Bachelor’s Thesis Daniela Belajová Brno, Fall 2018 Masaryk University Faculty of Informatics Analysis of SSL/TLS handshakes in malware encrypted traffic Bachelor’s Thesis Daniela Belajová Brno, Fall 2018 This is where a copy of the official signed thesis assignment and a copy ofthe Statement of an Author is located in the printed version of the document. Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Daniela Belajová Advisor: Mgr. Stanislav Špaček i Acknowledgements I would like to thank my advisor Mgr. Stanislav Špaček for all his guidance, patience, valuable advice and feedback through the whole process of writing this thesis. I would also like to thank my family for their support and encouragement. ii Abstract One of the current trends in cybersecurity is abusing encrypted net- work traffic to mask a malware activity and complicate its detection. Although encryption ensures the confidentiality of transmitted data, at the same time it prevents the use of some detection methods. In this bachelor’s thesis, I focus on the SSL/TLS protocol, which is one of the most widespread encryption protocols providing secure com- munication in nowadays networks. The main aim of this work is to analyze the SSL/TLS traffic gener- ated by malware in recent malware campaigns misusing encryption to hide their activity, concentrating on protocol metadata from the initial handshake of the connection and based on the results of comparison with benign encrypted traffic, find possible distinctive characteristics that would allow malware to be identified in benign encrypted traffic. iii Keywords SSL/TLS protocol, SSL/TLS handshake, encrypted traffic, analysis, malware, botnet, C&C server, pcap, detection iv Contents 1 Introduction 1 2 Encrypted network traffic 2 2.1 The present situation in encrypted network traffic ......2 2.2 Encryption protocols .....................3 2.3 SSL/TLS protocol .......................4 2.3.1 Versions . .4 2.3.2 Protocol features . .5 2.3.3 Protocol architecture . .5 2.3.4 The process of handshake . .6 2.3.5 The process of abbreviated handshake . .9 2.3.6 X.509 certificates . 10 2.3.7 Extensions . 11 3 Overview of recent malware campaigns 12 3.1 Recent malware campaigns .................. 12 3.1.1 Trickbot . 13 3.1.2 Gozi . 14 3.1.3 ZeusPanda . 15 3.1.4 IcedID . 16 3.1.5 Gootkit . 17 3.1.6 Dridex . 18 4 Malware handshakes analysis 20 4.1 Dataset ............................ 20 4.2 Wireshark ........................... 21 4.3 Results of the analysis ..................... 21 4.3.1 Trickbot . 23 4.3.2 Gozi . 24 4.3.3 ZeusPanda . 25 4.3.4 IcedID . 26 4.3.5 Gootkit . 27 4.3.6 Dridex . 28 4.4 Summary and comparison .................. 29 5 Malware and benign traffic comparison 30 v 5.1 Benign malware dataset .................... 30 5.2 Discussion about comparison results ............. 31 5.2.1 Extensions . 31 5.2.2 Cipher suites . 32 5.2.3 Certificates . 33 5.2.4 Versions . 34 5.3 Summary ........................... 35 6 Conclusion 36 6.1 Future work .......................... 37 Bibliography 38 A The summary of malware campaigns 43 B Results of SSL/TLS handshakes analysis 44 C Pcap files with analyzed samples 46 vi List of Tables 4.1 An overview of Trickbot malware samples 23 4.2 Trickbot C&C servers certificates 24 4.3 An overview of Gozi malware sample 24 4.4 Gozi C&C servers certificates 25 4.5 An overview of ZeusPanda malware samples 25 4.6 ZeusPanda C&C servers certificates 26 4.7 An overview of IcedID malware samples 26 4.8 IcedID C&C servers certificates 27 4.9 An overview of Gootkit malware samples 27 4.10 Gootkit C&C servers certificates 28 4.11 An overview of Dridex malware samples 28 4.12 Dridex C&C servers certificates 29 5.1 The occurrence of extensions in ClientHello/ServerHello messages in the benign traffic sample 31 A.1 The summary of recent malware campaigns 43 B.1 Mapping extensions to the hex code used in Table B.2 44 B.2 Malware SSL/TLS handshakes analysis results 45 vii List of Figures 2.1 TLS protocol structure - based on Source: [9] 5 2.2 The TLS full handshake - based on Source: [8] 7 2.3 The TLS abbreviated handshake - based on Source: [8] 9 4.1 Lists of malware-supported cipher suites 22 viii 1 Introduction Over the last few years, as the need to protect users’ privacy in the online world has grown, the encryption of data transmitted across networks has become nearly a standard situation. However, the adap- tation of this technology has been followed by efforts to exploit its wide usage as a security threat and malware (malicious software) dis- tribution tool. Considering the encrypted traffic poses a challenge for detecting anomalies and applying the conventional traffic monitoring methods, it is essential to develop new effective privacy-conscious ones. In this thesis, I focus on Secure Sockets Layer/Transport Layer Se- curity (SSL/TLS) encryption protocol which secures data transmission for most of the known application protocols in today’s networks. The aim of this work is to investigate the possibility of differen- tiating encrypted traffic generated by malware in recent malware campaigns from benign encrypted traffic based on SSL/TLS hand- shake parameters which are exchanged in unencrypted form while establishing a connection between the communicating parties. The thesis is divided into six chapters. Chapter 1 and Chapter 2 introduce the current issue of encrypted network traffic. Chapter 2 also constitutes the behavior process of SSL/TLS protocol and thus creates the theoretical basis for the analysis. Chapter 3 overviews the selected recent campaigns of malware families which benefit from encrypted traffic to hide their activities. The practical part of this work begins with Chapter 4, which out- lines the analysis results of collected pcap files with encrypted traffic generated by malware families described in Chapter 3. Chapter 5 dis- cusses the comparison of obtained data from malware analysis and the acquired benign traffic sample. Samples of regular and malware traffic are the output of the thesis together with the results oftheir analysis. Chapter 6 provides the summary of achieved outcomes and suggests a possible direction for further research. 1 2 Encrypted network traffic This chapter in Section 2.1 and Section 2.2 provides an explanation of the current trend in encrypted network traffic and a basic survey ofthe encryption protocols occurring in today’s networks. The essential part of the chapter consists of a description of SSL/TLS encryption protocol in Section 2.3, where I focus on SSL/TLS subprotocols responsible for the initial handshake and parameters exchanged and established during this process. 2.1 The present situation in encrypted network traffic Nowadays, encryption is one of the most effective and most commonly used techniques to secure stored or transmitted sensitive information. Encryption provides data confidentiality by converting any type of data to an encoded form which is readable only to an entity with an appropriate secret, a decryption key. Currently, we can label any information as sensitive whose collection and analysis can lead to tracking, profiling or other crimes against the network services user. It includes passwords, banking credentials, and email or another type of electronic communication theft. That is why in today’s Internet world, the number of websites using encryption protocols, especially the SSL/TLS protocol, to ensure the protection of their activities, is significantly rising. On the other hand, the properties of encrypted traffic have be- come very popular in the last years among cybercriminals who have started misusing them to hide malware activity and to prevent mal- ware detection based on pattern matching. Companies that have been in the network security market for many years, like SonicWall, have seen a growing number of attacks hidden by the SSL/TLS protocol [1]. Zscaler announces that it has blocked an average of 800,000 SSL transactions a day in the half-year period at the turn of 2017 and 2018 that hide some form of attack [2]. So the main question is: How to differentiate between legitimate and malicious encrypted traffic? One of the possible ways can be decryption of the communication’s packets followed by packet’s content analysis and their re-encryption. However, this method is too expensive and ineffective. Therefore, 2 2. Encrypted network traffic scientists are developing methods that do not require this approach and preserve user privacy. The majority of encryption protocols comprise of two main phases. Encrypted data transmission is preceded by an unencrypted hand- shake, the initial setting of parameters for further communication and possible authentication of the communicating parties [3]. Because packets are sent as a plaintext during the handshake, they are simply accessible and therefore they can be analyzed. 2.2 Encryption protocols The main function of encryption protocols is the use of cryptographic principles and methods to ensure secure data transfer between com- municating parties. In today’s networks, this feature is implemented on multiple layers. On the network layer of the ISO/OSI model operates IPSec proto- col suites. IPSec was designed as an improvement of security over the internet because the IPv4 protocol does not offer sufficient security features. It is currently an optional component of IPv4 protocol and the part of IPv6 protocol. The significant feature of IPSec is that it provides security of all traffic at IP level and does not require modifi- cation of each application.