Investigations into Decrypting Live Secure Traffic in Virtual Environments Peter William Lindsay McLaren A thesis submitted in partial fulfilment of the requirements of Edinburgh Napier University, for the award of Doctor of Philosophy July, 2019 i COPYRIGHT Copyright in the text of this thesis rests with the Author. Copies (by any process) either in full or of extracts may be made only by instructions given by the Author and lodged in the Edinburgh Napier University Library. Details may be obtained from the Librarian. This page must form part of any such copies made. Further copies (by any process) of copies made by such instruc- tions may not be made without the permission (in writing) of the Author. The ownership of any intellectual property rights which may be described in this thesis is vested in the Author, subject to any prior agreement to the contrary, and may not be made available for use by third parties without the written permission of the Author, which will prescribe the terms and conditions of any such agreement. Further information on the conditions under which disclosures and exploitation may take place is available from the Dean of the School of Computing. ii DECLARATION No portion of the work referred to in this thesis has been sub- mitted in support of an application for another degree or qualifi- cation of this or any other university or other institute of learning. iii ACKNOWLDEGMENTS The PhD research has been a thoroughly stimulating experience with many peaks and troughs along the way. I would not have completed the work without support from my family, friends, and my supervisory team at Edinburgh Napier University. I would like to acknowledge my late parents who instilled my strong desire for learning. I am most grateful to my wife, Penny McLaren, for her commitment to my reaching this stage. I would like to acknowledge her encouragement throughout as well as her shouldering significant household activities during this process. Other family members and friends, such as ’Oxford comma Phil’, have also provided encouragement and recommendations when I might otherwise have yielded. I have had the benefit of an extended supervisory team as well as an independent panel chair, Professor Hazel Hall. I would like to thank you all. I appreciate the continued encouragement, sup- port, and reflection over many supervisory sessions from Doctor Gordon Russell, in particular once I returned to Australia. Thanks to Doctor Zhiyuan Tan for seeding the VMI idea, providing Uni- versity of Technology Sydney contacts, and for his interesting, thought-provoking viewpoints. Finally, my profound thanks to Professor William Bill Buchanan OBE for his energy, excitement, and lateral thinking abilities that encouraged me to explore new opportunities. Contents ABSTRACT xiv 1 Introduction 1 1.1 Context . .1 1.2 Significance . .2 1.3 Approach . .5 1.4 Research Questions . .8 1.5 Ethics . .9 1.6 Contributions . 12 1.7 Aims and Objectives . 14 1.8 Organisation of Thesis . 16 1.9 Publications . 18 2 Background and Theory 20 2.1 Introduction . 20 2.2 Cryptography in Digital Networks . 21 2.3 Symmetric Block Algorithms . 22 2.3.1 Modes of Operation . 23 2.3.2 Algorithms . 25 2.3.3 Advanced Encryption Standard . 27 2.4 Symmetric Stream Algorithms . 29 2.4.1 Algorithms . 30 2.4.2 ChaCha20 . 31 2.5 Conclusions . 33 iv CONTENTS v 3 Literature Review 35 3.1 Introduction . 35 3.2 Implementation Attacks . 36 3.2.1 Memory . 38 3.2.2 Virtualised Environments . 41 3.3 Virtual Machine Monitoring . 41 3.3.1 Virtual Machine Introspection . 44 3.3.2 Disk Introspection . 47 3.3.3 Memory Introspection . 48 3.3.4 Network Introspection . 51 3.3.5 Monitor Frequency . 53 3.4 Conclusions . 55 4 MemDecrypt: A Framework for Decrypting Secure Communications 57 4.1 Introduction . 57 4.2 Requirements Definition and Terms . 58 4.3 Design . 61 4.3.1 Description . 61 4.3.2 Data Collection Component . 64 4.3.3 Memory Analysis Component . 66 4.3.4 Decrypt Analysis . 71 4.4 Construction . 72 4.4.1 Hypervisors . 72 4.4.2 Data Collection . 76 4.4.3 Memory Analysis . 80 4.4.4 Decrypt Analysis . 81 4.5 Evaluation . 82 4.5.1 Test Criteria . 83 4.5.2 Test Approach . 84 4.5.3 Test Environment . 86 4.6 Extensibility . 86 4.7 Conclusions . 88 CONTENTS vi 5 Determining Insider Attack Data Exfiltration 90 5.1 Introduction . 90 5.2 SSH Protocol . 91 5.2.1 Set-up Phase . 92 5.2.2 Authentication Phase . 95 5.2.3 Connection Phase . 96 5.2.4 Secure File Transfer . 97 5.3 SSH Extension Design . 100 5.3.1 Data Collection . 101 5.3.2 Memory Analysis . 101 5.3.3 Decrypt Analysis . 103 5.4 SSH Extension Implementation . 104 5.4.1 Data Collection . 104 5.4.2 Memory Analysis . 105 5.4.3 Decrypt Analysis . 107 5.5 Evaluation . 107 5.5.1 Experimental Set-up . 108 5.5.2 Experimental Results . 109 5.5.3 Analysis . 113 5.6 Conclusions . 114 6 Decrypting Web Traffic 117 6.1 Introduction . 117 6.2 Background . 118 6.2.1 Protocol Versions . 118 6.2.2 Handshake, Change Cipher Specification, Ap- plication Data . 120 6.2.3 Record Protocol . 122 6.3 TLS Extension Design . 125 6.3.1 Data Collection . 125 6.3.2 Memory Analysis . 126 6.3.3 Decrypt Analysis . 130 6.4 TLS Extension Implementation . 131 CONTENTS vii 6.4.1 Data Collection . 131 6.4.2 Memory Analysis . 132 6.4.3 Decrypt Analysis . 135 6.5 Evaluation . 135 6.5.1 Experimental Set-up . 136 6.5.2 Experimental Results . 137 6.6 Conclusions . 141 7 Discovering Malware Activity Without Prior Knowl- edge 143 7.1 Introduction . 143 7.2 Sourcing Malware Samples . 145 7.3 OpenSSL Extension Evaluation . 148 7.4 Windows Library Extension Design . 151 7.5 Windows Library Extension Evaluation . 153 7.6 Conclusions . 156 8 Deriving ChaCha20 Key Streams From Targeted Memory Analysis 159 8.1 Introduction . 159 8.2 Background . 160 8.2.1 ChaCh20 Description . 160 8.2.2 ChaCha20 Implementations . 161 8.3 ChaCha20 Extension Design . 162 8.4 ChaCha20 Extension Implementation . 164 8.5 Evaluation . 165 8.5.1 Experimental Set-up . 167 8.5.2 Experimental Results . 168 8.6 Conclusions . 171 9 Conclusions and Future Work 173 9.1 Key Conclusions . 173 9.2 Achievement of Aim and Objectives . 174 CONTENTS viii 9.3 Key Contributions . 176 9.4 Future Work . 177 9.4.1 Investigative Gaps . 177 9.4.2 Potential Research Areas . 178 A Package Dependencies 218 B Hypervisor Research Review 219 C NetScantbl Plugin Output 220 D Malware Client Downloads 221 List of Figures 1-1 Investigative Approach . .8 2-1 CBC Mode Encryption and Decryption Process . 24 2-2 CTR Encryption and Decryption Process . 25 2-3 AES Encryption Process . 29 2-4 ChaCha Encryption Process . 31 3-1 Decryption Approaches . 37 4-1 High-level architecture . 62 4-2 Framework Activity Flow Diagram . 64 4-3 Analysis Framework Component Interaction . 64 4-4 Guessing Entropy . 69 4-5 Shannon Entropy . 69 4-6 Cumulative Entropy Distribution Example . 70 5-1 SSH Handshake . 92 5-2 SSH Client Key Exchange . 94 5-3 SSH Message Flow . 99 5-4 SSH Decrypt Output . 110 5-5 SSH Analysis Durations for Variable Key Lengths . 112 5-6 SSH Analysis Durations for Variable Modes . 112 5-7 SSH Decrypted Session . 116 6-1 TLS AES-GCM Application Data Messages . 124 6-2 TLS AES-CBC Application Data Messages . 125 6-3 TLS Extension AES Data Collection Flow . 126 ix LIST OF FIGURES x 6-4 TLS Extension GCM Memory Analysis Flow . 127 6-5 TLS 1.2 AES-GCM Application Data Message . 132 6-6 TLS 1.2 Implicit AES-GCM IVs . 133 6-7 TLS 1.2 AES-GCM Candidate Key Blocks . 133 6-8 TLS 1.3 AES-GCM IVs . 135 6-9 TLS 1.2 AES-GCM Memory Analysis Log . 139 6-10 TLS 1.2 GCM and CBC Mode Analysis Durations 140 7-1 Zbot Fake . 146 7-2 Gozi Data Theft . 147 7-3 TorrentLocker/Crypt0L0cker Warning . 148 7-4 Zbot Application Data Message . 150 7-5 Windows Explorer High-entropy Regions . 152 7-6 Zbot Decrypt Log . 155 7-7 Zbot Server Log . 156 8-1 ChaCha20 SSH Base Structure in Memory . 169 8-2 ChaCha20 TLS Base structure in Memory . 169 8-3 ChaCha20 SSH Decrypt Output Example . 170 8-4 ChaCha20 TLS Memory & Decrypt Analysis Logs 170 8-5 ChaCha20 SSH Analysis Durations vs File Size . 171 C-1 Netscantbl Output . 220 List of Tables 3-1 VM Introspection Research Summary . 46 4-1 Logical Component Mapping to Requirements . 63 4-2 Hypervisor Comparison . 76 4-3 Entropy Thresholds . 81 4-4 Virtual Machine Configurations . 87 5-1 Applied SSH Algorithm Types . 94 5-2 SFTP Write File Message Types . 100 5-3 SSH Encrypted Payload Format . 103 5-4 Windows 7 vs Windows 10 Durations (secs) . 110 5-5 AES-CTR Upload File Size Analysis Durations (secs)111 5-6 AES-CTR Ubuntu Server Analysis Durations (secs) 113 6-1 TLS Version Usage Statistics . 119 6-2 TLS 1.2 Handshake Phase Messages . 121 6-3 TLS 1.2 GCM Key Block Fields . 123 6-4 AES-GCM Application Data Message Format . 123 6-5 AES-CBC Application Data Message Format . 124 6-6 Key Block Field Example . 134 6-7 TLS Analysis Duration Means - Operating Systems (secs) . 137 6-8 TLS Analysis W10 Duration Means - Other Varia- tions (secs) .