Micro-architectural Threats to Modern Computing Systems by Mehmet Sinan Inci_ A Dissertation Submitted to the Faculty of the WORCESTER POLYTECHNIC INSTITUTE In partial fulfillment of the requirements for the Degree of Doctor of Philosophy in Electrical and Computer Engineering by April 2019 APPROVED: Associate Professor William Robertson Assistant Professor Robert J. Walls Dissertation Committee Dissertation Committee Northeastern University CS Department Professor Thomas Eisenbarth Professor Berk Sunar Dissertation Committee Dissertation Advisor ECE Department ECE Department Professor Ludwig Reinhold Department Head ECE Department Abstract With the abundance of cheap computing power and high-speed internet, cloud and mobile computing replaced traditional computers. As computing models evolved, newer CPUs were fitted with additional cores and larger caches to accommodate run multiple processes concurrently. In direct relation to these changes, shared hard- ware resources emerged and became a source of side-channel leakage. Although side-channel attacks have been known for a long time, these changes made them practical on shared hardware systems. In addition to side-channels, concurrent ex- ecution also opened the door to practical quality of service attacks (QoS). The goal of this dissertation is to identify side-channel leakages and architectural bottlenecks on modern computing systems and introduce exploits. To that end, we introduce side-channel attacks on cloud systems to recover sensitive information such as code execution, software identity as well as cryptographic secrets. Moreover, we introduce a hard to detect QoS attack that can cause over 90+% slowdown. We demonstrate our attack by designing an Android app that causes degradation via memory bus locking. While practical and quite powerful, mounting side-channel attacks is akin to listening on a private conversation in a crowded train station. Significant manual labor is required to de-noise and synchronizes the leakage trace and extract features. With this motivation, we apply machine learning (ML) to automate and scale the data analysis. We show that classical machine learning methods, as well as more complicated convolutional neural networks (CNN), can be trained to extract useful information from side-channel leakage trace. Finally, we propose the DeepCloak framework as a countermeasure against side- channel attacks. We argue that by exploiting adversarial learning (AL), an inherent weakness of ML, as a defensive tool against side-channel attacks, we can cloak side- channel trace of a process. With DeepCloak, we show that it is possible to trick highly accurate (99+% accuracy) CNN classifiers. Moreover, we investigate defenses against AL to determine if an attacker can protect itself from DeepCloak by applying adversarial re-training and defensive distillation. We show that even in the presence of an intelligent adversary that employs such techniques, DeepCloak still succeeds. 2 Acknowledgments The research that resulted in this dissertation is funded by the National Science Foundation grants CNS-1618837, CNS-1318919, and CNS-1314770. I would like to thank the National Science Foundation for their support. I would like to extend my sincere gratitude to Professor Berk Sunar for his continuous support, guidance, and wisdom that made this dissertation possible. Through many setbacks and challenges during my Ph.D., his trust and the freedom he gave, enabled me to find a solution and make it work. His foresight, intelligence, and work discipline have resonated with me throughout my Ph.D. and will continue to do so in the future. He is truly an inspiration to find interesting research challenges of the future and tackle them. I would like to thank my co-advisor Professor Thomas Eisenbarth for his support and guidance throughout my research. His unique and original perspective and attention to detail was an invaluable asset for me. It gave me a more wholesome view of things and made me a better researcher. I would like to thank my dissertation committee members Professor Berk Sunar, Professor Thomas Eisenbarth, Professor Robert J. Walls, and Professor William Robertson. I am grateful for their guidance, feedback and their invaluable time spent for the preparation of this dissertation. I am grateful for working with the amazing people that make up the Vernam Lab. In the past years, we have shared a lot and bonded as brothers (and sister!) in arms. All of us spent more time together than we did with our close friends and family outside of academia. In not any particular order, I would like to thank my dear friends Gorka Irazoki, Berk Gulmezoglu, Yarkin Doroz, Gizem Cetin, Michael Moukarzel, Cong Chen, Dai Wei, Marc Green, Saad Islam, Koksal Mus, Okan Seker, Abraham Fernandez and Aria Shahverdi for their friendship and support. i I would like to thank my parents and for their continuous support through my education, starting from the first grade up until the end of the Ph.D. My father always stressed the importance of a good education and imprinted that on me. I would also like to thank my uncle Ibrahim and my grandfather for their support, belief in me and wishes to see me as a doctor one day. They both passed away during my Ph.D., may they rest in peace. Last but not least, I would like to thank my wife, Gulseren Gizem for being the incredible person that she is. Her support throughout the grad school and putting up with me -which is not an easy task- made it all possible. She is a caring wife, my friend, my other half. On top of that, I am eternally grateful to her for shouldering the raising our son Yunus Emre with a part-time father, and away from grandparents or any other close family to help her out. Not to mention the cold, harsh climate of the city of Worcester. This doctorate would not possible without her many sacrifices. As this Ph.D. journey reaches its conclusion, I intend to make up for the lost time and create infinite happy memories, and set sail to new adventures with her, as a family. ii Contents 1 Introduction 1 1.1 Contributions . .4 1.1.1 The publications resulted in this dissertation . .6 2 Background 7 2.1 Computer Architecture . .7 2.1.1 Memory Hierarchy . .7 2.1.2 Hardware Performance Counters . .8 2.2 Memory De-duplication . .9 2.2.1 Kernel Same-page Merging (KSM) . 11 2.3 Mobile Computing . 12 2.3.1 Overview of Android Security . 12 2.3.2 Android Permission System . 14 2.4 Atomic Operations . 16 2.5 Cache Side-Channel Attacks . 18 2.5.1 Flush+Reload (F+R) Attack . 20 2.5.2 Prime+Probe (P+P) Attack . 22 2.6 Machine Learning . 27 2.6.1 Convolutional Neural Networks . 28 iii 2.6.2 Adversarial Learning . 29 2.6.3 Defenses Against Adversarial Learning . 31 3 Related Work 34 3.1 Cache Side-channel Attacks . 34 3.2 Co-location Detection on Cloud . 37 3.3 Branch Prediction Attacks . 38 3.4 Hardware Performance Counters (HPC) . 39 3.5 Micro-architectural Attacks on Mobile Platforms . 39 4 Software Detection Through Shared CPU Cache 41 4.1 Motivation . 42 4.2 Detection Method . 49 4.2.1 Detection Stages . 50 4.2.2 Preventing Wrong Version Detection . 52 4.3 IP Address Recovery . 53 4.4 Additional Dangers of Version Detection . 54 4.5 Experiment Setup and Results . 57 4.5.1 Library Detection . 61 4.5.2 OpenSSL Version Detection . 63 4.5.3 IP Detection . 64 4.6 Countermeasures against the Flush+Reload . 65 4.7 Conclusion . 68 5 Co-location Detection on Cloud 70 5.1 Motivation . 70 5.2 Threat Models . 72 5.2.1 Random Victim Scenario . 72 iv 5.2.2 Targeted Victim Scenario . 74 5.3 LLC Covert Channel . 75 5.3.1 Prime+Probe in LLC . 76 5.4 Software Profiling on LLC . 77 5.5 Memory Bus Locking . 78 5.5.1 Atomic Operations . 79 5.5.2 Cache Line Profiling Stage . 80 5.5.3 Dual Socket Results . 80 5.6 Commercial Clouds Experiment Results . 81 5.6.1 LLC Covert Channel . 82 5.6.2 LLC Software Profiling . 82 5.6.3 Memory Bus Locking . 85 5.6.4 Comparison of Detection Methods . 86 5.7 Conclusion . 88 6 RSA Key Recovery on Cloud 89 6.1 Motivation . 90 6.2 Co-locating on Amazon EC2 . 93 6.2.1 Revisiting Known Detection Methods . 94 6.2.2 The LLC Co-location Method . 98 6.2.3 Unsuccessful Co-location Detection Methods . 100 6.3 Tricks and Challenges of Co-location Detection . 100 6.4 Cross-VM RSA Key Recovery . 105 6.5 Leakage Analysis Method . 108 7 A Micro-architectural QoS Attack on Smartphones 114 7.1 Motivation . 115 v 7.2 The QoS Attack Methodology . 117 7.2.1 Cache Line Profiling Stage . 122 7.2.2 Attacker App Design and Implementation . 125 7.3 Experiment Setup and Results . 126 7.3.1 Experiment Setup . 126 7.3.2 Test Targets: Performance Benchmarks . 128 7.3.3 Degradation Results . 129 7.3.4 Stealthiness of the Attacker App . 130 7.4 Detection and Mitigation . 135 7.5 Conclusion . 136 7.6 Ethical Concerns and Responsible Disclosure . 137 8 Machine Learning for and against Side-channel Attacks 138 8.1 Motivation . 138 8.2 Training Classifiers to Process Side-channel Leakage . 141 8.2.1 Profiling Software using HPC Traces . 142 8.2.2 Experiment Setup . 143 8.2.3 Classifier Design and Implementation . 144 8.2.4 Classification Results . 146 8.3 DeepCloak, A Framework to Cloak Side-channel Leakage . 153 8.3.1 Adversarial Learning Attacks . 155 8.3.2 AL Results on the Unprotected Model . 157 8.3.3 AL Results on the Hardened Model .