ABSTRACT ZHOU, YAJIN. Android Malware: Detection, Characterization, and Mitigation. (Under the direction of Xuxian Jiang.) Recent years, there is an explosive growth in smartphone sales and adoption. The popularity is partially due to the wide availability of a large number of feature-rich smartphone applications (or apps). Unfortunately, the popularity has drawn the attention of malware authors: there were reports about malicious apps on both official and alternative marketplaces. These malicious apps have posed serious threats to user security and privacy. The primary goal of my research is to understand and mitigate the Android malware threats. In this dissertation, we first presented a systematic study to gain a better understanding of malware threats on both official and alternative app marketplaces, by proposing a system called DroidRanger to detect malicious apps on them. Specifically, we first proposed a permission- based behavioral footprinting scheme to detect new samples of known Android malware families. Then we applied a heuristics-based filtering scheme to identify certain inherent behaviors of unknown malware families. This study showed that there is a clear need for a rigorous vetting process, especially for non-regulated alternative marketplaces. After that, we characterized Android malware from various aspects to provide better insights of existing malware threats to the whole research community. That's because without a deep understanding of Android malware, one can hardly imagine that effective mitigations could be proposed. The characterization and a subsequent evolution-based study of representative families revealed that they were evolving rapidly to circumvent the detection from existing mobile anti-virus software. The Android Malware Genome Project [11], spun off from this research, has helped researchers from more than 450 universities, research labs and companies all over the world to develop more effective solutions. In light of the threats, we further proposed a system called AppCage that thoroughly confines the run-time behavior of third-party Android apps. It leverages two complimentary user-level sandboxes to interpose and regulate the app's access to sensitive APIs, and further block mali- cious behaviors of Android malware. Specifically, the first sandbox named dex sandbox hooks into the app's Dalvik virtual machine instance and redirects each sensitive framework API to a proxy which strictly enforces the user-defined policies, and the second sandbox named native sandbox leverages software fault isolation to prevent app's native libraries from directly access- ing the protected APIs or subverting the dex sandbox. Our evaluation showed that AppCage can successfully detect and block attempts to leak private information by third-party apps, and the performance overhead caused by AppCage is negligible for apps without native libraries and minor for apps with them. © Copyright 2015 by Yajin Zhou All Rights Reserved Android Malware: Detection, Characterization, and Mitigation by Yajin Zhou A dissertation submitted to the Graduate Faculty of North Carolina State University in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Computer Science Raleigh, North Carolina 2015 APPROVED BY: Huiyang Zhou Vincent Freeh Kapil Singh Xipeng Shen Xuxian Jiang Chair of Advisory Committee DEDICATION To my wife Lily, without whom I cannot survive sleepless nights during my PhD journey. ii BIOGRAPHY Yajin Zhou is originally from Jiangsu, China. He received his Bachelor of Science from Suzhou University in 2003, and Master of Science from University of Science and Technology of China in 2007, respectively. After that, he spent three years working as a software engineer in industry. Since summer 2010, he came to USA to pursue his Doctor of Philosophy degree in Computer Science under the direction of Dr. Xuxian Jiang. His research focused on proposing and de- veloping practical solutions to improve mobile security and privacy, mainly from the aspect of enriching the app ecosystem. Based on his research, he has cofounded the Android Malware Genome Project (http://www.malgenomeproject.org) and shared the dataset of representative Android malware with more than 450 universities, research labs and companies worldwide. He will graduate with a Ph.D. degree in Computer Science from the North Carolina State University in summer 2015, and join Qihoo 360, a leading security company in China. iii ACKNOWLEDGEMENTS First, I would like to thank my advisor, Dr. Xuxian Jiang, for his numerous help during my PhD study. I was inspired not only from his insights, but also his enthusiasm and the attitude of curiosity in research. I would like to thank Prof. Zhi Wang from Florida State University for his kind help and useful discussion. I am thankful to Prof. Vincent W Freeh, Prof. Huiyang Zhou, Prof. Xipeng Shen and Dr. Kapil Singh for their time and efforts severing in my PhD thesis committee and the insightful comments that helped to improve this dissertation. Second, I am grateful to many persons for their support: Dr. Kapil Singh from IBM T.J. Watson Research Center who was the mentor during my internship there in summer 2012, Prof. Dongyan Xu from Purdue University who provided useful advise and help during my job searching, Prof. Zhenkai Liang from National University of Singapore who was acting as my referee and gave me constructive advise, Prof. Peng Ning from NC State University and all my academic brothers in the Cyber Defense Labs, Jinku Li, Michael Grace, Wu Zhou, Chiachih Wu, Lei Wu, Kunal Patel. I am thankful to my friends at NC State University, Pu Yang, Xiaoling Huang, Yi Xu, Rong Jin, Zhiming Shen, Cunfang Shen, Wenbo Shen, Juan Liu. Special thanks to my old friends from Suzhou University and University of Science and Technology of China, Yi Luo, Yan Zhang, Xuhui Li, Zhen Huang, Chunguang Tang, and all the brothers from 311, Tao Wang, Lei Jin, Anquan Zheng, Kun Shen, Mingdong Zhang. I really miss our good old times together. Third, I am also grateful to organizations that generously funded my graduate research. The work in this dissertation was supported in part by US National Science Foundation (NSF) and the US Army Research Office (ARO) under several grants. As usual, any opinions, findings, and conclusions or recommendations expressed in this dissertation do not necessarily reflect the views of the NSF and the ARO. Last but certainly not least, I deeply appreciate the support and encouragement from my family, in particular my wife, Lily Xu, and our kid, Kevin Zhou. I cannot imagine how I can survive sleepless nights during my PhD journey without the strength gained from them. I am grateful to my parents, who were giving me the life and the best they have, and my aunt and uncle, who gave me the opportunity of better education, which has changed my entire life. Thanks for the help and encouragement from my two sisters. I cannot forget how they protected me when I was young. Also I would like to thank the help from my parents-in-law's family. iv TABLE OF CONTENTS LIST OF TABLES ...................................... vii LIST OF FIGURES ..................................... viii Chapter 1 Introduction ................................... 1 1.1 Background and Problem Overview . 1 1.2 Our Approach . 2 1.3 Dissertation Contributions . 4 1.4 Terminology . 5 1.5 Dissertation Organization . 6 Chapter 2 Related Work .................................. 7 2.1 Android Malware Detection . 7 2.2 Android Malware Defense . 8 2.3 Mobile Application Security . 9 2.4 Software Fault Isolation . 11 2.5 Mobile Application Sandbox . 11 Chapter 3 Detecting Malicious Apps in Official and Alternative App Market- places ....................................... 13 3.1 Introduction . 13 3.2 Design . 15 3.2.1 Detecting Known Android Malware . 16 3.2.2 Detecting Unknown Android Malware . 19 3.3 Evaluation . 21 3.3.1 Detecting New Samples of Known Malware . 22 3.3.2 Detecting Zero-day Malware . 26 3.3.3 Summary of Detected Malware . 28 3.4 Discussion . 28 3.5 Summary . 29 Chapter 4 Dissecting Android Malware: Characterization and Evolution ... 31 4.1 Introduction . 31 4.2 Malware Timeline . 33 4.3 Malware Characterization . 35 4.3.1 Malware Installation . 35 4.3.2 Activation . 41 4.3.3 Malicious Payloads . 42 4.3.4 Permission Uses . 47 4.4 Malware Evolution . 48 4.4.1 DroidKungFu . 48 4.4.2 AnserverBot . 50 v 4.5 Malware Detection . 51 4.6 Discussion . 53 4.7 Summary . 55 Chapter 5 Sandboxing Third-party Android Apps with Hybrid User-level Sandboxes .................................... 56 5.1 Introduction . 56 5.2 Background and Threat Model . 58 5.2.1 Dalvik Virtual Machine . 58 5.2.2 Java Native Interface . 59 5.2.3 Dynamic Loading and Linking . 59 5.2.4 Threat Model . 60 5.3 System Design . 60 5.3.1 Overview . 61 5.3.2 Dex Sandbox . 62 5.3.3 Native Sandbox . 63 5.4 Implementation . 69 5.4.1 Dex Sandbox . 69 5.4.2 Native Sandbox . 70 5.4.3 Native Sandbox Optimizations . 73 5.5 Evaluation . 74 5.5.1 Effectiveness of AppCage . 74 5.5.2 Security Analysis . 76 5.5.3 Compatibility of AppCage . 77 5.5.4 Performance Evaluation . 79 5.6 Discussion . 81 5.7 Summary . 82 Chapter 6 Conclusion and Future Work ........................ 83 BIBLIOGRAPHY ....................................... 85 vi LIST OF TABLES Table 3.1 The number of remaining apps after filtering the repository with RECEIVE - SMS/SEND SMS permissions . 17 Table 3.2 The number (and percentage of) apps with native code not in the default location . 20 Table 3.3 Statistics of collected apps from existing app marketplaces . 21 Table 3.4 Ten known malware families used in this study . 22 Table 3.5 Essential permissions of 10 known malware families . 23 Table 3.6 The number of infected apps by 10 known malware families on 5 studied marketplaces . 24 Table 3.7 The missed known malware families by Lookout Security & Antivirus soft- ware .......................................