A MALWARE ANALYSIS AND DETECTION SYSTEM FOR MOBILE DEVICES ALI FEIZOLLAH Malaya THESIS SUBMITTED IN FULFILMENTof OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY FACULTY OF COMPUTER SCIENCE AND INFORMATION TECHNOLOGY UNIVERSITY OF MALAYA KUALA LUMPUR University 2017 UNIVERSITY OF MALAYA ORIGINAL LITERARY WORK DECLARATION Name of Candidate: Ali Feizollah Matric No: WHA140017 Name of Degree: Doctor of Philosophy Title of Project Paper/Research Report/Dissertation/Thesis (“this Work”): A Malware Analysis and Detection System for Mobile Devices Field of Study: Network Security, Malware Detection I do solemnly and sincerely declare that: (1) I am the sole author/writer of this Work; (2) This Work is original; (3) Any use of any work in which copyright exists was done by way of fair dealing and for permitted purposes and any excerpt or extract from, or reference to or reproduction of any copyright work has been disclosed expressly and sufficiently and the title of the WorkMalaya and its authorship have been acknowledged in this Work; (4) I do not have any actual knowledge nor do I ought reasonably to know that the making of this work constitutes an infringement of any copyright work; (5) I hereby assign all and every ofrights in the copyright to this Work to the University of Malaya (“UM”), who henceforth shall be owner of the copyright in this Work and that any reproduction or use in any form or by any means whatsoever is prohibited without the written consent of UM having been first had and obtained; (6) I am fully aware that if in the course of making this Work I have infringed any copyright whether intentionally or otherwise, I may be subject to legal action or any other action as may be determined by UM. Candidate’s Signature Date: UniversitySubscribed and solemnly declared before, Witness’s Signature Date: Name: Designation: ABSTRACT Smartphones, tablets, and other mobile devices have quickly become ubiquitous due to their highly personal and powerful attributes. Android has been the most popular mobile operating system. Such popularity, however, also extends to attackers. The amount of Android malware has risen steeply during the last few years, making it the most targeted mobile operating system. Although there have been important advances made on malware analysis and detection in traditional PCs during recent decades, adopting and adapting those methods to mobile devices poses a considerable challenge. Power consumption is one major constraint that makes traditional detection methods impractical for mobile devices, while cloud-based techniques raise many privacy concerns. This study examines the problem of Android malware, and aims to develop and implement new approaches to help users confront such threats more effectively, consideringMalaya the limitations of these devices. First, we present a comprehensive analysis on the development of mobile malware, specifically Android, over recentof years, as well as the most useful and salient analysis and detection methods for Android malware. We also discuss a compilation of available tools for Android malware analysis. Secondly, we propose a number of new and distinctive Android malware analysis and detection methods. More specifically, we introduce AndroDialysis, which is a static analysis method. Recent research has focused on analysing Android Intent in the XML file. We propose a new method of analysing Android Intent in Java code, which includes implicit intent and explicit intent. We used a DrebinUniversity data sample, which is a collection of 5,560 applications, as well as clean data sample containing 1,846 applications. The results show a detection rate of 91% using Android Intent against 83% using Android permission. We also introduce a dynamic analysis method, AndroPsychology, in order to analyse the network communications of Android applications. We extracted 30 different features from network traffic. We then used feature selection algorithms and deep learning algorithms to build a detection model. iii The results show that network traffic is an appropriate candidate for Android malware detection. Finally, we assembled AndroDialysis and AndroPsychology in order to build a comprehensive analysis and detection system for Android, called DroidProtect. Unlike current systems that either perform analyses on the device or send the whole application to a server for analyses, our system has the distinction of extracting features on the device and analysing them on the Google App Engine servers using an offloading technique. Our extensive experiments show that the energy consumption of the proposed system is less than currently available systems. Malaya of University iv ABSTRAK Telefon pintar, tablet dan peranti mudah alih berada dimana-mana sahaja dengan begitu cepat disebabkan oleh sifatnya yang sangat peribadi dan berkuasa. Sehingga 2016, Android merupakan sistem operasi mudah alih yang paling popular di kalangan pengguna. Populariti itu meliputi penyerang juga. Bilangan perisian hasad Android telah melonjak dalam beberapa tahun kebelakangan ini, menjadikannya sistem operasi mudah alih itu yang paling disasarkan. Walaupun kepentigan kemajuan telah dibuat bagi analisis pada perisian hasad dan pengesanan dalam tradisional komputer peribadi dalam tempoh sedekad yang lalu, mengguna pakai dan menyesuaikan analisis untuk peranti mudah alih merupakan satu masalah yang mencabar. Penggunaan kuasa adalah salah satu kekangan utama yang menyebabkan kaedah pengesanan tradisional tidak praktikal untuk dilaksanakan pada peranti mudah alih, manakala teknikMalaya berasaskan awan menimbulkan banyak kebimbangan privasi. Kajian ini mengkaji masalah perisian hasad Android, yang bertujuan untuk membangunkan dan melaksanakanof pendekatan baru untuk lebih membantu pengguna bagi menghadapi ancaman tersebut, dengan mempertimbangkan had peranti mudah alih. Pertama, kami membentangkan analisis komprehensif mengenai evolusi perisian hasad mudah alih, khususnya Android, sejak beberapa tahun lepas, serta kaedah yang paling berguna dan penting bagi kaedah analisis dan pengesanan dalam pengesanan perisian hasad Android. Kedua, kami mencadangkan beberapa kaedah analisis dan pengesanan terbaru bagi perisian hasad Android. Lebih khusus lagi, kita memperkenalkanUniversity AndroDialysis yang merupakan kaedah analisis static. Kerja penyelidikan yang terbaru telah memberi tumpuan kepada menganalisis tujuan Android dalam fail XML. Kami mencadangkan kaedah terbaru menganalisis tujuan Android didalam kod Java, dimana termasuk niat tersirat dan niat yang jelas. Selepas mengekstrak tujuan, model pengesanan dibina menggunakan algoritma Bayesian Network. Kami menggunakan sampel data Drebin iaitu terdapat 5,560 koleksi applikasi terdiri daripada v 179 keluarga perisian yang berbeza, serta sampel data bersih yang mengandungi 1,846 applikasi. Keputusan menunjukkan kadar pengesanan sebanyak 91% dengan menggunakan tujuan Android terhadap 83% yang menggunakan kebenaran aplikasi Android. Kami juga memperkenalkan kaedah analisis dinamik, AndroPsychology, untuk menganalisis komunikasi rangkaian bagi aplikasi Android. Kaedah ini memberi tumpuan kepada komunikasi rangkaian yang dijana oleh aplikasi Android. Kami mengekstrak 30 ciri yang berbeza daripada rangkaian trafik. Kemudian, kami menggunakan algoritma pemilihan ciri dan algoritma pembelajaran mesin, untuk membina sebuah model pengesanan. Keputusan menunjukkan bahawa rangkaian trafik adalah calon yang sesuai untuk pengesanan perisian hasad Android. Akhir sekali, kami mengabungkan AndroDialysis dan AndroPsychology untuk membina sistem analisis dan pengesanan yang komprehensif untuk Android, yang dipanggil DroidProtect.Malaya Berbeza dengan sistem semasa yang melaksanakan analisis pada peranti atau menghantar keseluruhan aplikasi kepada pelayan untuk dianalisis, sistem ofkami membawa sesuatu yang baru dalam mengekstrak ciri pada peranti, dan menganalisis aplikasi pada pelayan Engine Google App menggunakan teknik pemunggahan. Tidak perlu dikatakan bahawa eksperimen kami yang meluas menunjukkan penggunaan sistem tenaga adalah kurang pada sistem yang dicadangkan berbanding dengan sistem yang sedia ada. University vi ACKNOWLEDGEMENTS The past three years have so far been the most interesting, challenging, and rewarding years of my life. First of all, I would like to express my sincere gratitude to my supervisor Dr Nor Badrul Anuar Bin Juma'at for his patience and knowledge during this long journey; the journey that began at the commencement of my Master’s degree. He has been a devoted mentor not only in research, but in many aspects of life. I am grateful for his tremendous academic support, and for giving me wonderful opportunities during these years. Similar profound gratitude goes to Dr Rosli Bin Salleh, who has been a patient and dedicated mentor. His support and constant faith in my work encouraged me every day to be more diligent in my research. Malaya I am also hugely appreciative of Dr Lorenzo Cavallaro from the Royal Holloway University of London, for accepting my ofcollaboration offer. His professionalism and dedication have inspired me throughout our work. Of course, this work would not be possible without the support of my beloved parents and my dear siblings. Their continuous support has given me strength to finish this study. Above all, I want to thank God for all the blessings he has bestowed upon me. His benevolenceUniversity and grace enabled me to accomplish this study. “One does not discover new lands without consenting to lose sight of the shore