Intelligent OS X Malware Threat Detection with Code Inspection
Total Page:16
File Type:pdf, Size:1020Kb
This is a repository copy of Intelligent OS X malware threat detection with code inspection. White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/128371/ Version: Published Version Article: Pajouh, H.H., Dehghantanha, A. orcid.org/0000-0002-9294-7554, Khayami, R. et al. (1 more author) (2018) Intelligent OS X malware threat detection with code inspection. Journal of Computer Virology and Hacking Techniques, 14 (3). pp. 213-223. ISSN 2274-2042 https://doi.org/10.1007/s11416-017-0307-5 Reuse This article is distributed under the terms of the Creative Commons Attribution (CC BY) licence. This licence allows you to distribute, remix, tweak, and build upon the work, even commercially, as long as you credit the authors for the original work. More information and the full terms of the licence here: https://creativecommons.org/licenses/ Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing [email protected] including the URL of the record and the reason for the withdrawal request. [email protected] https://eprints.whiterose.ac.uk/ J Comput Virol Hack Tech DOI 10.1007/s11416-017-0307-5 ORIGINAL PAPER Intelligent OS X malware threat detection with code inspection Hamed Haddad Pajouh1 · Ali Dehghantanha2 · Raouf Khayami1 · Kim-Kwang Raymond Choo3,4 Received: 31 July 2017 / Accepted: 27 September 2017 © The Author(s) 2017. This article is an open access publication Abstract With the increasing market share of Mac OS X over 91% detection accuracy with 3.9% false alarm rate. operating system, there is a corresponding increase in the We also utilize Synthetic Minority Over-sampling Technique number of malicious programs (malware) designed to exploit (SMOTE) to create three synthetic datasets with different dis- vulnerabilities on Mac OS X platforms. However, existing tributions based on the refined version of collected dataset to manual and heuristic OS X malware detection techniques investigate impact of different sample sizes on accuracy of are not capable of coping with such a high rate of malware. malware detection. Using SMOTE datasets we could achieve While machine learning techniques offer promising results over 96% detection accuracy and false alarm of less than 4%. in automated detection of Windows and Android malware, All malware classification experiments are tested using cross there have been limited efforts in extending them to OS X validation technique. Our results reflect that increasing sam- malware detection. In this paper, we propose a supervised ple size in synthetic datasets has direct positive effect on machine learning model. The model applies kernel base Sup- detection accuracy while increases false alarm rate in com- port VectorMachine and a novel weighting measure based on pare to the original dataset. application library calls to detect OS X malware. For training and evaluating the model, a dataset with a combination of 152 Keywords OS X malware detection · RBF–SVM · Mach-O · malware and 450 benign were created. Using common super- Supervised classification · Cyber threat intelligence vised Machine Learning algorithm on the dataset, we obtain B Ali Dehghantanha 1 Introduction [email protected] Hamed Haddad Pajouh Malicious softwares (malware) are a serious threat to the [email protected] security of computing systems [1,2]. Kaspersky and Labs Raouf Khayami alone detected more than 121,262,075 unique malware in [email protected] 2015 [3] while Panda Labs predicted that half of security Kim-Kwang Raymond Choo issues are directly related to malware infections [4], McAffe [email protected] reported a rise of 744% OS X malware over 2015 in 2016 [5]. The increasing Mac OS X market size (second after 1 Department of Computer Engineering and Information, Microsoft Windows [6] and its fast adoption rate motivate Technology, Shiraz University of Technology, Shiraz, Iran cyber threat actors to shift their focus to developing OS X 2 School of Computing, Science and Engineering, University of malware. The “myth” that OS X is a more secure system Salford, Salford, UK only further increases malware success rate. For example, 3 Department of Information Systems and Cyber Security, The OS X Flashback Trojan successfully infected over 700,000 University of Texas at San Antonio, San Antonio, TX 78249, USA machines in 2012 [7]. Security researchers have developed a wide range of anti- 4 School of Information Technology and Mathematical Sciences, University of South Australia, Adelaide, SA 5095, malware tools and malware detection techniques in their Australia battle against the ever increasing malware and potentially 123 H. H. Pajouh et al. Fig. 1 Research methodology malicious programs, including approaches based on super- The False Alarm Rate (FAR) is the rate that a classifier vised and unsupervised machine learning techniques for wrongly detected a goodware as malware and computed as: malware detection [7]. In approaches using supervised tech- niques, tagged datasets of malicious and benign programs are FP FAR = (2) required for training. Approaches using unsupervised tech- FP + TN niques generally do not require the separation of malware and goodware, and programs are generally classified based Our research methodology is presented in Fig. 1. on observable similarities or differences [8]. The organization of this paper is as follows. Section 2 While there have been promising results on the use of discusses related research, and Sect. 3 describes our dataset machine learning in Windows and Android malware detec- development. Sections 4 and 5 presents our malware classi- tion [9,10], there has been no prior work on using machine fication and a discussion of this work, respectively. Finally, learning for OS X malware detection. This could be, per- we conclude in the last section. haps, due to the lack of a suitable research dataset and the difficulties in collecting OS X malware. In this paper, we propose a machine learning model to 2 Literature review detect OS X malware based on the Radial Base Function (RBF) in the SVM technique. This provides us a novel mea- Machine learning techniques have been used for malware sure based on application’s library calling to detect malware detection. Nauman et al. [11] used game-theoretic rough sets from benign samples. We then propose a new weighting mea- (GTRS) and information-theoretic rough sets (ITRS) to show sure for classifying OS X goodware and malware based on that a three-way decision-making approach (acceptance, the frequency of library calling. This measure weights each rejection and deferment) outperforms two-way (accept, library based on its frequency of occurrence in malware and reject) decision-making techniques in network flow analysis benign applications. for Windows malware detection. Fattori et al. [12]devel- These datasets are then evaluated using four main clas- oped an unsupervised system-centric behavioral Windows sification techniques, namely: Nave Bayes, Bayesian Net, malware detection model with reportedly 90% in accuracy. Multi Layer Perceptron (MLP), Decision Tree-J48, and Their approach monitors interactions between applications Weighted Radial Basis Function Kernels-Based Support and underlying Windows operating system for classifica- Vector Machine (Weighted-RBFSVM). The following per- tion of malicious applications. Mohaisen et al. [13] proposed formance indicators are used for evaluating the performance an unsupervised behavioral based (dynamic) Windows mal- of our machine learning classifiers: ware classification technique by monitoring file system and True Positive (TP): shows the ratio of goodware classified memory interactions and achieved more than 98% preci- as benign; sion. Huda et al. [14] proposed a hybrid framework for True Negative (TN): shows the ratio of malware correctly malware detection based on programs interactions with Win- detected as malware; dows Application Program Interface (API) using Support False Positive (FP): shows that the ratio of malware files Vector Machines (SVM) wrappers and statistical measures identified as benign; and and obtained over 96% detection accuracy. False Negative (FN): shows the ratio of goodware classified Nissim et al. [15] proposed an SVM-based Active Learn- as malware. ing framework to detect novel Windows malware using Accuracy (ACC): measures the ratio that a classifier cor- supervised learning with an average accuracy of 97%. rectly detected malware and benign samples (goodware), and Damodaran et al. [16] utilized Hidden Markov Models is computed using following formula: (HMMs) to trace APIs and Opcodes of Windows malware sequences and developed a fully dynamic approach for mal- TP + TN ware detection based on API calls with over 90% accuracy. ACC = (1) FN + TP + FP + TN Mangialardo and Duarte [17] proposed a hybrid supervised 123 Intelligent OS X malware threat detection with code inspection machine learning model using C5.0 and Random Forests (RF) algorithms with an accuracy of 93.00% for detecting Linux malware. Due to the increasing use of smart devices such as Android and iOS devices, there has been a corresponding increase in the number of Android and iOS malware [18–20]. Suarez- Tangil et al. [21], for example, proposed an Android malware detection model. Yerima et al. [22] utilized ensemble learn- ing techniques for Android malware detection and reportedly had an accuracy rate between 97.33 and 99%, with a rela- tively low false alarm rate (less than 3%). Saracino et al. [23] designed a system called MADAM which is a host-based Android malware detection. The MADAM was evaluated using real world apps. OS X malware has also been on the increase [24], but there is limited published research in OS X malware analysis and detection. For example, a small number of researchers have developed OS X malware and Rootkit detection tech- niques, and malware detectors by tracing suspicious activities in memory (like unwanted access, read, write and execute) [25–27]. However, applying machine learning to detect OS X malware is limited to the Walkup approach [28], which utilized Information Gain (IG) to select effective features for supervised classification of OS X malware.