Hittite Journal of Science and Engineering, 2021, 8 (2) 103–121 ISSN NUMBER: 2148–4171 DOI: 10.17350/HJSE19030000221 Gaining New Insighto M int achine-Learning Datasets via Multiple Binary-Feature Frequency Ranks with a Mobile Benign/Malware Apps Example Gurol Canbek ASELSAN, Ankara, Turkey Article History: ABSTRACT Received: 2021/01/08 Accepted: 2021/03/08 Online: 2021/06/30 esearchers compare their Machine Learning (ML) classification performances with Rother studies without examining and comparing the datasets they used in training, Correspondence to: Gurol Canbek, validating, and testing. One of the reasons is that there are not many convenient meth- ASELSAN, 06200, Ankara, TURKEY E-Mail: [email protected], ods to give initial insights about datasets besides the descriptive statistics applied to in- [email protected] dividual continuous or quantitative features. After demonstrating initial manual analysis techniques, this study proposes a novel adaptation of the Kruskal-Wallis statistical test to compare a group of datasets over multiple prominent binary features that are very common in today’s datasets. As an illustrative example, the new method was tested on six benign/ malign mobile application datasets over the frequencies of prominent binary features to explore the dissimilarity of the datasets per class. The feature vector consists of over a hun- dred “application permission requests” that are binary flags for Android platforms’ primary access control to provide privacy and secure data/information in mobile devices. Permis- sions are also the first leading transparent features for ML-based malware classification. The proposed data analytical methodology can be applied in any domain through their prominent features of interest. The results, which are also visualized in three new ways, have shown that the proposed method gives the dissimilarity degree among the datasets. Specifically, the conducted test shows that the frequencies in the aggregated dataset and some of the datasets are not substantially different from each other even they are in close agreement in positive-class datasets. It is expected that the proposed domain-independent method brings useful initial insight to researchers on comparing different datasets. Keywords: Machine learning; Binary classification; Dataset comparison; Malware analysis; Feature engineering; Quantitative analysis. INTRODUCTION he success and performance of Machine Learning Indeed, some statistical methods could be used to (ML) algorithms closely depend on the datasets describe datasets. However, those statistical approaches Tused, their sample and feature spaces, and sampling summarize a dataset based on a single feature that is quality. Researchers who build a classifier that is tra- usually continuous. A box plot, for example, visualizes ined and tested on a dataset publish their classificati- and compares the descriptive statistics such as mean, on performances in terms of standard metrics such as median, range, and outliers [2]. Likewise, the statistics accuracy, true positive rate, or F1 [1]. The classifiers related to the shape of the feature distribution, such as are compared with other classifiers that are trained skewness, kurtosis, and the number of peaks, can be and tested on different datasets via the same perfor- analyzed [3]. Dataset profiling based on other statistical mance metrics. The datasets are usually not compa- properties such as timeliness (freshness of the samples), red or analyzed. On the other hand, researchers who sample duplication, and feature density gives extra in- wish to enrich their datasets usually merge new data- sight among the compared datasets [4]. Nevertheless, sets they acquired from other sources without analy- interpreting and comparing statistical figures alone are zing them. They could not be sure how these datasets not convenient; besides, they are usually not suitable for are different from the existing ones. discrete or qualitative features. To avoid such problems, new methods should be developed to give insights about THE CASE STUDY CLASSIFICATION one or comparatively more than one dataset. Better, the PROBLEM DOMAIN methods should be enhanced by visualization. The following subheadings introduce the case study This study has proposed a method to compare datasets problem domain, the binary features to be used in com- by adapting the Kruskal-Wallis test with a novel approach parisons, and dataset usage in the related literature. to compare the medians of a prominent feature’s frequen- cies to determine if the samples come from the same po- Android Mobile-Malware Classification Problem pulation or equivalently having the same distribution. This study aims to provide a new method for the researchers to Android is a mobile platform that provides a large num- compare more than one dataset over the common binary fe- ber and a wide range of mobile applications. Android atures. The study also adopts three visualization techniques applications are developed by anyone and released on to assess the comparisons based on the proposed method’s third-party application markets besides the official mar- outputs. A developed API described in Appendix A to cal- ket named Google Play. Despite this diversity, the plat- culate and visualize the method is provided to conduct such form could be the target of malicious people who develop comparisons conveniently. or make injections into existing applications that exposes some risks against end-users. Malware authors develop The method was tested and evaluated on Android mo- and use different techniques in those applications appe- bile benign applications and malware datasets in the litera- aring as legitimate to overcome the platform’s security or ture. The mobile malware classification problem was chosen exploit human factors. Therefore, mobile malware detec- because it is a critical emerging cyber security field where tion, which labeling a given application as ‘benign’ (‘ne- ML-based classification approaches are highly studied and gative’) or ‘malign’ (‘positive’, also known as ‘malware’), practiced in the literature and the industry to enhance the is one of the urging areas to be studied by the security capacities related to the human factor [5]. The results of the sector and academia. Experts examine the applications proposed comparison method summarized in Section 6 are manually with the help of specialized tools (e.g., reverse encouraging, and shed light on using datasets on malware engineering software) and decide whether they are be- classification. Note that the proposed method is not specific nign or malign. This human-involved process is called to malware analysis, and it is expected that it could be used malware analysis [6]. In addition to dynamic malware in any other area for comparing datasets in binary and even analysis that concentrates on applications’ behaviors ob- multi-class classification problems. served at run-time, static malware analysis examines bi- The rest of the paper is organized as follows. Section naries, files, and codes to classify Android malware from 2 introduces the classification problem domain. Section 3 benign applications [7]. describes and demonstrates techniques for an initial ma- nual analysis of the reviewed datasets, namely basic quan- Mobile Application Permission Requests as titative comparison of sample/feature spaces and binary-fe- Features ature space graphical analysis. It summarizes the negative and positive-class datasets to be compared in this study. Manual analysis is impossible to conduct, considering Two suggested graphics, one of which is provided online as the excessive number of applications. Solely in Google an interactive chart, to support such analysis are also de- Play Store, on average, 3,700 new mobile applications are 121 released every day [8]. To some degree, machine learning – monstrated. Section 4 presents the followed methodology 103 and the activities for comparing the datasets from different comes as a promising solution to classify malware among perspectives, including how to aggregate datasets. Section 5 many mobile applications based on various features [9]. explains the proposed comparison method based on a novel Android’s permission mechanism limits the specific ope- 2021, 8 (2) adaptation of the Kruskal-Wallis test. Section 6 provides the rations performed by applications or provides ad hoc ac- dataset comparison results enhanced with the suggested cess to particular data at the end-users discretion [10]. If visualization techniques. The last two sections present the an application is required to initiate a phone call without discussion and summarize the advantages of the proposed going through the standard dialer user interface for the comparison methods and outline this study’s contributions. user to confirm the call, for example, it must manifest Appendix A lists online supplementary materials (open-so- or request CALL_PHONE permissions. Please, refer to urce API, interactive chart, and datasets). Appendix B sur- Android API (Application Programming Interface) docu- veys the related chosen pieces of work about Android app- mentation for the list of the permissions and their desc- G. Canbek/ Hittite JSci Eng, lication permissions and highlights the Android permission riptions [11]. For static analysis, application permissions mechanism’s significant aspects related to static malware requested are the first natural and noticeable (i.e., promi- analysis. nent) feature category to be examined among the wide 104