Challenges and Pitfalls in Malware Research
Total Page:16
File Type:pdf, Size:1020Kb
Challenges and Pitfalls in Malware Research Marcus Botacin1 Fabricio Ceschin1 Ruimin Sun2 Daniela Oliveira2 Andr´eGr´egio1 1Federal University of Paran´a(UFPR) { fmfbotacin,fjoceschin,[email protected] 2University of Florida { gracesrm@ufl.edu { [email protected]fl.edu Abstract 1 Introduction As the malware research field became more established As the malware research field has grown and became over the last two decades, new research questions arose, more established over the last two decades, many re- such as how to make malware research reproducible, how search questions have arisen about challenges not yet to bring scientific rigor to attack papers, or what is an ap- completely overcome by the malware research commu- propriate malware dataset for relevant experimental re- nity. For example, How to make malware research re- sults. The challenges these questions pose also brings pit- producible?; Does papers based on attacks strictly require falls that affect the multiple malware research stakehold- scientific rigor?; What constitutes a well-balanced and ers. To help answering those questions and to highlight appropriate dataset of malware and goodware to evalu- potential research pitfalls to be avoided, in this paper, we ate detection solutions?; How to define a relevant, ap- present a systematic literature review of 491 papers on propriated ground-truth for malware experiments?; and malware research published in major security conferences What criteria should be used to evaluate offline and on- between 2000 and 2018. We identified the most common line defense solutions? are important questions for cur- pitfalls present in past literature and propose a method rent and future research that cannot be left unanswered for assessing current (and future) malware research. Our in further papers on the field. Failing in answering those goal is towards integrating science and engineering best questions may lead to pitfalls that cause delays in anti- practices to develop further, improved research by learn- malware advances, such as: (i) offensive research are not ing from issues present in the published body of work. scientific enough to be published; (ii) the development As far as we know, this is the largest literature review of offline and online engineering solutions being devel- of its kind and the first to summarize research pitfalls oped and evaluated under the same assumptions; (iii) in a research methodology that avoids them. In total, the adoption of unrealistic and biased datasets; (iv) con- we discovered 20 pitfalls that limit current research im- sidering any AV labeled samples as ground-truth without pact and reproducibility. The identified pitfalls range measuring their distribution and relevance, among oth- from (i) the lack of a proper threat model, that compli- ers. cates paper's evaluation, to (ii) the use of closed-source As pointed out by Herley and P. C. van Oorschot [82] in solutions and private datasets, that limit reproducibility. their survey of the security research field: \The science We also report yet-to-be-overcome challenges that are in- seems under-developed in reporting experimental results, herent to the malware nature, such as non-deterministic and consequently in the ability to use them. The research analysis results. Based on our findings, we propose a set community does not seem to have developed a generally of actions to be taken by the malware research and de- accepted way of reporting empirical studies so that peo- velopment community for future work: (i) Consolidation ple could reproduce the work". In this blurry scenario, all of malware research as a field constituted of diverse re- stakeholders are affected: beginners are often not aware search approaches (e.g., engineering solutions, offensive of the challenges that they will face while developing their research, landscapes/observational studies, and network research projects, being discouraged after the first unex- traffic/system traces analysis); (ii) design of engineering pected obstacles; researchers facing those challenges may solutions with clearer, direct assumptions (e.g., position- not be completely aware that they are falling for a pitfall; ing solutions as proofs-of-concept vs. deployable); (iii) industry experts often do not understand the role of aca- design of experiments that reflects (and emphasizes) the demic research in security solutions development; paper target scenario for the proposed solution (e.g., corpora- reviewers end up with no guidelines about which project tion, user, country-wide); (iv) clearer exposition and dis- decisions are acceptable regarding the faced challenges, cussion of limitations of used technologies and exercised making it difficult to decide what type of work is good norms/standards for research (e.g., the use of closed- work. source antiviruses as ground-truth). Although understanding malware research challenges 1 and pitfalls is crucial for the advancement of the field and others may have different understandings. Thus, we in- the development of the next generation of sound anti- tend to stimulate the security community to discuss each malware security solutions, a few work have focused on pitfall and how they should be approached. attempting to answer those questions and shedding some From the lessons learned, we proposed a set of recom- light on these pitfalls. Previous works have only consid- mendations for different stakeholders (researchers, re- ered isolated malware research aspects, such as informa- viewers, conference/workshop organizers, and industry), tion flow [34] or sandbox limitations [127]. Therefore, aiming at the next generation of research in the field in this paper, we decided to investigate such phenomena and to discuss open challenges that the community has scientifically: we conducted a systematic review of the yet to tackle. For example, we propose for researchers: malware literature over a period of 18 years (2000-2018), clearly state their assumptions about malware and good- which encompasses 491 papers from the major security ware repositories to allow for bias identification and re- conferences. Based on this systematization, we discuss search reproducibility; for reviewers: be mindful of the practices that we deemed scientific, thus reproducible, Anchor bias [64] when evaluating the appropriateness of and that should be included in the gold standard of mal- a dataset, since the representativeness of the environ- ware research. ment in which an engineering tool is supposed to oper- Overall, malware research integrates science and engi- ate (corporate, home, lab) might be the most important neering aspects. Therefore, we describe \malware re- evaluation criteria, despite the dataset size; for con- search" in terms of a malware research method, accord- ferences/workshop organizers: increase support for ing to the following steps (see Figure 2): (i) Common the publication of more research on threat landscape, as Core (from the Scientific Method); (ii) Research Ob- they establish a foundation for new defense tools; and for jective Definition; (ii) Background Research; (iii) Hy- AV companies: disclose the detection engines and/or pothesis/Research Requirements; (iv) Experiment De- methods leveraged for detecting samples as part of the sign; (v) Test of Hypothesis/Evaluation of Solution; and AV labels. We do not expect that all of our proposed (vi) Analysis of Results. Based on this framework, we guidelines be fully addressed in all papers, since it can reviewed the selected literature and identified 20 falla- be almost impossible in some circumstances due to re- cies about malware research, which were described and search/experiment limitations. However, our goal is to organized according to their occurrence in the Malware position them as a statement of good research practices Research Method. The identified pitfalls range from the: to be pursued. (Reasoning Phase) unbalanced development of research To the best of our knowledge, this is the first comprehen- work, which is currently concentrated on engineering so- sive systematization of pitfalls and challenges in almost lution proposals, with a scarcity of research on threat two decades of malware research, and in which the pitfalls panoramas, which should provide the basis for support- and challenges have been placed in the context of differ- ing work on engineering solutions; (Development Phase) ent phases of a proposed Malware Research Method, with the lack of proper threat model definitions for many so- a concrete actionable roadmap for the field. We limited lutions, which makes their positioning and evaluation our critical evaluation to the malware field, because it is harder; and (Evaluation Phase) the use of private and/or the field we have experience as authors, reviewers and PC non-representative datasets, that do not streamline re- members. However, we believe that many of the points producibility or their application in real scenarios. here discussed might be extrapolated for other security domains (along with the proper reasoning and adapta- In addition to pitfalls, we also identified challenges that tions), also providing a certain level of contribution to researchers face regarding practical considerations, a step their scientific enhancement. we modeled in our proposed Malware Research Method. In summary, the contributions of our paper are threefold: For example, when developing a malware detection so- lution, researchers soon realize that many stakeholders, 1. We propose a Malware Research method that inte- such as AV companies, do not disclose