Linköping University | Department of Computer and Information Science Master’s thesis, 30 ECTS | Datateknik 202017 | LIU-IDA/LITH-EX-A--2017/072--SE Evaluation of two vulnerability scanners accuracy and consis- tency in a cyber range Utvärdering av två sårbarhetsscanners med avseende på träff- säkerhet och konsekvens i en cyber range Erik Hyllienmark Supervisor : Chih-Yuan Lin Examiner : Kristian Sandahl Linköpings universitet SE–581 83 Linköping +46 13 28 10 00 , www.liu.se Upphovsrätt Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publicer- ingsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka ko- pior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervis- ning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säker- heten och tillgängligheten finns lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsman- nens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/. Copyright The publishers will keep this document online on the Internet - or its possible replacement - for a period of 25 years starting from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to down- load, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/. © Erik Hyllienmark Abstract One challenge when conducting exercises in a cyber range is to know what applications and vulnerabilities are present on deployed computers. In this paper, the reliability of application- and vulnerability reporting by two vulnerability scanners, OpenVas and Nexpose, have been eval- uated based on their accuracy and consistency. Followed by an experiment, the configurations on two virtual computers were varied in order to identify where each scanner gathers informa- tion. Accuracy was evaluated with the f1-score, which combines the precision and recall metric into a single number. Precision and recall values were calculated by comparing installed ap- plications and vulnerabilities on virtual computers with the scanning reports. Consistency was evaluated by quantifying how similar the reporting of applications and vulnerabilities between multiple vulnerability scans were into a number between 0 and 1. The vulnerabilities reported by both scanners were also combined with their union and intersection to increase the accu- racy. The evaluation reveal that neither Nexpose or OpenVas accurately and consistently report installed applications and vulnerabilities. Nexpose reported vulnerabilities better than OpenVas with an accuracy of 0.78. Nexpose also reported applications more accurately with an accuracy of 0.96. None of the scanners reported both applications and vulnerabilities consistently over three vulnerability scans. By taking the union of the reported vulnerabilities by both scanners, the accuracy increased by 8 percent compared with the accuracy of Nexpose alone. However, our conclusion is that the scanners’ reporting does not perform well enough to be used for a reliable inventory of applications and vulnerabilities in a cyber range. Acknowledgments I would like to thank my supervisor, Teodor Sommestad, at the Swedish Research Defence Agency. Teodor has continuously given me clear feedback and brought new ideas into this thesis work. I am grateful to my examiner Kristian Sanddahl and supervisor Chih-Yuan Lin for supporting me throughout this process. iv Contents Abstract iii Acknowledgments iv Contents v List of Figures vii List of Tables viii 1 Introduction 1 1.1 Cyber range event . 1 1.2 Motivation . 2 1.3 Research questions . 3 1.4 Delimitations . 3 2 Background 4 2.1 Vulnerability scanning . 4 2.2 Reporting standards . 4 2.3 Cyber Range And Training Environment . 5 2.4 Tools requirements . 5 3 Related work 6 3.1 Vulnerability scanners . 6 3.2 Environment validation . 7 3.3 Ourwork ............................................. 7 4 Tools 8 4.1 OpenVas.............................................. 8 4.2 Nexpose Community . 9 4.3 Other scanners . 9 5 Method 10 5.1 Experimental setup . 10 5.2 Definitions and techniques . 12 5.3 Classification methodology . 13 5.4 Metrics .............................................. 16 6 Results 17 6.1 Accuracy ............................................. 17 6.2 Consistency . 17 6.3 Interfering with scanners information gathering . 18 7 Discussion 21 v 7.1 Results . 21 7.2 Method . 23 8 Conclusion 26 9 Appendix 27 9.1 Network . 27 9.2 Evaluation . 28 9.3 Inconsistencies evaluation . 33 9.4 Experiment............................................ 34 Bibliography 38 vi List of Figures 1.1 Cyber range event . 2 1.2 Cyber range event phases . 2 4.1 OpenVas architecture . 8 4.2 Nexpose architecture . 9 5.1 Organization . 10 5.2 Applications, Dynamic Link Memory, Kernel mode and User mode . 11 5.3 Collect ground truth, applications . 14 5.4 Collect ground truth, vulnerabilities . 14 5.5 Confirmgroundtruth........................................ 15 5.6 Verify scan result samples . 15 6.1 Vulnerabilities comparison . 18 6.2 OpenVas "Control Panel uninstall" . 19 6.3 Nexpose "Control Panel uninstall" . 19 6.4 OpenVas "uninstall registry" . 19 6.5 Nexpose "uninstall registry" . 19 6.6 OpenVas "program files moved" . 19 6.7 Nexpose "program files moved" . 19 6.8 OpenVas "/usr/bin" . 19 6.9 Nexpose "/usr/bin" . 19 6.10 OpenVas "App Paths" . 19 6.11 OpenVas "/usr/sbin" . 19 6.12 Nexpose "services disabled" . 20 6.13 Combination 1, OpenVas . 20 6.14 Combination 1, Nexpose . 20 6.15 Combination 2, OpenVas . 20 vii List of Tables 2.1 Event incidents . 5 4.1 Discarded scanners . 9 5.1 Vulnerability verdict techniques . 12 6.1 Accuracy (abbreviations: false positives (FP), false negatives (FN) and true positives (TP)) 17 6.2 Number of reported applications and vulnerabilities . 18 6.3 Consistency evaluation . 18 9.1 CRATE organization . 27 9.2 Inconsistencies scans 1-2, OpenVas . 33 9.3 Inconsistencies scans 1-3, OpenVas . 33 9.4 Inconsistencies scans 1-2, Nexpose . 34 9.5 Inconsistencies scans 1-3, Nexpose . 34 viii 1 Introduction A cyber range consists of hardware, software, support tools and other capabilities which creates a good environment to arrange events in cyber space [8]. Real world scenarios can be created by forming networks with different configurations. Virtualization is often used to increase the scala- bility of events in cyber ranges [3], which makes it possible to deploy multiple virtual computers on a single server. Cyber ranges are valuable assets to practice, conduct research and do testing to improve defense against cyber-attacks [8]. Cyber ranges are often used in the military, but also in academia and the private sector [10]. It is desirable for system administrators to do reliable inventory of cyber ranges during events. A reliably inventory of a cyber range includes identifying what operating systems are deployed on the computers, how the computers are configured and if the computers contain vulnerabilities. The in- ventory is helpful before an exercise to validate if the cyber range has been correctly set up. During an exercise it is desirable to know if configurations on computers have changed. Computer inven- tory management is a difficult task and the need for automation increase as the ranges becomes bigger [3]. Vulnerability scanners can be used to automatically gather configurations and vulnera- bilities on virtual computers. However, vulnerability scanners often produce erroneous results [2]. The vulnerability scanning tools OpenVas and Nexpose has been evaluated and studied in this paper. The evaluation is based on the metrics accuracy and consistency. The evaluation is followed by an experiment, which was made to better understand how each scanner gather information. 1.1 Cyber range event Two teams usually play against each other in a cyber range event. The "blue" team is a group of defenders and is supposed to detect and protect against cyber-attacks. The "red" team consists of attackers and they perform real world attacks with the objective to gain access to other computers and perform malicious activities. During an event, a "Range Support" team typically operates the support tools. The "white" team is responsible