A Performance Analysis of Intru- Sion Detection with Snort and Se

Linköping University | Department of Computer and Information Science Master’s thesis, 30 ECTS | Datateknik 2021 | LIU-IDA/LITH-EX-A--21/068--SE

A Performance Analysis of Intru- sion Detection with Snort and Se- curity Information Management En Prestandaanalys av Intrångsdetektering med Snort och Hantering av Säkerhetsinformation

Christian Thorarensen

Supervisor : Mohammad Borhani Examiner : Andrei Gurtov

External supervisor : Villiam Rydfalk

Linköpings universitet SE–581 83 Linköping +46 13 28 10 00 , www.liu.se Upphovsrätt

Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publicer- ingsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka ko- pior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervis- ning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säker- heten och tillgängligheten ﬁnns lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/.

The publishers will keep this document online on the Internet - or its possible replacement - for a period of 25 years starting from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/.

Network intrusion detection systems (NIDSs) are a major component in cybersecurity and can be implemented with open-source software. Active communities and researchers continue to improve projects and rulesets used for detecting threats to keep up with the rapid development of the internet. With the combination of security information management, automated threat detection updates and widely used software, the NIDS security can be maximized. However, it is not clear how different combinations of software and basic settings affect network performance. The main purpose in this thesis was to find out how multithreading, standard ruleset configurations and near real-time data shipping affect Snort IDS’ online and offline performance. Investigations and results were designed to guide researchers or companies to enable maximum security with minimum impact on connectivity. Software used in performance testing was limited to Snort 2.9.17.1-WIN64 (IDS), Snort 3.1.0.0 (IDS), PulledPork (rule management) and Open Distro for Elasticsearch (information management). To increase the replicability of this study, the experimentation method was used, and network traffic generation was limited to 1.0 Gbit/s hardware. Offline performance was tested with traffic recorded from a webserver during February 2021 to increase the validity of test results, but detection of attacks was not the focus. Through experimentation it was found that multithreading enabled 68-74% less runtime for offline analysis on an octa-thread system. On the same system, Snort’s drop rate was reduced from 9.0% to 1.1% by configuring multiple packet threads for 1.0 Gbit/s traffic. Secondly, Snort Community and Proofpoint ET Open rulesets showed approximately 1% and 31% dropped packets, respectively. Finally, enabling data shipping services to in- tegrate Snort with Open Distro for Elasticsearch (ODFE) did not have any negative impact on throughput, network delay or Snort’s drop rate. However, the usability of ODFE needs further investigation. In conclusion, Snort 3 multithreading enabled major performance benefits but not all open-source rules were available. In future work, the shared security information management solution could be expanded to include multiple Snort sensors, triggers, alerting (email) and suggested actions for detected threats. Acknowledgments

First, I am extremely grateful to Andrei Gurtov and Mohammad Borhani for the great feedback during the whole thesis project. Secondly, a big thank you to MindRoad AB for providing me with this assignment and a great work environment at their COVID-safe ofﬁce in Linköping Science Park. Finally, I would like to thank Cornelia Folke for her love, support and never-ending belief in me.

iv Contents

Abstract iii

Acknowledgments iv

Contents v

List of Figures viii

List of Tables ix

Listings x

1 Introduction 1 1.1 Motivation ...... 1 1.2 Aim...... 2 1.3 Research questions ...... 3 1.4 Delimitations ...... 3 1.5 MindRoad AB ...... 4

2 Theory 5 2.1 Intrusion detection ...... 5 2.2 Intrusion detection system categories ...... 5 2.3 Intrusion prevention ...... 7 2.4 IDS and IPS differences ...... 7 2.5 Snort ...... 9 2.5.1 Snort architecture ...... 9 2.5.2 Snort snifﬁng, logging and network interfaces ...... 9 2.5.3 Snort pcap readback ...... 10 2.5.4 Snort rules ...... 12 2.5.5 Snort NIDS ...... 13 2.5.6 Snort packet I/O and LibDAQ ...... 13 2.5.7 Preprocessors ...... 13 2.5.8 PulledPork ...... 14 2.6 Rulesets ...... 14 2.6.1 Snort Community Ruleset ...... 14 2.6.2 Snort Subscriber Ruleset ...... 14 2.6.3 Snort Registered Ruleset ...... 14 2.6.4 Proofpoint Emerging Threats Open Rules ...... 15 2.6.5 Proofpoint Emerging Threats Pro Rules ...... 15 2.7 Elastic Stack ...... 15 2.7.1 Elasticsearch ...... 16 2.7.2 Kibana ...... 16 2.7.3 Logstash ...... 16

v 2.7.4 Beats ...... 18 2.7.5 Open source and software licenses ...... 18 2.8 Open Distro for Elasticsearch ...... 18 2.9 D-ITG ...... 18

3 Related Work 19 3.1 IDS: Detection of attacks ...... 19 3.2 IDS: Performance, load and stress testing ...... 20 3.3 IDS: Packet I/O performance ...... 22 3.4 IDS: Network trafﬁc generation for testing ...... 22 3.5 IDS: Encrypted trafﬁc ...... 23 3.6 Security information and event management ...... 24 3.7 Related work compared to this thesis ...... 25

4 Method 26 4.1 Pre-study ...... 26 4.1.1 Experimentation in software engineering ...... 26 4.1.2 Definitions in experimentation ...... 27 4.1.3 Methods in related work ...... 28 4.2 Experiment planning ...... 29 4.2.1 Snort, PulledPork and SIEM ...... 29 4.2.2 Variable definitions ...... 29 4.2.3 Datasets ...... 30 4.2.4 Snort rulesets ...... 33 4.2.5 Snort performance test rule and payload ...... 34 4.2.6 D-ITG network traffic generation configurations ...... 34 4.2.7 SIEM integration and test environment ...... 35 4.2.8 Information management ...... 35 4.2.9 Snort 3 multiple packet threads and DAQ ...... 37 4.3 Experiment design ...... 37 4.3.1 Experiment: Snort pcap readback and packet threads ...... 37 4.3.2 Experiment: Snort pcap readback, standard rulesets and policies . . . . 39 4.3.3 Experiment: Snort IDS real-time performance ...... 41 4.4 Experiment presentation ...... 44

5 Results 45 5.1 Snort pcap readback and packet threads ...... 45 5.2 Snort pcap readback, standard rulesets and policies ...... 47 5.2.1 MR1: Snort 2 Community Max-detect alerts ...... 47 5.2.2 MR1: ET Open alerts ...... 49 5.2.3 MR1: Snort 2 built-in rules alerts ...... 49 5.2.4 MR1: All rulesets ...... 49 5.3 Snort IDS real-time performance ...... 50 5.3.1 Generated network trafﬁc with and without Snort ...... 50 5.3.2 Snort 3 packet threads ...... 50 5.3.3 Snort 2 open-source rulesets ...... 51 5.3.4 SIEM integration ...... 51 5.3.5 Snort 3 DAQs: pcap vs afpacket ...... 52

6 Discussion 53 6.1 Results ...... 53 6.1.1 Snort pcap readback and packet threads ...... 53 6.1.2 Snort pcap readback and rulesets ...... 54

vi 6.1.3 Snort rulesets and detection of attacks ...... 54 6.1.4 Snort IDS real-time performance ...... 55 6.1.5 SIEM integration ...... 55 6.1.6 Snort 3 DAQs: pcap vs afpacket ...... 55 6.1.7 Snort IDS real-time performance hypotheses ...... 56 6.2 Method ...... 56 6.2.1 Experimentation ...... 56 6.2.2 Datasets ...... 56 6.2.3 Snort IDS and environment ...... 57 6.2.4 Snort, NIC and truncation ...... 57 6.2.5 SIEM ...... 58 6.2.6 Network traffic generation ...... 58 6.2.7 Hardware ...... 58 6.2.8 Source criticism ...... 58 6.3 Validity ...... 59 6.4 The work in a wider context ...... 59 6.4.1 Snort IDS and encrypted traffic ...... 59 6.4.2 Confidentiality ...... 60 6.4.3 Power consumption ...... 60

7 Conclusion 61

A Appendix 63 A.1 Terminology ...... 63 A.2 Excluded results ...... 65 A.3 Snort 2.9.17.1 on Windows 10 ...... 66 A.4 Open Distro for Elasticsearch install instructions ...... 71 A.5 Metricbeat OSS install and Open Distro for Elasticsearch integration ...... 72 A.6 Logstash OSS, Snort and Open Distro for Elasticsearch integration ...... 74 A.7 Snort commands ...... 76 A.7.1 Snort 2.9.17.1-WIN64 commands ...... 76 A.7.2 Snort 3.1.0.0 commands ...... 76 A.8 Open Distro for Elasticsearch screenshots ...... 77

Bibliography 79

vii List of Figures

2.1 An NIDS example network, including one IDS computer per network segment. . 6 2.2 An HIDS example network, including one HIDS per server...... 7 2.3 A DIDS example network, including four NIDSs and one central management station...... 8 2.4 A simpliﬁed view of the Snort architecture and packet processing ﬂow...... 9 2.5 Snort multithreading architecture...... 10 2.6 Network interfaces in Windows. The IP Address and Device Name columns have been truncated...... 11 2.7 Network interfaces in Ubuntu Linux...... 11 2.8 The Elastic Stack and its components...... 15

4.1 A visualization of experimentation and its components...... 28 4.2 A simpliﬁed model of a single server environment hosting a web-based application and using Snort as a packet logger...... 31 4.3 A simpliﬁed model of the security information and event management integration used when testing...... 35 4.4 Snort alerts visualized with Open Distro for Elasticsearch and Kibana...... 36 4.5 An example CPU visualization of a Snort test run...... 36 4.6 The Snort pcap readback process...... 38 4.7 A testing environment for a single server hosting a web-based application and using Snort for intrusion detection...... 41

A.1 A system and Snort process dashboard populated by Metricbeat OSS...... 78 A.2 A Snort 2 alert dashboard populated by Filebeat OSS and Logstash OSS...... 78 A.3 A Snort 3 alert dashboard populated by Filebeat OSS and Logstash OSS...... 78

viii List of Tables

2.1 A collection of basic Snort options...... 11 2.2 A collection of basic Snort rule options...... 12

4.1 Packet length statistics generated with Wireshark for the MR1 dataset...... 32

5.1 Snort 2 pcap readback and packet threads tests from Section 4.3.1.2. 8-thread and 4-thread refer to HP-i7 and Asus-i5, respectively...... 45 5.2 Snort 3 pcap readback, packet threads and MR1 test results, with tests deﬁned in Section 4.3.1.2...... 46 5.3 Snort 3 pcap readback, packet threads and DARPA test results, with tests deﬁned in Section 4.3.1.2...... 46 5.4 Snort 2 pcap readback and rulesets test results. Tests were planned in Section 4.3.2.2. 48 5.5 Snort 3 pcap readback and rulesets test results. Tests were planned in Section 4.3.2.2. 48 5.6 Snort 2 IDS test results from Section 4.3.3...... 50 5.7 Snort 3 IDS with DAQ afpacket test results from Section 4.3.3...... 50 5.8 Snort 2 IDS rulesets test results from Section 4.3.3. Max-detect has been shortened to M-d...... 51 5.9 Snort 2 IDS and SIEM integration test results from Section 4.3.3...... 52 5.10 Snort 3 IDS pcap vs afpacket test results from Section 4.3.3...... 52

A.1 Snort 2 pcap readback and rulesets test results for Asus-i5. Tests were planned in Section 4.3.2.2...... 65

ix Listings

2.1 Logstash input sample code from logstash.conf on Windows...... 17 2.2 Logstash ﬁlter sample code from logstash.conf on Windows...... 17 2.3 Logstash output sample code from logstash.conf on Windows...... 17

x 1 Introduction

1.1 Motivation

Cybersecurity is a business risk for all IT organizations. Many of them have their data and services available twenty-four hours a day through cloud services and storage. Customers and users expect personal and business-related information to be kept safe, secure and available at all times. The fact that services and company data are accessible online to authorized users means that they can also be vulnerable to network intrusions. A network intrusion can be used to steal sensitive data, carry out blackmail, make services unavailable and carry out additional cyberattacks. Consequences of network intrusions include loss of customers, dam- aged reputation, negative impact on shareholders and economic loss. Typical threats that can be enabled through network intrusions are ransomware, viruses, worms, malware, spyware, botnets, spam, phishing and Distributed Denial-of-Service (DDoS) [21]. Most successful IT businesses follow the current trends to maximize their proﬁts. These trends include mobile payments, cloud services, big data, social media, internet of things (IoT) and electronic commerce, all of which increase the risks for cyberattacks. Additionally, the following expected changes in global internet trafﬁc from 2018 to 2023 will add to the need for cybersecurity [21]:

• DDoS attacks will increase from 7.9 billion to 15.4 billion.

• Internet users will increase from 3.9 billion to 5.3 billion.

• Connected network devices will increase from 18.4 billion to 29.3 billion.

• Public Wi-Fi hotspots will increase from 169 million to 628 million.

• Mobile connectivity subscribers will increase from 5.1 to 5.7 billion.

• Downloaded mobile applications will increase from 194 billion to 299 billion.

Having more connected users results in more targets for hackers. An increase in downloaded applications will result in a larger attack surface for attacking the users and businesses. From 2018 to 2020, more than 3750 actual data breaches were identiﬁed by the Identity Theft Resource Center [1]. Only during 2019, more than 163 million data records were leaked. Because the breach data only includes reported and conﬁrmed incidents, the actual amount of

1 1.2. Aim breaches may be even higher. To invest in cybersecurity for business, government, financial and education sectors should be an obvious choice, considering the available data. Breaches and online threats have for the last years been shared frequently by the media as the interest has grown [21]. Some hackers, or cybercriminals, are motivated by financial gains. Easy targets are more likely to be subject to attacks, even if the potential payout is small. By installing the proper protection and tools, the effort and risk for attackers will increase and result in a business being less likely to be a target for an attack [40]. Being able to detect attacks and putting the most basic protection in place is a good start for a small business. The open-source software Snort has gained the reputation of being the de facto standard for intrusion detection [75]. It is used by more than 1200 companies [39], e.g., Cisco Sys- tems. More specifically, Snort is a network intrusion detection system (NIDS) and intrusion prevention system (IPS). Snort can be installed and configured to analyze all network traffic with the objective to find, prevent and warn against network intrusions and known attacks. The analysis of network traffic can be customized using Snort rulesets which include matching patterns. New rules are constantly added, and current rules are improved by the Snort Community. A security information and event management (SIEM) solution can collect and store any log data for analysis. More importantly, it can solve the business requirements of having security controls in place, monitoring important events and providing visualizations of security tools being effective [77, p. 3]. Some of the key features of a good SIEM solution include [2, pp. 3–4]:

• Easy installation, conﬁguration and management.

• Scalable deployment.

• Real-time monitoring and analysis.

• Customizable dashboards, charts and log queries for displaying data.

• Detection of threats and conﬁgurable alerts.

With Snort and Open Distro for Elasticsearch, all the features can be covered. These tools help create a complete solution while remaining free and open source, so that any individual or organization can take a simple step towards better cybersecurity.

1.2 Aim

The purpose of this study is to propose methods for installing, configuring and managing Snort for network intrusion detection. Firstly, a few different alternatives for installation, configuration and management will be suggested. Secondly, a performance analysis with the different alternatives will be completed. Finally, the results of the whole study will be discussed. The proposed method will include Open Distro for Elasticsearch and Snort as a security information and event management (SIEM) solution. Web-based systems, and companies developing them, are increasing in number. There- fore, the need for suitable methods for quickly implementing and maintaining network intrusion detection is also increasing. This study can help the reader gain a basic understanding of how to fulfil certain security requirements and to know beforehand if connectivity will be an issue. Connectivity is assumed to be an issue when the throughput or network delay between a client and application server is severely affected by Snort. A company employee or IT student should be able to install this complete Snort and Open Distro for Elasticsearch intrusion detection solution with minimal effort. Therefore, it is important that no valuable information about installation, configuration and management is left out. The contributed SIEM solution of this study has the purpose of being able to meet

2 1.3. Research questions potential minimum requirements for intrusion detection and being easily expandable to ﬁt more advanced requirements of any company.

1.3 Research questions

R1. How much does Snort IDS beneﬁt from multithreading in terms of throughput, network delay and Snort packet drop rate, with a standard ruleset?

R2. How much does Snort IDS ofﬂine analysis beneﬁt from multithreading, in terms of runtime, with a standard ruleset?

R3. When using Open Distro for Elasticsearch to visualize Snort alerts and system metrics, how is the throughput, network delay and Snort packet drop rate affected?

1.4 Delimitations

This study will not consider methods for managing Snort that can only be implemented by a large-scale company. It will focus on methods suitable for small-scale companies with approximately 20 employees. The hardware will be limited to:

• D-Link DGS-1008A: An 8-port gigabit switch.

• Cat 6 Ethernet cables.

• ASUS R557L: A laptop with Intel Core i5-4210U @ 1.70 GHz (2 Cores, 4 Threads), 4 GB DDR3-1600 RAM, Realtek PCIe GbE Family Controller with 1.0 Gbit/s speed, Windows 10 Education and Ubuntu 20.04 dual boot.

• HP Pavilion dv6: A laptop with Intel Core i7-2630QM @ 2.00 GHz (4 Cores, 8 Threads), 8 GB DDR3-1333 RAM, Realtek PCIe GbE Family Controller with 1.0 Gbit/s speed, Windows 10 Pro and Ubuntu 20.04 dual boot.

• HP ProDesk 600 G1 Tower: A desktop computer with Intel Core i5-4590 @ 3.30 GHz (2 Cores, 4 Threads), 16 GB DDR3-1600 RAM, gigabit Ethernet (Intel I217-LM, 1.0 Gbit/s speed) and Windows 10 Pro.

Other delimitations include:

• No rules or rulesets will be improved or analyzed scientiﬁcally in terms of detecting attacks in this study. Related work about Snort and detection of attacks will be examined brieﬂy in Section 3.1.

• Snort settings, including preprocessors, will be conﬁgured according to Section 4.2.1.

• There are alternatives to Open Distro for Elasticsearch, e.g., Splunk, Zabbix, OSSIM and Elastic Stack. The study will not consider a comparison between SIEM tools or claim that Open Distro for Elasticsearch is a better choice in any way.

• Visualizations are included in the study for the purpose of contributing and sharing a more complete solution. They will not be evaluated.

• The experiments will only include Snort versions 2.9.17.1-WIN64 and 3.1.0.0.

• Snort will not be tested on any other operating systems than Windows 10 Pro/Educa- tion (Version 1909) and Ubuntu 20.04.2 LTS.

3 1.5. MindRoad AB

• When testing Snort with real-time trafﬁc, performance will be measured on a computer where both Snort and a trafﬁc receiver is installed. Performance of the computer that is only running Open Distro for Elasticsearch for security information management will not be considered.

• Cloud-based SIEM solutions will not be explored.

• Snort alerts will be considered as both security information and events. Therefore, the terms security information management (SIM) and security information and event management (SIEM) will be used interchangeably in this thesis.

1.5 MindRoad AB

This thesis work was performed at MindRoad AB1. The original assignment was to inves- tigate intrusion detection methods suitable for a web-based system. MindRoad employs skilled engineers for their Academy, Embedded, Application and Source divisions. Their work involves embedded systems with real-time properties, signal processing, automatic control, data communication, interactive systems, algorithms, application development and agile methods.

1https://www.mindroad.se

4 2 Theory

This chapter regards background information about intrusion detection, Snort, rulesets, security information management and related software. It is designed to aid in understanding the following chapters in this thesis. Sections 2.7 and 2.8 include software that can be used for SIEM systems.

2.1 Intrusion detection

Intrusion detection means detecting unauthorized access in a computer network. An intrusion is typically made by a computer to attempt to harm other devices in the network. An intrusion detection system (IDS) can monitor network traffic to detect intrusions. This is done by storing known attacks, called attack signatures, in a database. The system com- pares current network traffic to the signatures in order to detect possible attacks. This method is called signature detection, or the IDS can be called signature-based. An IDS can also be anomaly-based if it uses anomaly detection. This type of detection can use normal traffic profiles that are created using statistical sampling, rules or neural networks. Any network traffic that does not fit the profile could be considered an anomaly. When an IDS detects a possible intrusion, anomaly or attack, it can perform different actions. These actions can be to filter the malicious traffic, disconnect the host or notify the network administrator [20, pp. 2–3].

2.2 Intrusion detection system categories

Intrusion detection systems (IDSs) can be classiﬁed as:

• Network-based IDS (NIDS)

• Host-based IDS (HIDS)

• Distributed IDS (DIDS)

NIDSs use promiscuous mode for the computer’s network interface card (NIC) to scan all network trafﬁc that normally just passes through. When the NIC is set to nonpromiscuous

5 2.2. Intrusion detection system categories

Figure 2.1: An NIDS example network, including one IDS computer per network segment [20, p. 6]. mode, only packets with the computer’s MAC address as destination will be sent to the application layer where they can be parsed and analyzed. The placement of the NIDS computer determines which packets are available for analysis. Only packets that pass through it can be analyzed, though it can be considered CPU intensive for a single computer to analyze packets for a whole network. A solution with multiple NIDS computers, as shown in Figure 2.1, is still classified as an NIDS. For NIDSs, it is recommended to use a Switched Port Analyzer (SPAN) or a network tap. This makes the duplicated network traffic available for analysis [20, p. 5]. HIDSs use nonpromiscuous mode for the computer’s network interface card (NIC), which is the default behavior. Besides network traffic, HIDSs can also be configured to parse system calls, file system changes and system logs on the host. Every computer in the network can have its own HIDS with an individual ruleset, which is shown in Figure 2.2. An advantage with HIDSs is that certain rules can be excluded for services that the host is not running. Excluding rules will make the parsing faster due to reduced processor overhead. A disad- vantage with HIDSs is that many hosts need to be analyzed to decide which rules could be excluded without impairing the intrusion detection [20, p. 6]. DIDSs have several computers who act as sensors and a single centralized management station. A sensor can use either promiscuous or nonpromiscuous mode and will always up- load its detection results to the manager. This enables combining HIDS and NIDS solutions, having specific rules for each sensor and pushing out new rules from the manager. The communication between the sensors and the manager needs to be secured using encryption or a virtual private network (VPN), as depicted in Figure 2.3 [20, pp. 7–8].

6 2.3. Intrusion prevention

Figure 2.2: An HIDS example network, including one HIDS per server [20, p. 7].

2.3 Intrusion prevention

An inline IDS can be used for intrusion prevention by dropping packets that match certain attack signatures. This requires the inline IDS node to be placed at the network’s choke point, a certain communication link where all network trafﬁc is forced to pass through. If an attacker can get around the choke point, the inline IDS offers no protection. This can happen if a user sets up another internet connection. An intrusion prevention system (IPS) can be installed as an inline node and drop potentially malicious packets. Other functions include adding potential attackers to an IP blacklist in real-time and scanning the potential attacker. Performing counter actions for potential attackers can result in legal and connectivity issues, which is why it is generally not used without human supervision [20, p. 26].

2.4 IDS and IPS differences

When using an IDS, packets are monitored and copied in real time. Because IDSs work on data that was copied, they will allow some malicious traffic to pass before any response is possible. A benefit of using an IDS is that analysis of packets does not slow down forwarding of packets. An IDS alone usually cannot stop attacks, but it is often combined with a firewall, an IPS or other tools that can block unwanted network traffic. IDS and IPS installations are typically called sensors. Some routers and switches have preinstalled software that support deploying an IDS or IPS sensor. It is also possible to install an IDS sensor on almost any computer with sufficient power and random-access memory (RAM) [43]. An IPS sensor must be deployed inline and does not work on copied traffic. Thus, IPSs can drop packets before they arrive at their destination and are therefore more effective in stopping single-packet attacks. A sensor failure can cause all network traffic to stop, which could be catastrophic, and overloading the sensor with traffic makes the network slow. On

7 2.4. IDS and IPS differences

Figure 2.3: A DIDS example network, including four NIDSs and one central management station [20, p. 8].

8 2.5. Snort

Figure 2.4: A simplified view of the Snort architecture and packet processing flow [20, pp. 40, 44, 227]. the other hand, overloading an IDS sensor does not slow down the network. It only affects its ability to detect attacks. An IDS sensor failure does not affect the network traffic either, which means applications running on a trusted network can continue to function [68, pp. 437–442].

2.5 Snort

2.5.1 Snort architecture Snort consists of four major components: The packet sniffer, the preprocessor, the detection engine and the output module. How the traffic flows between the different components is shown in Figure 2.4. In reality, the preprocessor consists of multiple optional preprocessors and there are several different output modules to choose from. The preprocessor, detection and output subcomponents have been made into plug-ins that are compatible with the Snort plug-in API. This makes it easy to expand Snort features without changing Snort’s core func- tionality [20, pp. 39–40]. Snort 3 can run multiple packet processing threads in a single Snort process [73, p. 1]. The option --max-packet-threads n enables this feature with n packet threads. Only --max-packet- threads, without the n, uses the number of CPU cores reported by the OS instead. A simplified view of the multithreaded design is shown in Figure 2.5. To run multiple packet threads with Snort 2, it requires a new Snort process per thread with different configurations. The improved memory allocation also makes Snort 3 run more efficiently [73, p. 174].

2.5.2 Snort snifﬁng, logging and network interfaces Snort can be used in three different modes:

• Sniffer

• Logger

9 2.5. Snort

Figure 2.5: Snort multithreading architecture [50].

• Network intrusion detection system

In all modes, Snort will intercept network trafﬁc that reaches the NIC and perform different actions. If the NIC has been set to promiscuous mode, Snort gets access to packets with any destination address. Not only those that are addressed to the current node where Snort is set up. The sniffer mode can be run with the following command, which displays network packets in the console:

snort −d −e −v § ¤ ¦Here, the console refers to a terminal in Linux or the command prompt in Windows. To log ¥ packets to a ﬁle instead, one of the following commands can be used [27, pp. 9–10]. The ﬁrst command is for Windows, the second one is for Ubuntu Linux:

snort −d −e −v −l C:\Snort\log §snort −d −e −v −l /var/log/snort ¤

¦ By default, Snort will listen on network interface number one which is often not the correct ¥ choice. In Windows, the available network interfaces can be displayed using Snort and the -W flag, as shown in Figure 2.6. From the index and description columns, it can be determined that interface number five (5) is a Realtek Ethernet NIC. In Ubuntu Linux, Snort cannot be used to display network interfaces. Instead, the command ip link show can be used, as shown in Figure 2.7. The en in enp2s0 represents Ethernet and the wl in wlp3s0 represents wireless LAN. From the information in Figure 2.7, it can be determined that the system has two NICs. Snort can listen to Ethernet traffic on interface two (enp2s0), or multiple interfaces, using one of the following commands:

snort −d −e −v −i enp2s0 §snort −d −e −v −i "enp2s0 wlp3s0" ¤

¦ On Windows, the following command can be used to listen to interface ﬁve (5) [20, ¥ pp. 142–143]:

snort −d −e −v −i 5 § ¤ ¦It should be noted that Snort is missing features for proper wireless intrusion detection [67]. ¥

2.5.3 Snort pcap readback Snort can read network traffic from one or multiple packet capture files which is useful for offline testing, optimizing and debugging Snort. The packets are processed by Snort as if

10 2.5. Snort

Figure 2.6: Network interfaces in Windows. The IP Address and Device Name columns have been truncated.

Figure 2.7: Network interfaces in Ubuntu Linux.

-v Activates Snort packet sniffer with only IP and TCP/UD- P/ICMP headers. -d Adds application layer data to packet sniffer. -e Adds data link layer headers to packet sniffer. -l Activates Snort logger mode. Saves recorded packets to disk in the specified . -W Lists available network interfaces in Windows. -i Listen to a single interface or a list of interfaces. -q Quiet mode. This removes logging, Snort banner and status report from stdout. -r Activates Snort pcap readback with the specified file as input. --pcap-dir Activates Snort pcap readback. All pcap files in the specified will be read. -c Activates Snort NIDS mode with the given as configuration. -A Specifies the alert , which is an output option. For example none, fast, full or console. --max-packet-threads <#> Enables multiple packet threads with the specified number <#> as the maximum number of threads. 0, or no value, sets the number of cores reported by the system as max packet threads. -z <#> Same as --max-packet-threads.

Table 2.1: A collection of basic Snort options. they were read live from a network interface. Table 2.1 shows the pcap readback command options and some other useful options previously used in Section 2.5 [27, pp. 17–18]. The pcap format is also used by Wireshark, tcpdump and many other applications capable of packet snifﬁng [20, p. 418]. This makes Snort able to use recorded network trafﬁc from many different sources, such as the 1999 DARPA Intrusion Detection Evaluation Dataset [60]. Other locations where pcaps for testing are shared can be found in Security Onion’s open- source documentation [80].

11 2.5. Snort

msg:""; Snort will output when the rule is triggered. reference:, ; If the rule is made for a specific attack or vulnerability that previously has been defined, Snort will include a link to an external source that describes it. A valid example reference is: reference:cve, CVE-2014-0160;. Some of the supported id systems include bugtraq, cve, nessus, mcafee, osvdb, msb and url (any url). sid:; This specifies a unique identifier for each rule. Numbers 100-999,999 are reserved for Snort rules (from Cisco) and numbers >=1,000,000 can be used for local rules. rev:; Each time a rule is revised, the revision integer is incre- mented. content:[!]""; The packet payload must match the content string exactly, including case sensitivity. A single rule can include several content options.

Table 2.2: A collection of basic Snort rule options.

2.5.4 Snort rules Snort rules are sometimes synonymous with Snort attack signatures. Once a network packet matches a Snort rule, an action speciﬁed in the rule will be performed [20, p. 296]. The Snort rule syntax has the following structure [73, p. 12]:

action protocol source direction destination ( body ) § ¤ ¦Rule header components can consist of some of the following values: ¥ • Actions: log, alert, drop.

• Protocols: ip, icmp, tcp, udp.

• Direction: ->, <>.

• Source/destination address: any, !192.168.1.0/24, [192.168.1.0/24,10.10.0.0/24].

• Source/destination port: any, 80, 1:1024, 1024:, !80.

For example, the following rule will alert for any ICMP packets, including ICMP echo requests (pings):

alert icmp any any −> any any ( msg:"ICMP packet detected!"; ) § ¤ ¦In Snort 3, the rule can be simplified to only include a protocol or a service [73, p. 12]: ¥ alert icmp ( msg:"ICMP packet detected!"; ) § ¤ ¦For the rule header directions, <> means bidirectional and the <- operator does not exist. ¥ The ! operator is used for negation, meaning everything but the specified IP address or port range. CIDR notation is allowed in the Snort rules, where the number 24 in 192.168.1.0/24 indicates a block of 256 IP addresses. Other IP address formats are lists with square brackets and commas, and : is used for port ranges only. A few of the common rule options are included in Table 2.2. These can be added to the Snort rule body. For a rule to trigger an alert, all header and body conditions must be fulfilled [27, pp. 182–191].

12 2.5. Snort

2.5.5 Snort NIDS Snort can run as an NIDS by specifying which configuration file to use [20, p. 55]. The first of the following commands is for Snort 2.9.17.1 on Windows and the second is for Snort 3.1.0.0 on a Linux-based system:

snort −c C:\Snort\etc\snort.conf §snort −c /usr/local/etc/snort/snort.lua ¤

¦ One of the first configurations to make is the HOME_NET variable. The value should be ¥ changed to include the network addresses that Snort is protecting [37, p. 12]. For the rest of the Snort NIDS settings, it is recommended to follow a Snort install tutorial like Snort 3.1.0.0 on Ubuntu 18 & 20 [37]. It is also recommended to use the PulledPork tool which is included in the tutorial and described briefly in Section 2.5.8.

2.5.6 Snort packet I/O and LibDAQ On Windows, Snort uses the Npcap library to capture network traffic [20, p. 99]. Npcap replaced WinPcap which was used in earlier Snort versions, and both are based on the libpcap library [61]. Since Snort 2.9, Snort can use LibDAQ and DAQ modules. LibDAQ, The Data AcQuisition Library, can be described as an abstraction layer that Snort uses for packet I/O. Before, Snort called libpcap functions directly, which has now been replaced with LibDAQ API calls. This makes Snort more modular, and it is easier to change the data acquisition method without making changes to Snort itself. DAQ modules can be used both for pcap readback and inline operation. Afpacket, pfring, pcap and dump are examples of DAQ modules that are available [73, pp. 282–283]. DAQ afpacket supports kernel load balancing by utilizing PACKET_FANOUT ioctl (input/output control) in Linux. This allows parallel packet processing from the same network interface for increased Snort throughput [73, p. 284]. Snort 3’s multithreading feature utilizes DAQ afpacket for load balancing network traffic between each packet processing thread. It can be configured in the Snort 3 config file, snort.lua, or with the following terminal command [62]:

snort −c snort.lua −−daq−dir /usr/local/lib/daq −−daq afpacket −−daq−var §lb_total=2 −−daq−var fanout_type=hash −i eno1 −−daq−var lb_id=1 −i eno1 −−daq−var lb_id=2 −z 2 ¤

¦ ¥ 2.5.7 Preprocessors Preprocessors allow Snort to process network packets in different ways before they reach the detection engine, as shown in Figure 2.4. IP Defragmentation preprocessors, like frag3, are needed to detect attacks that are fragmented between packets. Any single packet part of the attack might not match with an attack signature. Only when reassembled will the packets be considered malicious. Defragmentation is a legitimate attack tactic to avoid intrusion detection, but also occurs naturally in networks [20, pp. 226–228]. The stream4 preprocessor can decide if packets are part of a valid TCP session or not. With stream4, Snort can identify packets that seem to be part of an established session, even though they are not. Another beneﬁt is that the Snort detection engine can ignore invalid session packets, which saves resources. This feature is called stateful inspection. Another feature that stream4 supports is stream reassembly. When it is activated, packets will be reassembled based on the session they belong to and matched against attack signatures [20, pp. 237–240, 247]. Application preprocessors can be used to decode and normalize network trafﬁc based on protocols. For example, the http_inspect preprocessor normalizes HTTP strings so that they match rules regardless of which encoding is used [20, pp. 253–254].

13 2.6. Rulesets

The ssl preprocessor enables ssl_state and ssl_version options to be included in Snort rules. Some encrypted trafﬁc will not be inspected by Snort based on trusted servers and ports conﬁgured for the ssl preprocessor [24].

2.5.8 PulledPork PulledPork is a tool that is recommended for downloading, managing and tuning rulesets in Snort [37]. It is based on the Perl programming language, and some of the features include [34]:

• Automatic recurring ruleset and IP blocklist updates.

• Setting ruleset policy to connectivity, balanced, security or max-detect.

• Great terminal output, logging and feedback after changing ruleset conﬁgurations in PulledPork.

• Support for conﬁguring Proofpoint Emerging Threats rulesets and Cisco Snort rulesets.

• GNU General Public License v2.0, permitting commercial use.

Using the balanced policy results in a ruleset optimized for both security and connectivity. Choosing the security policy results in security over connectivity. More rules will be activated, and Snort will be more aggressive in detecting attacks, sometimes causing more false positive alerts. The connectivity policy will disable some rules so that the connectivity is pri- oritized over security [37]. When connectivity is not important, i.e., for pcap readback or in a testing scenario, the max-detect policy can be used. In the max-detect subset of rules more false positives are tolerated [31].

2.6 Rulesets

2.6.1 Snort Community Ruleset The Snort Community Ruleset uses an open-source license [46] and is certiﬁed by the Cisco Talos Intelligence Group. It includes daily updates by Snort Integrators [28] and the Snort community. This ruleset is also a subset of the Snort Subscriber Ruleset [30].

2.6.2 Snort Subscriber Ruleset The Snort Subscriber Ruleset, sometimes called the VRT or Talos Ruleset, is managed by Cisco Talos [29]. Updates can occur at any time for recently discovered threats, but are usually recurring on Tuesdays and Thursdays [30]. Only Cisco customers may use the Subscriber Ruleset [42].

2.6.3 Snort Registered Ruleset The Snort Registered Ruleset is available to registered users and does not require a subscrip- tion. Updates from Cisco Talos occur with a 30-day delay compared to the Subscriber Ruleset [42]. Community rule updates, which are included, can still occur daily [30]. Both the Registered and Subscriber rulesets use a Snort Subscriber License. Any use or download of the registered rules is not allowed for commercial purposes, unless the user switches to a commercial license in agreement with Cisco [26]. For a solution or product to remain open source, only Snort community rules can be included.

14 2.7. Elastic Stack

Figure 2.8: The Elastic Stack and its components [78, p. 14].

2.6.4 Proofpoint Emerging Threats Open Rules The Proofpoint ET Open IPS/IDS ruleset is used by more than 20 000 organizations and individuals. Rules are updated and shared daily by a large and active community, under a free and open-source license [76] (BSD [70]). Supported IDS versions include [51]:

• Suricata 4.0.x, 4.1.x, 5.0.x and 6.0.x.

• Snort 2.9.x.

Snort 2.X rules can be converted to the Snort 3.X format with the Snort2lua tool, which is a manual process. Errors can occur during the conversion and that requires the user to manually resolve them [86].

2.6.5 Proofpoint Emerging Threats Pro Rules The ET Pro Ruleset requires an OEM license and includes ET Open. It has over 37 000 rules and 10-50 (or more) rules are added daily by Proofpoint’s research team [71].

2.7 Elastic Stack

The main components of the Elastic Stack are Elasticsearch, Kibana, Logstash and Beats. To- gether they support gathering, searching, analyzing and visualizing all types of data [6].

15 2.7. Elastic Stack

2.7.1 Elasticsearch Elasticsearch is a data analytics engine which can be described as near real-time, schema- less, document-oriented, easy to scale horizontally and highly customizable through many libraries. It can be integrated with any application through the REST API, to add storage and management of data. In the Elastic Stack, Elasticsearch is a required component for adding these central features [78, pp. 9, 14].

2.7.2 Kibana Kibana is often pictured as being placed on top of Elasticsearch in the stack, as shown in Figure 2.8. It adds visualization features and a GUI to Elasticsearch. Typical templates for visualizing data are line charts, pie charts, maps, tables, gauges and horizontal bars [78, p. 14].

2.7.3 Logstash Logstash is a plugin-based data flow engine that can collect, filter, convert and output data to any location. In the Elastic Stack it typically outputs filtered data to Elasticsearch [78, pp. 14– 15]. Plugin categories and some examples of plugins are:

• Input plugins [9].

– beats: Events are received from Beats. – ﬁle: Events are fetched from ﬁles. – twitter: Events are fetched using Twitter Streaming API. – github: Events are received from a GitHub webhook.

• Filter plugins [8].

– csv: Events are parsed into individual ﬁelds by specifying a separator, for example a comma. – date: Any date format can be parsed to be compatible with the Elastic Stack. – geoip: Geographical coordinates are added based on the given IP address and if it exists in the Maxmind GeoLite2 database. – json: Handles JSON format events. – mutate: Any individual ﬁeld can be mutated. Some mutations include rename, update, convert, split, join, merge and copy.

• Output plugins [11].

– csv: Events are converted to csv format. – elasticsearch: Events are stored in Elasticsearch. – email: Events can trigger an email notification which can include event data. – file: Events are written to a file. – stdout: Events are printed to the standard output of the shell (i.e., the command line tool used to run Logstash).

There are more than 200 supported plugins available which makes Logstash suitable for many different data flows [78, p. 15]. Listing 2.1 shows a working Logstash input configuration using the file plugin. start_position can be set to either beginning or end of the file. The end value results in Logstash only handling live updates. If a sincedb file exists, this setting will not have any effect, and

16 2.7. Elastic Stack

Logstash will continue handling updates from the position in the file that was last recorded. The sincedb_path setting specifies the path to the sincedb file [7].

input{ § ﬁle{ ¤ path =>"C:/PerfLogs/Admin/snort/snort_system_log.csv" start_position =>"beginning" sincedb_path =>"sincedb.txt" } }

¦ Listing 2.1: Logstash input sample code from logstash.conf on Windows. ¥

Listings 2.2 and 2.3 show working conﬁgurations of the following Logstash plugins: csv, mutate, date, elasticsearch and stdout.

ﬁlter{ § csv{ ¤ separator =>"," columns => [ "(PDH−CSV 4.0) (W. Europe Standard Time)(−60)", "\\CHRIS−HP−WIN10\Process(snort)\% Processor Time", "\\CHRIS−HP−WIN10\Process(snort)\Working Set", "\\CHRIS−HP−WIN10\Memory\Available Bytes" ] } mutate{ rename => { "(PDH−CSV 4.0) (W. Europe Standard Time)(−60)" =>"time" "\\CHRIS−HP−WIN10\Process(snort)\% Processor Time" =>"snort_%cpu" "\\CHRIS−HP−WIN10\Process(snort)\Working Set" =>"snort_working_set" "\\CHRIS−HP−WIN10\Memory\Available Bytes" =>"mem_available_ram_bytes" } convert => { "snort_%cpu" =>"ﬂoat" "snort_working_set" =>"integer" "mem_available_ram_bytes" =>"integer" "Bytes Received/sec" =>"integer" } } date{ match => ["time","MM/dd/yyyy HH:mm:ss.SSS"] } }

¦ Listing 2.2: Logstash ﬁlter sample code from logstash.conf on Windows. ¥

output{ § elasticsearch{ ¤ hosts =>"http://localhost:9200" index =>"snort_and_system_info" } stdout{} }

¦ Listing 2.3: Logstash output sample code from logstash.conf on Windows. ¥

17 2.8. Open Distro for Elasticsearch

2.7.4 Beats A Beat is a single-purpose lightweight data shipper based on the Beats open-source platform. The simplest Beat, Filebeat, forwards data from log ﬁles to a Logstash or Elasticsearch server. Metricbeat can forward metrics from Apache, MongoDB, MySQL, Nginx, PostgreSQL, Redis, System (Linux), Windows and more [10]. All Beats are based on the libbeat library, which anyone can use to create a new Beat or improve current ones. Because of its lightweight characteristic, Beats is useful for gathering log data and metrics from clients and servers with minimal impact on other services. The heavier work performed by the rest of the Elastic Stack can be left to centralized server [78, p. 15].

2.7.5 Open source and software licenses If an implementation with Elastic Stack is to remain open source, it is important that all components use the Apache license, version 2.0 (ALv2). Currently, Elastic offers the latest versions of Logstash and Beats as OSS (open-source software) distributions. However, from version 7.11 and in the future, the Elasticsearch and Kibana components will only use the Elastic License (ELv2). A feature of ELv2 is that any product using the license may not be provided to others as a managed service, unless it is in agreement with Elastic’s terms. However, free use of the software is allowed in some cases [12].

2.8 Open Distro for Elasticsearch

Open Distro for Elasticsearch (ODFE) includes open-source distributions based on Elastic- search OSS 7.10.2 and Kibana OSS 7.10.2. The project, which is supported by Amazon Web Services (AWS), features Elasticsearch OSS, Kibana OSS, alerting and enterprise-grade security. All components of ODFE use ALv2 which make them suitable for an open source SIEM solution [4].

2.9 D-ITG

Distributed Internet Traffic Generator (D-ITG) is an open-source and free software used for generating synthetic network traffic. It supports Windows, Linux, multiple processor archi- tectures, IPv4, IPv6, ICMP, TCP, UDP, SCTP, DCCP and more. D-ITG can be used to generate realistic network workloads, which is a major component in testing IDSs and other network performance research [15]. D-ITG can calculate both round-trip time (RTT) and one-way delay (OWD) for its generated traffic. The accuracy of the OWD measurement is dependent on clock synchronization between the sender and receiver, which is not included in the D-ITG tools. RTT is accurately measured without any synchronization [16, p. 11]. ITGLog, ITGRecv, ITGSend and ITGDec are D-ITG components. They can all be started on the same host or on different hosts. ITGLog starts a log server, ITGRecv starts a receiver, ITGSend starts a sender and ITGDec decodes log files from D-ITG. By specifying a script file when starting ITGSend, multiple parallel flows can be generated from the same host [16, pp. 26–27]. Both ITGRecv and ITGSend use a multithreaded design which benefits the multi- flow mode [16, p. 3].

18 3 Related Work

This chapter includes theory and ﬁndings from related work, and it is split into different sections based on research topic. It should be noted that intrusion detection for encrypted trafﬁc, Section 3.5, is included for completeness and is not part of the research questions.

3.1 IDS: Detection of attacks

Steinvall used Metasploit to generate network traffic containing 1143 attacks. The traffic was stored in the pcap format and later used for offline analyses using different rulesets and Snort versions 2.9.2.1, 2.9.9.0 and 2.9.12. Different versions of Emerging Threats [52], Snort Com- munity and Snort Subscriber (formerly VRT [30]) rulesets were included in the study. The study found that activating the Snort preprocessor resulted in 14.78% more detected attacks. An ambiguity is that Snort can have multiple preprocessors activated and there was no information about which of the Snort preprocessors were included. One way to interpret the result is to assume that activating all available or default Snort preprocessors will increase the number of detected attacks. The payload and Snort version were also shown to affect Snort’s detection rate. A comparison between the rulesets showed a major difference in the detection rate:

• Snort Community: 12.41% detection rate.

• Snort Subscriber (formerly VRT): 40.07% detection rate.

• Emerging Threats: 64.83% detection rate.

For future studies, Steinvall suggested that a balance between security and performance should be considered. Snort’s performance, other than detection of attacks, was not examined in the study [83]. Gupta and Sharma created new Snort rules and tested their effectiveness using Snort 2.9.11 and Ubuntu 16.04 server. The rules were shown to be effective in detecting the following DoS and Port Scan attacks: SYN-flood, Land attack, Mail bomb, HTTP flood, TCP reset, Xmas tree, UDP flood, DNS flood, ICMP flood, Smurf, Normal traffic, ACK scan, FIN scan, Null scan and FTP bounce. Snort did not perform well against the Ping of Death attack. Attacks were generated with Scapy [13] and normal network traffic with D-ITG [15]. Gupta

19 3.2. IDS: Performance, load and stress testing and Sharma concluded that their future research would consist of improving Snort rules for R2L and U2R attacks [47]. Khamphakdee et al. improved Snort rules and tested them using Snort 2.9.2.2 on CentOS. The snort.conf file was modified to use the MIT-DARPA 1999 dataset [60] consisting of normal and malicious network traffic. A total of 71 rules were improved for detecting network probe attacks. The attacks included portsweep attack, IP sweep attack, Satan attack, ls domain attack, ntinfoscan attack and Queso attack. When the improved Snort rules were compared to the Data Scoring Truth from MIT, a higher detection accuracy was achieved. The process of improving Snort rules was to: 1. Analyze existing Snort rules for a specific attack or rule type. 2. Use WireShark to analyze malicious data packets from the chosen dataset. 3. Modify the rules. 4. Compare the new rules’ test results to the Data Scoring Truth. The study showed that Snort rules are actively improved by researchers and suggested further research for detecting other types of attacks (DoS, U2R and R2L) [55].

3.2 IDS: Performance, load and stress testing

Brumen and Legvart investigated the performance of Snort 2.9.7.0 and Suricata 2.0.7 on two different OSs: Microsoft Windows Server 2008 SP 2 and CentOS 7 (Linux 3.10.0). Bad Traffic, Fragmented Packets, Brute Force, Evasion Techniques, DoS and Test Rules were the different categories of attacks used to generate the results. CPU utilization, RAM utilization, dropped packets and detected intrusions were used as performance metrics. The investigation showed less CPU and RAM usage on Windows but a higher percentage of dropped packets. There- fore, it indicated that Linux should be used for both Snort and Suricata. It was suggested that using an IDS on only a single server is not suited for large volumes of network traffic. Brumen and Legvart also concluded that based on their experiments, Suricata 2.0.7 had better performance than Snort 2.9.7.0. It should be noted that during the time of the study, only Suricata had multithreading support for handling packets, which the authors also discussed multiple times [18]. Fadhilah and Marzuki completed a performance analysis of Snort 3.0.0-263 and Suricata 5.0.3 using VMware and several laptops with Intel Core i5-5200U @ 2.20 GHz (2 Cores, 4 Threads) and 8 GB RAM. The testing scenario included one computer for DoS attacks, five computers for DDoS attacks and a single target computer running both a VM web server and a VM IDS (Snort or Suricata). Fadhilah and Marzuki concluded that Snort’s detection rate for DDoS and DoS attacks improved when the number of cores increased. The detection rate did not improve for Suricata when the number of cores increased. The authors mentioned that Windows was installed on all laptops, CentOS was used on the VM web server and Ubuntu was used on the IDS VMs. OS versions were not specified. The study could be classified as Snort IDS real-time performance testing, because at no point in the study was network traffic recorded into a pcap file for offline analysis [44]. Hu et al. analyzed the performance of Snort 3.0 and Suricata 4.1 in 10-100 Gbit/s networks. The hardware used for experiments consisted of: • Dell PowerEdge R740XD: A rack server with two Intel Xeon Gold 6126 @ 2.6 GHz (12 Cores, 24 Threads) processors, 196 GB (DDR4-2666 [53]) RAM and Mellanox ConnectX- 5 100GE with 100 Gbit/s speed. Used to generate network traffic. • Dell PowerEdge R740XD: A rack server with two Intel Xeon Gold 6126 @ 2.6 GHz (12 Cores, 24 Threads) processors, 196 GB (DDR4-2666) RAM and Mellanox ConnectX-5 100GE with 100 Gbit/s speed. Used to receive network traffic.

20 3.2. IDS: Performance, load and stress testing

• Dell Z9100: A 100 Gbit/s switch.

Performance metrics included throughput (number of received packets), drop rate, CPU utilization and RAM utilization. During the experiments, all measurements were saved to local storage. This was observed to result in a low operation cost and therefore hard drive usage was not included in the analysis. Software to perform the experiments consisted of:

• Iperf3: A tool for generating and receiving TCP ﬂows with 1500 bytes packet size.

• Pytbull: An IDS testing framework for testing the accuracy of Snort and Suricata.

• TRex: A tool for generating transport and application layer trafﬁc based on templates with real-world trafﬁc.

• Node_exporter, prometheus and tcpdump for measuring CPU, RAM, network utilization and drop rate.

Results were generated by running a specific test ten times and calculating the average. Which OS was used was not specified. Since a comparison between libpcap and AF_PACKET was made, it can be assumed that Linux was used. Hu et al. concluded that Snort and Suri- cata could handle a maximum of 5 Gbit/s throughput before starting to drop packets in their experiments. The performance was still considered good with less than 60 Gbit/s of network traffic, in terms of detected attacks and with a low drop rate. Another recommendation was that IDSs and the receiver of the network traffic should not run on the same server in parallel, because the interrupt handling in Linux can be overloaded causing a high drop rate. The authors ran tests with a default configuration and default ruleset for both Snort and Suricata. To increase reproducibility, both the ruleset name and version should be specified. However, they were not included in the article [50]. Gupta and Sharma analyzed the performance of Snort 2.9.11 on Windows Server 2016, Virtual Windows Server 2016 and Ubuntu Server 16.04. The experiment plan included installing Snort 2.9.11 with default configuration and the VRT ruleset including 9453 rules (also known as the Snort Subscriber ruleset [30]). All experiments had the following hardware and roles:

• Dell computer with i3-3110M CPU @ 2.40 GHz (2 Cores, 4 Threads), 8 GB RAM, Ubuntu 16.04 and D-ITG trafﬁc generator. For generating TCP trafﬁc.

• Dell computer with i3-3110M CPU @ 2.40 GHz (2 Cores, 4 Threads), 8 GB RAM, Ubuntu 16.04 and D-ITG trafﬁc generator. For generating UDP trafﬁc.

• Dell computer with i3-3110M CPU @ 2.40 GHz (2 Cores, 4 Threads), 8 GB RAM, Ubuntu 16.04 and D-ITG trafﬁc generator. For generating ICMP trafﬁc.

• HP computer with i5-3210M CPU @ 2.40 GHz (2 Cores, 4 Threads), 8 GB RAM, Ubuntu Server 16.04 and Snort 2.9.11.

• HP computer with i5-3210M CPU @ 2.40 GHz (2 Cores, 4 Threads), 8 GB RAM, Win- dows Server 2016 and Snort 2.9.11.

• D-Link DGS-1210-16 gigabit switch, connecting all computers in a LAN.

During some of the experiments, network traffic was evenly split between TCP, UDP and ICMP packets with a fixed packet size of 256 bytes, and the number of packets per second varied from 5000 to 30 000. Another set of experiments had a fix rate of 15 000 packets per second and varied the packet size from 512 to 3072 bytes. Virtually the same testing environment was set up using Microsoft Azure cloud computing service, including:

• Virtual Windows Server 2016 with Snort 2.9.11 installed.

21 3.3. IDS: Packet I/O performance

• Standard_D2s_v3 with 2 VCPU and 8GB RAM (D-ITG TCP trafﬁc generator).

• Standard_D2s_v3 with 2 VCPU and 8GB RAM (D-ITG UDP trafﬁc generator).

• Standard_D2s_v3 with 2 VCPU and 8GB RAM (D-ITG ICMP trafﬁc generator).

• Switch (virtual).

Gupta and Sharma concluded that the rate of dropped packets increased in all environments when the packet size or the number of packets per second increased. Virtual Windows Server 2016 showed poor performance when analyzing large amounts of malicious trafﬁc and did not meet the requirements of security administrators according to the authors. Ubuntu Server 16.04 showed the best performance [48]. Karim et al. experimented with Snort and the following hardware and operating systems:

• Dell OptiPlex 7010, Intel Core (TM) i5/i7 CPU, 16 GB RAM, 1.0 Gbit/s Full Duplex Intel NIC.

• Cisco Catalyst 3650 Series gigabit switch.

• Cisco 1941 Series router.

• Gigabit CAT6 Ethernet Cables.

• Windows 7 (64 bit).

• Windows Server 2012 (64 bit).

• Linux (Fedora 3.5.3-1.fc17.i686).

Speciﬁcations about processor models, RAM and Snort version were missing. Most likely, Intel Core i5-3470 CPU @ 3.2 GHz (4 Cores, 4 Threads), Intel Core i7-3770 CPU @ 3.40 GHz (4 Cores, 8 Threads) and DDR3-1600 were used [36]. It was concluded that Snort had 10% better performance on Linux than on Windows, and 8% better performance using an i7 processor compared to using an i5 processor [54].

3.3 IDS: Packet I/O performance

Gupta and Sharma did not mention which DAQ (packet I/O module) was used when analyzing Snort’s performance [48]. It can be assumed that Npcap [61] was used on Windows and libpcap was used on Linux, because pcap is the default DAQ in Snort 2.9 [27, p. 14]. Fad- hilah and Marzuki most likely used libpcap because it was mentioned in their description of Snort’s architecture. The DAQ afpacket was not mentioned [44]. Hu et al. observed a signiﬁcant improvement in throughput in high-speed networks when using afpacket compared to libpcap [50]. In an earlier study, Hu et al. compared libpcap, afpacket and PF_RING (DAQ). PF_RING was not tested with Snort (only Suricata IDS) and less packets were dropped when using afpacket compared to libpcap [49].

3.4 IDS: Network trafﬁc generation for testing

Gupta and Sharma used D-ITG to generate normal and malicious traffic. How the malicious traffic was generated, what it contained and which Snort rule detected it, were not specified. There were three different computers running D-ITG as a traffic generator simultaneously. This means that there were at least three flows. It was not specified if any client was running D-ITG in multi-flow mode, which allows multiple traffic flows from a single computer. The protocols TCP, UDP and ICMP were used for both tests performed by the authors. During the

22 3.5. IDS: Encrypted trafﬁc

first test, packets were sent at a rate of 15 000 pps (5000 pps per client), for 120 seconds. The treatment was to vary the packet sizes between 512, 1024, 1536, 2048, 2560 and 3072 bytes, for each test session. During the second test, a fix packet size of 256 bytes was used. The treatment was to vary the packets per second rate between 5000, 10 000, 15 000, 20 000, 25 000 and 30 000 pps. Each test session again lasted for 120 seconds [48]. Hu et al. used GridFTP to send data at a 10 Gbit/s rate, when evaluating Snort’s performance in a high-speed network. The data was transferred in a single flow in the first test case, and in 20 flows in a second test case. Consequently, the 20 flows setting was chosen because GridFTP previously had been observed to drop too many packets when using more flows. In the experiments, only a single sender and a single receiver were present, and the test sessions lasted one hour each. A third test case used mirrored traffic from the University of Auckland campus network 10 Gbit/s link. The GridFTP generated traffic was described as only consisting of TCP flows, and the campus traffic was described as a mix of flows. No packet size was specified, only the fact that the authors used GridFTP and jumbo frames [49]. Hu et al. completed three different experiments. First, iPerf3 [38] was used to generate 10- 100 Gbit/s traffic with a single and multiple TCP flows. The packet size was set to 1500 bytes. A test during the first experiment was run for 1800 seconds. It was repeated 10 times and the average was used as result. In the second experiment, pytbull [35] and iPerf3 were run simultaneously to add attack traffic. In the third experiment, TRex [85] was used to generate simulated real world network traffic. The characteristics of the generated traffic from TRex were flows ending at different times, different network protocols and adjustable throughput. All the experiments included a single sender, a single receiver and a switch [50].

3.5 IDS: Encrypted trafﬁc

Radivilova et al. proposed a method for decrypting TLS traffic with the purpose of detecting hidden threats with an IDS. The method was described as based on the Wireshark traffic analyzer, being automated, able to decrypt data almost online and able to intercept TLS traffic by listening to a network. There was no mention of performance metrics, such as throughput, network delay or CPU utilization, for the proposed method. The method was described in detail, but the automated solution could have been shared to increase reproducibility [72]. Erlacher et al. presented a new TLS interception proxy with real-time pcap export. The proxy exports decrypted application layer payloads together with lower layer information. Its source code has been shared by the authors [89], which increases reproducibility. Erlacher et al. claim that other tools that can be used as TLS proxies, e.g., Fiddler and GenuGate, do not support real-time, open format and lower layer information exports. All these properties make the new proxy’s exported data more suitable to analyze with an IDS. No performance metrics were mentioned in the study, except that the proxy was able to decrypt the SSL content [41]. Poh et al. completed a survey of methods for inspecting encrypted network traffic over middleboxes. The authors used an existing definition of middleboxes, "any intermediary box performing functions apart from normal, standard functions of an IP router on the data path between a source host and destination host" [19]. It was clarified that typical middlebox services include Firewall, NAT, IDS and load balancing. Poh et al. mentioned the following classifications (simplified):

• Middlebox models.

– Client-oriented: The middlebox is closer to the client and the goal is protecting the client. – Server-oriented: The middlebox is closer to the server and the goal is protecting the server. – Outsourced: Client or server connects to middlebox cloud service.

23 3.6. Security information and event management

• Middlebox methods. – Man-in-the-Middle (MitM): Client treats the middlebox as an endpoint for the encrypted traffic. The middlebox decrypts, inspects, re-encrypts and forwards the traffic to the server endpoint. In fact, the middlebox acts as both TLS client and server. – Searchable Encryption (SE): Matching encrypted keywords with encrypted rulesets to detect malicious traffic. No traffic is decrypted by the middlebox. – Access Control (AC): Client and server have access to all middleboxes and can assign read/write keys to allow access to the encrypted traffic. – Machine Learning (ML): ML is applied to inspect the network traffic without any modification or decryption. – Trusted Hardware (TH): Intel SGX [33] is used on the middlebox to enclose the decryption, inspection and re-encryption processes, which prevents the hosting service provider from accessing any related data or information. SSLsplit [74] and mitmproxy [32] are two open-source applications for MitM encrypted traffic inspection, which were brought up as examples but not analyzed or studied. Some enterprise MitM solutions included in the study’s references are Symantec proxy [17], Untangle NG Fire- wall, pfSense, WebTitan Gateway, Microsoft TMG, UserGate Web Filter and Cisco Ironport WSA [88]. The authors concluded that there is a need to fill the gap between the research and actual employment of the methods. Actual employment was mainly the MitM method where proper end-to-end encryption is broken [69].

3.6 Security information and event management

Kotenko et al. suggested an approach for constructing a security information and event management system using Elastic Stack. The system was developed as a prototype, featured in performance experiments and included in a comparison with similar systems. Supported input data for the prototype was added by using the Elastic Stack software Filebeat, Winlogbeat, Metricbeat and Packetbeat, which include syslog events, Windows event log, telemetry hardware, OS information, services information and network traffic flow information. Collected data was in the design meant to be sent through the internet to the Elastic Server. The study included a testbench with a quad-core processor @ 1.9 GHz, 16 GB DDR3 RAM, Elasticsearch 5.1.1, Kibana 5.1.1, Logstash 5.1.1, Winlogbeat 5.1.1, Nginx 1.10.2, Java(TM) SE Runtime En- vironment (build 1.8.0_73-b02) and CentOS 7 Linux 3.10.0-327.36.3.el7.x86_64. Experiments with the testbench included measuring throughput (number of packets processed for a time period) and CPU utilization. Missing specifications were the processor model, number of computers/nodes, Docker version and network hardware. In the relevant solutions section of the study, IDSs were mentioned, but no IDS was included in the prototype. Kotenko et al. explained that any specific input data could be added to the prototype by creating a new Beat-collector using the libbeat library in Elastic Stack [57]. Krych et al. generated artificial Snort alerts and implemented a Snort-Elasticsearch- Logstash-Kibana Stack named SELK. It was based on Snort and Elastic Stack (formerly ELK Stack). A dashboard with visualizations of the intrusion detection, which the authors claimed to provide, was shown in the study. The dashboard file dek_kibana_viz_setup_Aug-16.json was not found anywhere on the internet or in the report, which could be interpreted as the dashboard not being provided by the authors. SELK, the product which was implemented in the study, was claimed to be available from several locations in the form of a virtual machine file (OVA file, supported by VirtualBox and VMware). However, there seems to be no way to get access to the SELK product without contacting the authors of the report, which decreases reproducibility. Performance metrics such as CPU utilization, throughput and network delay were not mentioned in the report [58].

24 3.7. Related work compared to this thesis

3.7 Related work compared to this thesis

From the related works in Section 3.1, it can be concluded that Snort’s detection rate is actively analyzed and improved not only by the Snort community but researchers as well. As mentioned in Section 1.4, this study will not include improving or analyzing rules for a better detection rate. Snort 3 was officially released on January 19, 2021, with the version number 3.1.0.0 [63]. It is therefore unlikely that Snort 3.1.0.0 has been thoroughly investigated because of how recently it was released. In the previous works included in Section 3.2, only earlier alpha or beta versions of Snort 3 have been used. The experiments in this study will be performed with cheap low/medium performance hardware, which increases the chance of a student or small company being able to repro- duce the experiments with similar or better results. For example, the maximum theoretical network speed is 1.0 Gbit/s. In the previous works, when similar hardware was used, experiments were done with Snort 2. These factors will make this thesis’ experiments different from previous experiments. Efforts will also be made to not leave out any specifications about hardware, software versions, configuration and the experiment process. This thesis work will not make any contribution in improving or scientifically analyzing Snort and other tools to inspect encrypted network traffic. However, related research about the subject is included Section 3.5 for discussion and completeness. In Section 3.6, it can be observed that Snort and Elastic Stack have been combined as a SIEM solution before, but the solutions were not easily accessible. In addition, Snort 3 was not used. This thesis work will contribute by sharing a SIEM solution capable of visualizing Snort 2 and Snort 3 alerts with only open-source software.

25 4 Method

This chapter includes a pre-study, experiment planning and experiment design. The experiment planning and design sections are based on the pre-study and chapters 1, 2 and 3.

4.1 Pre-study

This pre-study section brieﬂy describes the chosen method, experimentation in software engineering, and key points about methods used in related work. Following sections and chapters are constructed with the pre-study information as a base.

4.1.1 Experimentation in software engineering A controlled experiment in software engineering can be described as an empirical enquiry where one variable in the environment is changed. Other controlled variables should be kept constant, and the result consists of measurements on the outcome variables [90, p. 11]. When conducting a study using experimentation, the goal could be to show that one method or software is superior to another. After collecting experiment data, it needs to be analyzed and interpreted using statistical inference. This process includes the following tasks [90, pp. 76–80]: • Characterize the experiment data with descriptive statistics. • Reduce the dataset by removing false data points. • Test the hypotheses regarding the experiment, by evaluating data statistically. • Draw conclusions. Experiments are in general conducted when there is a need to control a scenario and change the outcomes in an exact and systematic way. An ofﬂine scenario is easier to ma- nipulate the outcomes in because the environment can be considered closed off. In an online scenario, all factors cannot always be manipulated because of outside inﬂuences. Human-oriented and technology-oriented are two categories of experiments. Technology- oriented experiments can be repeated without bias because the tools will not remember previous outcomes. Human subjects do not forget previous outcomes and will be biased in repeated experiments, even if factors and methods are changed [90, p. 16].

26 4.1. Pre-study

Experiments can be used to:

• Conﬁrm that existing theories are valid.

• Find out if there is a relationship between for example the amount of available RAM in the system and compile time for a computer program.

• Conﬁrm that measurements are correct in terms of what was expected to be measured.

A requirement for experiments is that more than one value is measured, but typically there are several variables that are sampled. In addition to ﬁnding variables, a context and objects that represent certain characteristics must be chosen. In the design phase of an experiment, it is important to choose objects that correspond to the methods that are interesting for research purposes. The goals of experiments can include [90, p. 17]:

• Prove in which scenarios hypotheses are true.

• Recommend a choice of methods and tools in certain situations or contexts.

Before the experiment process can begin, the researcher needs to realize that experimentation is a proper method to answer the current research question. Not all questions can be answered with experimentation. Once that has been realized, the researcher can continue with the following steps which the experiment process consists of [90, pp. 76–80]:

• Scoping: Deﬁne and write down problem description, objective and goal.

• Planning: Design the experiment, choose instruments and evaluate risks.

• Operation: Gather samples according to the experiment design. This will result in a dataset.

• Analysis and interpretation: Use descriptive statistics to interpret and visualize the dataset which was generated in the previous step. Next, remove false data points and redundant variables, if any. Finally, decide if the hypothesis can be rejected or accepted.

• Presentation and package: Document the results by generating an experiment report. The report can later be included in a thesis, research paper or company documentation, which can be published.

4.1.2 Deﬁnitions in experimentation Some deﬁnitions in the context of experimentation are:

• Independent variables are controlled or changed, and used as input in the experiment process. Studying the effect of manipulating independent variables is loosely the deﬁ- nition of experimentation. The independent variables that are changed during experimentation are called factors, and the other ones are controlled to a ﬁxed level.

• Dependent, outcome or response variables are measured to know the effect of the changes made in the input. There needs to be at least one dependent variable in an experiment.

• A speciﬁc value chosen for a factor is called a treatment.

• A subject is someone who applies a treatment.

• Treatments can be applied to subjects and objects.

• Independent variables that cannot be controlled can instead be deﬁned as objects in technology-oriented experiments.

27 4.1. Pre-study

Figure 4.1: A visualization of experimentation and its components [90, pp. 74–75].

• A test is a set of treatments, subjects and objects. Multiple tests are performed in a single experiment.

Figure 4.1 includes some of the previous deﬁnitions and can aid in understanding the experiment process. The cause and effect are part of a relationship or hypothesis that is to be conﬁrmed or refuted [90, pp. 73–75].

4.1.3 Methods in related work Zarpelão et al. [91] made a survey of intrusion detection where some of the following methods were discussed:

• A dataset can be used to evaluate how good an IDS is, for example the 1999 DARPA Intrusion Detection Evaluation Dataset. This dataset is well-studied but because of its old age considered to be outdated and missing the latest attacks.

• Empirical methods are often used when studying IDSs. Data is gathered through experiments with different settings and then evaluated.

• There are tools that can be used to simulate an intrusion attempt or a speciﬁc scenario that is relevant for a research question or subject. A potential problem when evaluating research about IDSs is that many authors have used different tools and network conﬁgurations for their simulations.

Verendel [87] examined 90 papers and argued that there is a lack of repeated large-sample empirical validation of the methods commonly used in the security ﬁeld. The author suggested that improvements can be made by repeating experiments, using shared datasets and comparing different methods. Hypothetical, empirical, simulation and theoretical are examples of the validation methods that were included in the paper. Modeling, quantiﬁed security and the current validation methods were not rejected, although limitations and risks were discussed. An assumption can be made, that existing methods for evaluating security research can be used and that these methods are widely accepted. This is due to the large number of existing research using the methods discussed by Zarpelão [91] and Verendel [87]. While

28 4.2. Experiment planning doing security research, efforts should be made to repeat experiments, share any valuable datasets and compare multiple methods if applicable.

4.2 Experiment planning

This section regards deﬁning variables and examining tools needed for performing experiments with Snort, datasets, rulesets, generated network trafﬁc and security information management. Detailed install instructions for Snort, PulledPork and security information management software are included in Section 4.2.1 to increase validity.

4.2.1 Snort, PulledPork and SIEM To prepare for experimentation, there was a need to install software and make conﬁgurations on the computers mentioned in Section 1.4. It should be noted that the suggested security information and event management solution uses software that supports Slack, Webhook and SMTP notiﬁcations/alerting based on events as well, but it was not set up or explored in this thesis. Snort 3.1.0.0 was installed on Ubuntu Linux by following the Snort 3 install guide [37]. The chapters Splunk, Using Splunk, Cleaning up your install and Reverse proxy for Splunk Web were skipped because Splunk is an alternative to the SIEM software chosen in this thesis. Snort 2.9.17.1 and PulledPork 0.8.0 were installed on Windows 10 using the description in Appendix A.3. PulledPork was installed on all computers with Ubuntu Linux by following the Snort 3 install guide [37]. For the SIEM solution in the thesis to remain open source and free for commercial use, the following software components were installed:

• On Ubuntu Linux:

– opendistroforelasticsearch-1.13.2-linux-x64 – opendistroforelasticsearch-kibana-1.13.2-linux-x64 – logstash-oss-7.12.0-x86_64 – metricbeat-oss-7.12.0-x86_64 – ﬁlebeat-oss-7.12.0-x86_64

• On Windows 10:

– metricbeat-oss-7.12.0-windows-x86_64 – ﬁlebeat-oss-7.12.0-windows-x86_64

Open Distro for Elasticsearch and Kibana were installed on Ubuntu Linux by using install instructions on the ofﬁcial website [4] and the information in Appendix A.4. The ODFE visualizations and dashboards used are shown in Appendix A.8.

4.2.2 Variable definitions Throughput is defined as the amount of successfully delivered data per second between two hosts in a computer network. A throughput value can be the average throughput during a time interval, or the instantaneous throughput at a specific timestamp [59, pp. 63, 71]. CPU utilization is how much of the elapsed time is utilized for processing program or OS instructions. The unit is a percentage value between 0 and 100. E.g., when a single-core CPU is continuously processing instructions from a single process, the CPU utilization is 100 percent [79, pp. 265, 288]. In the Windows 10 Performance Monitor, the total CPU utilization

29 4.2. Experiment planning is called % Processor Time. Windows measures how much of the time is used to execute the idle thread and subtracts that value from 100 percent. For clarification, an idle thread is executed by the processor when no other thread can be processed. Whether the CPU is idle or not is sampled every 10 ms, which can result in some inaccuracy. The CPU utilization for a process is the percentage used only for threads and instructions belonging to that process. To approximately measure how much RAM a process is utilizing, the process’ current working set size is commonly used. When any of the threads in a process touches a memory page, the page is brought into physical memory if needed. All recently touched pages will add up to the working set size in bytes [79, pp. 402–403, 427–429]. According to the description in Windows 10 Performance Monitor, a process’ working set can contain pages that are currently not in use if there is a certain amount of free RAM in the system. If the amount of free RAM is low, unused pages will be removed from the working set. Free RAM is the amount of primary memory available for the OS and processes to allocate immediately [79, p. 360]. Round-trip time (RTT) is a time measurement for a network packet to travel from client to server and back to the client again [59, p. 130]. End-to-end delay is a time measurement for a network packet to travel from source to destination [59, p. 69]. One-way delay (OWD) is another term to describe end-to-end delay in a network [3]. RTT will be used to measure network delay in the experiments when needed. When Snort has terminated successfully, it will output Packet I/O Totals, including the number of received, analyzed, dropped, filtered and outstanding packets. Outstanding packets are buffered in a queue to be processed by Snort and the rest of the mentioned Packet I/O totals are described as self-explanatory in the Snort FAQ [22]. According to Section 2.4, an IDS does not support dropping malicious packets to stop attacks (also known as packet filtering). Because Snort in IDS mode works on copied packets, a dropped packet will only result in it not being analyzed by Snort. The Snort Packet I/O Totals could show that it has dropped 681 packets when all those packets have ended up at their destination without being dropped. It is therefore important to differentiate Snort’s dropped (not analyzed) packets from the actual drop rate between two hosts in a network. The filtered packets statistic is expected to always show zero unless Snort is installed as an IPS. Asus-i5, HP-i7 and HP-ProDesk were chosen as variable names to refer to the computers specified in Section 1.4. The following differences between Asus-i5 and HP-i7 were identified:

• Processor release dates: Q2 2014 and Q1 2011.

• Processor brand modiﬁer and threads: i5 with 4 threads and i7 with 8 threads.

• Memory (RAM): 4 GB DDR3-1600 and 8 GB DDR3-1333.

• Storage (SSD): Samsung 850 EVO 250 GB and Samsung 870 EVO 500GB.

• OS: Windows 10 Education Version 1909 and Windows 10 Pro Version 1909.

HP-ProDesk was chosen to only be used for trafﬁc generation as a client in all testing scenarios. Asus-i5 and HP-i7 were chosen as objects for comparisons to show how much Snort packet threads could potentially beneﬁt different performance metrics on each system.

4.2.3 Datasets The 1999 DARPA Intrusion Detection Evaluation dataset includes two weeks of testing data with combined attacks and background traffic, Week 4 and Week 5. For this thesis, all tcpdump (pcap format) data files from Week 4 and Week 5 were selected, but with one exception. The outside.tcpdump.gz file from Friday on Week 5 was determined to be broken and was therefore

30 4.2. Experiment planning excluded. In total, the DARPA object chosen for experimentation consisted of 18 pcap ﬁles with size 7.65 GB [60]. Wireshark was used to generate the following protocol hierarchy statistics for the DARPA dataset, based on Monday Week 4:

• Ethernet 100.0% of packets.

– IPv6 0.0% of packets. – IPv4 97.0% of packets. * UDP 10.1% of packets, 0.4% of bytes. * TCP 84.8% of packets, 71.3 % of bytes. · HTTP 2.6% of packets, 31.8% of bytes. · SSL/TLS 0.0% of packets, 0.0% of bytes. · SMTP 2.7% of packets, 2.6% of bytes. * ICMP 2.1% of packets, 3.2% of bytes. – Logical-Link Control 1.6% of packets. – Conﬁguration Test Protocol (loopback) 0.5% of packets. – ARP 0.9% of packets.

In the protocol hierarchy statistics for the whole DARPA dataset, there were no SSL/TLS packets or bytes. More than 125 protocols were found and therefore many of them were omitted in the list above. The MR1 dataset consists of recorded network traffic from a server hosting a web-based application, recorded during a single day in February 2021. Figure 4.2 shows a simplified model of how such a dataset can be generated using Snort in Packet Logger Mode. In total, 12 pcap files with size 1.49 GB were identified in the dataset. All files were selected to be part of the MR1 object for experimentation.

Figure 4.2: A simpliﬁed model of a single server environment hosting a web-based application and using Snort as a packet logger.

Wireshark was used to generate the following protocol hierarchy statistics for the MR1 dataset:

• Ethernet 100.0% of 3111948 packets.

– IPv6 0.0% of packets.

31 4.2. Experiment planning

– IPv4 71.1% of packets. * UDP 0.5% of packets, 0.0% of bytes. * TCP 70.6% of packets, 91.0% of bytes. · TLS 34.4% of packets, 88.0% of bytes. · SMTP 0.0% of packets. · HTTP 0.0% of packets. * ICMP 0.0% of packets. – ARP 28.9% of packets, 2.7% of bytes

It was observed that all packets contain the Ethernet protocol, and either the IPv4 or ARP protocol. Some packets do not contain any higher-level protocols, and that is why TLS, SMTP and HTTP only added up to 34.4% of packets for the MR1 dataset [45]. Other interesting statistics that were generated with Wireshark are:

• Maximum packets per second: 4004 pps

• Maximum bits per second: 46 Mbit/s

• Maximum bytes per second: 5.75 MB/s

• Average bits per second: 668 kbit/s

• Average bytes per second: 83 kB/s

Wireshark was also used to generate packet length statistics for the MR1 dataset, which are shown in Table 4.1.

Packet Lengths Count Average Min Val Max Val Percent All packets 3111948 501.48 42 245334 100% 1-53 11 46.00 42 50 0.00% 54 269796 54.00 54 54 8.67% 55-59 174 58.00 58 58 0.01% 60 1482324 60.00 60 60 47.63% 61-65 2324 62.00 61 65 0.07% 66 134463 66.00 66 66 4.32% 67-77 161691 73.46 67 77 5.20% 78-159 450700 106.08 78 159 14.48% 160-319 123862 219.57 160 319 3.98% 320-639 179093 372.78 320 639 5.76% 640-1279 194321 1004.53 640 1279 6.24% 1280-2559 58674 1506.20 1280 2558 1.89% 2560-5119 15195 3568.00 2560 5109 0.49% 5120-9999 10254 6896.15 5120 9998 0.33% 10000-19999 15174 14907.82 10017 19999 0.49% 20000-99999 13445 44379.46 20004 99454 0.43% 100000+ 447 140955.75 100694 245334 0.01%

Table 4.1: Packet length statistics generated with Wireshark for the MR1 dataset.

According to the definitions in Section 4.1.2, the DARPA and MR1 datasets were defined as objects in the context of experimentation. In a Snort pcap readback scenario, Snort rulesets were identified as treatments that can be applied to the dataset objects. Suitable outcome variables could be the number of alerts and runtime reported by Snort.

32 4.2. Experiment planning

4.2.4 Snort rulesets The following ruleset and policy combinations were chosen as valid variables for experimentation:

• Snort 2 built-in rules

• Snort 2 Community Default (no policy selected)

• Snort 2 Community Connectivity

• Snort 2 Community Balanced

• Snort 2 Community Security

• Snort 2 Community Max-detect

• Snort 2 Registered Default (no policy selected)

• Snort 2 Registered Connectivity

• Snort 2 Registered Balanced

• Snort 2 Registered Security

• Snort 2 Registered Max-detect

• ET Open (Snort 2 only)

• Snort 3 built-in rules

• Snort 3 Community Default (no policy selected)

• Snort 3 Registered Default (no policy selected)

• Snort 3 Registered Connectivity

• Snort 3 Registered Balanced

• Snort 3 Registered Security

• Snort 3 Registered Max-detect

All rulesets were downloaded and stored on the 30th of April, 2021. On the 13th of May, 2021, PulledPork 0.8.0 was still the latest available version which does not support downloading and configuring Snort 3 Community rules, only Registered and Subscriber rules. Instead, the Snort 3 Community ruleset was downloaded manually from the Snort official website. The Snort 2 Community ruleset manual download was compared to the PulledPork download with no policy selected. This resulted in identical ruleset files. Because ET Open does not support Snort 3, it could not be chosen as a valid ruleset variable for Snort 3 tests [51]. The Snort 3 built-in rules were activated with the enable_builtin_rules configuration setting, which showed 538 rules in total. These rules did not have to be downloaded separately and were included in the Snort 3 installation. In the Snort 2 installation, the decoder.rules, preprocessor.rules and sensitive-data.rules files were included in the preproc_rules folder. Because they did not have to be downloaded manually, the variable name Snort 2 built-in rules was chosen to refer to them in this thesis. To activate them, the files had to be included individually in the snort.conf file. Snort does not support Shared Object rules on Windows [43]. Such rules are written in the C programming language instead of using the Snort rule syntax. This enables obfuscation of how the detection is performed and adding more complex conditions that would not be

33 4.2. Experiment planning

possible using regular Snort rules. Shared Object rules are also called SO rules or pre-compiled rules [25]. SO rules do not have to be closed source, but no SO rules were identified in the open-source rulesets (Snort Community and ET Open). Normally, all Snort Registered rulesets would include SO rules, but they were chosen to be excluded from all tests in this thesis. It was expected that the Snort 2 and Snort 3 Community rulesets would include the same rules. After downloading both rulesets on the same date, without making any changes to the rule files, all enabled rules were compared. Only 156 rules were enabled in both rule files, while Snort 2 Community had 342 unique rules and Snort 3 Community had 673 unique rules. Some enabled rules in the Snort 2 Community ruleset did not exist at all in the Snort 3 rule file, even when considering disabled rules. It was still decided that each default community ruleset would be used when comparing Snort 2 and Snort 3, without making any changes to the ruleset files.

4.2.5 Snort performance test rule and payload The string testattack987654321 was put in a payload ﬁle in the middle of 500 As, in order to increase Snort’s CPU utilization needed for content matching. Following is the Snort rule which was made to match the payload:

alert tcp any any −> any any §(msg:"Test attack content matching rule"; ¤ content:"|74 65 73 74 61 74 74 61 63 6b 39 38 37 36 35 34 33 32 31 |"; sid:1000002)

¦Note that the hexadecimal representation of the string was generated using an ASCII to hex ¥ converter.

4.2.6 D-ITG network traffic generation configurations To test network traffic with a high throughput in terms of packets per second (pps), the following D-ITG configuration was made:

• 16 TCP ﬂows with 500 bytes packet size, 4000 pps each and 15 000 ms duration.

• 1 TCP ﬂow, a single packet with the payload speciﬁed in Section 4.2.5 and 5000 ms delay.

• Expected trafﬁc 960 001 packets in total with a throughput of 0.256 Gbit/s.

The configuration was named D-ITG #1 for future references. To test network traffic with a high throughput and a mix of different packet sizes, the following D-ITG configuration was made:

• 2 TCP ﬂows with 480 bytes packet size, 10 240 pps each and 15 000 ms duration.

• 2 TCP ﬂows with 960 bytes packet size, 5120 pps each and 15 000 ms duration.

• 2 TCP ﬂows with 1920 bytes packet size, 2560 pps each and 15 000 ms duration.

• 2 TCP ﬂows with 3840 bytes packet size, 1280 pps each and 15 000 ms duration.

• 2 TCP ﬂows with 7680 bytes packet size, 640 pps each and 15 000 ms duration.

• 2 TCP ﬂows with 15360 bytes packet size, 320 pps each and 15 000 ms duration.

• 2 TCP ﬂows with 30720 bytes packet size, 160 pps each and 15 000 ms duration.

• 2 TCP ﬂows with 61440 bytes packet size, 80 pps each and 15 000 ms duration.

34 4.2. Experiment planning

Figure 4.3: A simpliﬁed model of the security information and event management integration used when testing.

• 1 TCP ﬂow, a single packet with the payload speciﬁed in Section 4.2.5 and 5000 ms delay.

• Expected network trafﬁc throughput 0.629 Gbit/s.

The configuration was named D-ITG #2 for future references. D-ITG’s uniform distribution feature for packet sizes was experimented with at first. It frequently caused the application to crash and was therefore exchanged with fix packet sizes instead. HP-i7 was observed to have better D-ITG performance, in terms of throughput, than Asus- i5 and was therefore chosen to be used in experiments with network traffic treatments. D- ITG’s maximum bitrate did not improve when pps values were increased in the D-ITG #2 configuration. It was decided that Snort could not be tested with higher D-ITG throughput than 0.629 Gbit/s on HP-i7.

4.2.7 SIEM integration and test environment The SIEM integration was set up as shown in Figure 4.3. Only Metricbeat OSS and Filebeat OSS lightweight data shippers were running on the system under test (SUT). Open Distro for Elasticsearch, Kibana and Logstash OSS, potentially resource heavy applications, were set up to run on a separate host during all tests. The performance of the host used for ofﬂoading the SUT was not considered or measured in any way. Variables for the SIEM integration were chosen to be named Metricbeat OSS service: On/Off and Filebeat OSS service: On/Off. Turning on the Metricbeat OSS service resulted in measuring system and Snort process metrics. The Filebeat OSS service was previously set up to ship Snort alert output, and turning the service on caused Snort alerts to be shipped in real-time. If a test generated alerts without any real- time data shipping, the service could later be turned on to visualize alerts without affecting the SUT. When Snort had generated thousands of alerts during a short pcap readback test, Filebeat OSS was observed to continue to ship log data after Snort had ﬁnished executing.

4.2.8 Information management Open Distro for Elasticsearch and Kibana were conﬁgured to sort Snort alerts for a better overview of test results. An example of how ODFE can present alerts is shown in Figure 4.4. Average and max values were also calculated automatically with ODFE, by only specifying test start and end times, as shown in Figure 4.5.

35 4.2. Experiment planning

Figure 4.4: Snort alerts visualized with Open Distro for Elasticsearch and Kibana.

Figure 4.5: An example CPU visualization of a Snort test run.

36 4.3. Experiment design

4.2.9 Snort 3 multiple packet threads and DAQ Available DAQs and DAQ settings for Snort 3 were identiﬁed by using the following command:

snort −−daq−dir /usr/local/lib/daq −−daq−list § ¤ ¦To look for the latest DAQs, the following GitHub repository link was used: ¥ https://github.com/snort3/libdaq/tree/master/modules § ¤ ¦It was determined that afpacket, with conﬁguration according to Section 2.5.6, was the only ¥ option for Snort 3 multithreading. DAQ PF_RING was determined not to be supported in Snort 3. It was however a valid choice for Snort 2.9 [65]. Due to the DAQ library not being included in the Snort Windows installer, it was determined not to be supported in Snort 2.9.17.1-WIN64.

4.3 Experiment design

This section includes both scoping and planning from the experimentation process. It describes problems, objectives, goals, hypotheses, environments and tests used to generate experiment data for Chapter 5. Snort example commands to use for the following tests can be found in Appendix A.7. To generate results, tests are repeated ﬁve times and the average value for each output variable is used when applicable.

4.3.1 Experiment: Snort pcap readback and packet threads 4.3.1.1 Problem description, objectives, goals and environment When tuning Snort, it is recommended to record network traffic, store it and use Snort’s pcap readback feature for offline analysis. The readback process, which is shown in Figure 4.6, is great for reproducibility and moving a potentially resource heavy intrusion detection process from a server to another computer. For simplicity, the Snort logging process and transferring the pcap files from the server to another computer have been left out of this experiment. Metricbeat OSS will be used to measure RAM and CPU utilization during all tests. The objectives of this experiment are:

• Find out how much beneﬁt Snort 3 multithreading provides for pcap readback. The beneﬁt is in terms of how fast the runtime is.

• List the runtime, CPU utilization and RAM utilization for two different systems to get an idea of how the number of Snort packet threads affects utilization of system resources.

This can help a company choose a suitable environment for running Snort, which is the goal of this experiment.

4.3.1.2 Tests The ﬁrst set of tests for this experiment will be called Snort 2 tests with Windows 10 and Com- munity ruleset and include the following:

• Objects

– Dataset: DARPA – Dataset: MR1

• Fixed level independent variables

37 4.3. Experiment design

Figure 4.6: The Snort pcap readback process.

– Snort version: 2.9.17.1-WIN64 – Ruleset: Snort 2 Community default (no policy) – Metricbeat OSS service: On – Filebeat OSS service: Off

• Treatments

– System: HP-i7 – System: Asus-i5

• Outcome variables

– Runtime until successful termination [mm:ss.fff] – Snort’s average CPU utilization – Snort’s max CPU utilization – Snort’s average RAM utilization – Snort’s max RAM utilization

The second set of tests for this experiment will be called Snort 3 tests with Ubuntu 20.04, Community ruleset and MR1/DARPA dataset and include the following:

• Objects

– Dataset: DARPA – Dataset: MR1 – System: HP-i7 – System: Asus-i5

• Fixed level independent variables

– Snort version: 3.1.0.0 – Ruleset: Snort 3 Community default (no policy) – Metricbeat OSS service: On

38 4.3. Experiment design

– Filebeat OSS service: Off

• Treatments

– Snort packet threads: 1, 2, 4, 8 or 16

• Outcome variables: Same as the ﬁrst test set

4.3.1.3 Hypotheses The experiment objective, in terms of cause and effect, is to ﬁnd out if the following relation- ships hold:

1. Increasing packet threads for Snort pcap readback will result in less runtime before Snort terminates successfully. The percental decrease in runtime will be signiﬁcantly larger in an octa-thread system compared to a quad-thread system.

2. Increasing Snort packet threads to more than the total amount of CPU threads in the system will not result in decreased runtime.

3. Snort 3 will perform better than Snort 2 in terms of runtime, without activating the max packet threads feature. (Snort 3 then uses a single packet thread.)

4. Increasing Snort packet threads will increase CPU and RAM utilization.

5. The maximum CPU utilization for a single-threaded application like Snort 2 will at most be 12.5% (1/8) for an octa-thread system and 25.0% (1/4) for a quad-thread system.

6. Increasing packet threads will increase RAM utilization. Because the packet threads share their memory, doubling the packet threads will result in less than twice the amount of RAM utilization.

4.3.2 Experiment: Snort pcap readback, standard rulesets and policies 4.3.2.1 Problem description, objectives, goals and environment Even when only considering standard rulesets, there are still a lot of options to choose from, as shown in Sections 4.2.4 and 2.6. The objectives of this experiment are:

• Find out how standard rulesets and policies affect the number of alerts and runtime.

• Share information about which rules triggered the alerts and where the potential malicious trafﬁc originated from (based on the geoip plugin for Logstash OSS). Note that Filebeat OSS, Logstash OSS or any SIEM integration should not be running during this experiment, so that the runtime of Snort is not affected. The alert information will not be evaluated scientiﬁcally but can still be valid for discussion.

The goal of the experiment is to give an example of how different rulesets and policies affect runtime and alerts for two different datasets. Results generated in this experiment can also help a researcher make choices about which ruleset to use for other experiments.

4.3.2.2 Tests The ﬁrst set of tests for this experiment will be called Snort 2 ruleset tests on HP-i7/Asus-i5 with Windows 10 and include the following:

• Objects

39 4.3. Experiment design

– Dataset: DARPA – Dataset: MR1 – System: HP-i7 – System: Asus-i5

• Fixed level independent variables

– Snort version: 2.9.17.1-WIN64 – Metricbeat OSS service: Off – Filebeat OSS service: Off

• Treatments

– All Snort 2 compatible rulesets from Section 4.2.4

• Outcome variables

– Runtime until successful termination [mm:ss.fff] – Number of alerts – Snort 2 default conﬁguration alert.csv ﬁle

The second set of tests for this experiment will be called Snort 3 ruleset tests on HP-i7 with Ubuntu 20.04 and include the following:

• Objects

– Dataset: DARPA – Dataset: MR1 – System: HP-i7

• Fixed level independent variables

– Snort version: 3.1.0.0 – Snort packet threads: 8 – Metricbeat OSS service: Off – Filebeat OSS service: Off

• Treatments

– All Snort 3 compatible rulesets from Section 4.2.4

• Outcome variables

– Runtime until successful termination [mm:ss.fff] – Number of alerts – Snort 3 default conﬁguration alert.json ﬁle

4.3.2.3 Hypotheses The experiment objective, in terms of cause and effect, is to ﬁnd out if the following relation- ships hold:

1. Choosing a stricter ruleset policy, e.g., Connectivity, Balanced, Security and Max-detect, will increase the number of alerts in that order.

2. Choosing a ruleset with more rules will result in a longer runtime.

40 4.3. Experiment design

4.3.3 Experiment: Snort IDS real-time performance 4.3.3.1 Problem description, objectives, goals and environment When deploying a Snort IDS, the goal should be to analyze 100% of the passing network trafﬁc. Otherwise, network attacks can be carried out without any alerts. For all packets to be analyzed, Snort’s I/O totals should show zero dropped packets, according to Section 4.2.2. A problem with deploying a Snort IDS is knowing beforehand which hardware and Snort setup to use for a certain throughput. Figure 4.7 shows a simpliﬁed model of the test environment for this experiment. In the tests that include a SIEM integration, Figure 4.3 shows the chosen setup. The environment and machines used can be described in the following way:

• HP-ProDesk is a client and trafﬁc generator to the left in Figure 4.7.

• HP-i7 is a server, trafﬁc receiver and Snort IDS sensor. It is both to the right in Figure 4.7, and to the left in Figure 4.3. In some tests, it is running Metricbeat OSS and Filebeat OSS services to ship data to another server.

• Asus-i5 is a server running SIEM services which are shown to the right in Figure 4.3. Its performance is not considered, and its purpose is to ofﬂoad the Snort sensor and trafﬁc receiver. All machines, including Asus-i5, are connected to a 1.0 Gbit/s local area network.

Figure 4.7: A testing environment for a single server hosting a web-based application and using Snort for intrusion detection.

The goal of the experiment is to ﬁnd out how network throughput, packet threads, rulesets and DAQs affect Snort’s performance. Results generated in this experiment can also give a researcher or company an idea of how much network trafﬁc Snort can handle with similar hardware and less than 1.0 Gbit/s throughput.

4.3.3.2 Tests The ﬁrst set of tests for this experiment will be called Snort 2 IDS real-time network trafﬁc tests on HP-i7 with Windows 10 and include the following:

• Objects

– System: HP-i7

• Fixed level independent variables

– Snort version: 2.9.17.1-WIN64 – Ruleset: Snort 2 Community default (no policy) – Metricbeat OSS service: Off

41 4.3. Experiment design

– Filebeat OSS service: Off – Network trafﬁc: D-ITG #1 or D-ITG #2 according to Section 4.2.6

• Treatments

– Snort running: Yes – Snort running: No

• Outcome variables

– Snort packet drop rate – Number of alerts – Packet drop rate – Packets received – Max delay (RTT) – Average delay (RTT) – Average bitrate

The second set of tests for this experiment will be called Snort 3 IDS packet threads and real-time network trafﬁc tests on HP-i7 with Ubuntu 20.04 and include the following:

• Objects

– System: HP-i7

• Fixed level independent variables

– Snort version: 3.1.0.0 – Snort ruleset: Snort 3 Community default (no policy) – Metricbeat OSS service: Off – Filebeat OSS service: Off – Network trafﬁc: D-ITG #2 according to Section 4.2.6 – DAQ: afpacket

• Treatments

– Snort running: No (0 packet threads) – Snort packet threads: 1, 4 or 8

• Outcome variables: Same as the ﬁrst test set

The third set of tests for this experiment will be called Snort 2 IDS rulesets and real-time network trafﬁc tests on HP-i7 with Windows 10 and include the following:

• Objects

– System: HP-i7

• Fixed level independent variables

– Snort version: 2.9.17.1-WIN64 – Metricbeat OSS service: Off – Filebeat OSS service: Off – Network trafﬁc: D-ITG #2 according to Section 4.2.6

42 4.3. Experiment design

• Treatments

– Snort ruleset: Snort 2 Community default – Snort ruleset: Snort 2 Community Max-detect – Snort ruleset: ET Open – Snort ruleset: ET Open and Snort 2 Community Max-detect

• Outcome variables: Same as the ﬁrst test set

The fourth set of tests for this experiment will be called Snort 2 IDS with SIEM integration real-time network trafﬁc tests on HP-i7 with Windows 10 and include the following:

• Objects

– System: HP-i7

• Fixed level independent variables

– Snort version: 2.9.17.1-WIN64 – Ruleset: Snort 2 Community default (no policy) – Network trafﬁc: D-ITG #2 according to Section 4.2.6

• Treatments

– Metricbeat OSS and Filebeat OSS services: Off – Metricbeat OSS and Filebeat OSS services: On

• Outcome variables

– All the outcome variables from the ﬁrst test set – System memory, CPU, network and process statistics from Metricbeat OSS

The ﬁfth set of tests for this experiment will be called Snort 3 pcap vs afpacket real-time network trafﬁc tests on HP-i7 with Ubuntu 20.04 and include the following:

• Objects

– System: HP-i7

• Fixed level independent variables

– Snort version: 3.1.0.0 – Snort ruleset: Snort 3 Community Default – Metricbeat OSS service: Off – Filebeat OSS service: Off – Network trafﬁc: D-ITG #2 according to Section 4.2.6 – Snort packet threads: 1

• Treatments

– DAQ: pcap (default) – DAQ: afpacket

• Outcome variables: Same as the ﬁrst test set

43 4.4. Experiment presentation

4.3.3.3 Hypotheses The experiment objective, in terms of cause and effect, is to ﬁnd out if the following relation- ships hold:

1. Running Snort IDS will not affect the network trafﬁc throughput or RTT.

2. Increasing the number of Snort packet threads will decrease Snort’s packet drop rate.

3. Using Metricbeat OSS and Filebeat OSS services to ship data to Open Distro for Elastic- search will not affect the network trafﬁc throughput or RTT.

4. Choosing default rulesets with more rules will increase Snort’s packet drop rate.

5. Using DAQ afpacket instead of DAQ pcap will decrease Snort 3’s packet drop rate.

6. Using Snort 3 with the standard DAQ pcap will result in a lower Snort packet drop rate than Snort 2.

4.4 Experiment presentation

Experiment outcomes and results are presented in Chapter 5, and all tests from Section 4.3 are summarized in tables that include variables and recorded values. When comparing test results from Section 4.3.1, the percental difference in runtime is calculated as follows:

T2 ´ T1 P = ¨ 100 T1 Hypotheses from sections 4.3.1.3, 4.3.2.3 and 4.3.3.3 are informally accepted or refuted in Section 6.1.

44 5 Results

This chapter contains the data gathered from running tests designed in Section 4.3 and observed facts that could be valid for discussion. It could be considered a display of the operation phase in the experimentation process. Redundant results are moved to Appendix A.2.

5.1 Snort pcap readback and packet threads

The following observations were made about the tests in tables 5.1, 5.2 and 5.3:

• Asus-i5 showed a better single-thread performance, in terms of runtime, when compared to HP-i7. With Snort 2, the runtime was 6% less for the MR1 dataset and 17% less for the DARPA dataset. With Snort 3, the runtime was 27% less for MR1 and 26% less for DARPA.

• HP-i7 showed a better multi-thread performance, in terms of runtime, when compared to Asus-i5. With four Snort packet threads, the runtime was 10% and 22% less for the MR1 and DARPA datasets, respectively. With eight Snort packet threads, the runtime was 23% and 36% less for MR1 and DARPA, respectively.

• HP-i7 had a better Snort packet threads performance gain, in terms of runtime, when compared to Asus-i5. Switching from one to four Snort packet threads on Asus-i5 re-

Snort 2 tests with Windows 10 and Community ruleset Objects Treat. Outcome variables Dataset System Time [mm:ss.fff] Avg. CPU Max CPU Avg. RAM Max RAM MR1 8-thread 00:20.915 5.262% 12.3% 87 MB 94 MB MR1 4-thread 00:19.669 8.105% 15.6% 83 MB 88 MB DARPA 8-thread 05:28.082 11.807% 12.6% 89 MB 128 MB DARPA 4-thread 04:33.363 23.344% 25.2% 90 MB 134 MB

Table 5.1: Snort 2 pcap readback and packet threads tests from Section 4.3.1.2. 8-thread and 4-thread refer to HP-i7 and Asus-i5, respectively.

45 5.1. Snort pcap readback and packet threads

Snort 3 tests with Ubuntu 20.04, Community ruleset and MR1 dataset Objects Treat. Outcome variables System Threads Time [mm:ss.fff] Avg. CPU Max CPU Avg. RAM Max RAM 8-thread 1 00:23.109 11.939% 12.7% 194 MB 216 MB 8-thread 4 00:08.748 37.850% 50.0% 343 MB 456 MB 8-thread 8 00:07.485 59.671% 97.9% 503 MB 672 MB 8-thread 16 00:07.329 66.200% 97.8% 578 MB 871 MB 4-thread 1 00:16.796 23.341% 25.3% 193 MB 212 MB 4-thread 2 00:11.069 42.282% 50.0% 264 MB 312 MB 4-thread 4 00:09.672 75.544% 95.2% 345 MB 461 MB 4-thread 8 00:09.708 76.944% 97.1% 495 MB 669 MB

Table 5.2: Snort 3 pcap readback, packet threads and MR1 test results, with tests deﬁned in Section 4.3.1.2.

Snort 3 tests with Ubuntu 20.04, Community ruleset and DARPA dataset Objects Treat. Outcome variables System Threads Time [mm:ss.fff] Avg. CPU Max CPU Avg. RAM Max RAM 8-thread 1 08:54.964 12.582% 12.8% 218 MB 282 MB 8-thread 4 02:55.048 45.265% 50.1% 420 MB 522 MB 8-thread 8 02:21.182 82.130% 97.4% 629 MB 763 MB 8-thread 16 02:29.256 78.501% 96.8% 938 MB 1037 MB 4-thread 1 06:36.986 24.984% 25.5% 217 MB 284 MB 4-thread 2 04:20.970 47.713% 50.5% 288 MB 360 MB 4-thread 4 03:45.704 88.559% 96.8% 413 MB 513 MB 4-thread 8 03:41.285 92.046% 96.8% 626 MB 765 MB

Table 5.3: Snort 3 pcap readback, packet threads and DARPA test results, with tests deﬁned in Section 4.3.1.2.

sulted in 42% and 43% less runtime for the MR1 and DARPA datasets, respectively. On HP-i7, switching from one to eight threads resulted in 68% and 74% less runtime for MR1 and DARPA, respectively.

• Using twice as many Snort packet threads as the system default number of threads, showed a 199-274 MB increase in maximum RAM utilization. The runtime was virtually the same but slower in some tests.

• Snort 3.1.0.0 (with the Snort 3 Community ruleset) did not show better single-thread performance in terms of runtime, than Snort 2.9.17.1-WIN64 (with the Snort 2 Commu- nity ruleset).

• Using the -z ﬂag without specifying any number caused Snort 3 to create eight packet threads for an octa-thread system and four threads for a quad-thread system.

• Snort 3, with y number of max packet threads speciﬁed and x number of pcap ﬁles available for analysis, created min(x, y) number of packet threads.

• Snort 2’s maximum CPU utilization was observed to be 12.6% on an octa-thread system and 25.2% on a quad-thread system.

• Snort 3’s maximum CPU utilization was observed to be at least 97.1% for both systems.

• Snort 3 used at most 763 MB RAM for an octa-thread system and 513 MB RAM for a quad-thread system, when utilizing eight and four Snort packet threads, respectively.

46 5.2. Snort pcap readback, standard rulesets and policies

• On an octa-thread system, Snort 3 with the default community ruleset had 64.21% and 56.97% less runtime than Snort2, for MR1 and DARPA, respectively. On a quad-thread system, the runtime was 50.83% and 17.43% less for the same datasets.

• Even if the number of Snort threads and pcap ﬁles matched, some threads would ﬁnish before others which lead to them doing no work for the latter part of the test. The pcap readback was less effective towards the end of each test in terms of CPU utilization when using multiple Snort packet threads.

• On an octa-thread system, Snort 2’s pcap readback speed was approximately 42.9 seconds/GB for DARPA and 14.0 seconds/GB for MR1.

• On a quad-thread system, Snort 2’s pcap readback speed was approximately 35.7 seconds/GB for DARPA and 13.2 seconds/GB for MR1.

• On an octa-thread system, Snort 3’s best pcap readback speed was approximately 18.5 seconds/GB for DARPA and 5.0 seconds/GB for MR1.

• On a quad-thread system, Snort 3’s best pcap readback speed was approximately 29.5 seconds/GB for DARPA and 6.5 seconds/GB for MR1.

5.2 Snort pcap readback, standard rulesets and policies

After all tests from tables A.1, 5.4 and 5.5 were ﬁnished, the alert.csv output from each test was examined using Open Distro for Elasticsearch, Filebeat OSS and Logstash OSS.

5.2.1 MR1: Snort 2 Community Max-detect alerts All 11 alerts from MR1 and the Snort 2 Community Max-detect ruleset showed that the trafﬁc originated from United States, China, India, Japan and Russia. There was a total of eight unique IP addresses, which were chosen to be referred to as USA #1-4, China #1, India #1, Japan #1 and Russia #1. The following observations were made for the alerts:

• Eight unique rules were triggered.

• PROTOCOL-RPC portmap listing UDP 111 was triggered by USA #1, USA #4 and Russia #1.

• INDICATOR-SCAN UPnP service discover attempt was triggered by USA #1 and USA #3.

• PROTOCOL-VOIP inbound INVITE message was triggered by USA #2.

• PROTOCOL-DNS named version attempt was triggered by USA #1.

• PROTOCOL-DNS Malformed DNS query with HTTP content was triggered by Japan #1.

• SERVER-WEBAPP MVPower DVR Shell arbitrary command execution attempt was triggered by India #1.

• China #1 triggered both SERVER-WEBAPP Netgear DGN1000 series routers authentication bypass attempt and SERVER-WEBAPP Netgear DGN1000 series routers arbitrary command execution attempt.

47 5.2. Snort pcap readback, standard rulesets and policies

Snort 2 ruleset tests on HP-i7 with Windows 10 Treatments Objects Outcome variables Ruleset # Rules Dataset Duration [mm:ss.fff] # Alerts Built-in 451 MR1 00:20.819 10155 Community Default 504 MR1 00:20.841 2 Community Connectivity 15 MR1 00:21.738 0 Community Balanced 381 MR1 00:22.214 2 Community Security 550 MR1 00:22.138 4 Community Max-detect 1353 MR1 00:23.387 11 Registered Default 10005 MR1 00:35.051 10158 Registered Connectivity 468 MR1 00:20.834 0 Registered Balanced 8524 MR1 00:32.447 2 Registered Security 15347 MR1 00:43.434 4 Registered Max-detect 33687 MR1 01:21.829 12270 ET Open 21404 MR1 01:09.444 93624 Built-in 451 DARPA 08:34.953 8201256 Community Default 504 DARPA 05:24.177 0 Community Connectivity 15 DARPA 04:03.055 0 Community Balanced 381 DARPA 05:24.367 0 Community Security 550 DARPA 05:27.717 10 Community Max-detect 1353 DARPA 06:06.334 46502 Registered Default 10005 DARPA 18:58.636 8206085 Registered Connectivity 468 DARPA 05:31.841 0 Registered Balanced 8524 DARPA 08:32.652 11 Registered Security 15347 DARPA 10:04.960 167 Registered Max-detect 33687 DARPA 23:10.453 1289290 ET Open 21404 DARPA 29:58.214 127381

Table 5.4: Snort 2 pcap readback and rulesets test results. Tests were planned in Section 4.3.2.2.

Snort 3 ruleset tests on HP-i7 with Ubuntu 20.04 Treatments Objects Outcome variables Ruleset # Rules Dataset Time [mm:ss.fff] # Alerts Built-in 538 MR1 00:05.950 59820 Community Default 829 MR1 00:07.248 0 Registered Default 53926 MR1 01:19.518 1245324 Registered Connectivity 474 MR1 00:07.390 0 Registered Balanced 8561 MR1 00:13.172 2 Registered Security 15400 MR1 00:20.662 3 Registered Max-detect 33493 MR1 00:46.100 2858 Built-in 538 DARPA 02:12.007 820348 Community Default 829 DARPA 02:16.084 10 Registered Default 53926 DARPA 09:41.806 31755211 Registered Connectivity 474 DARPA 02:11.446 0 Registered Balanced 8561 DARPA 02:33.424 11 Registered Security 15400 DARPA 02:43.277 118 Registered Max-detect 33493 DARPA 04:27.072 268405

Table 5.5: Snort 3 pcap readback and rulesets test results. Tests were planned in Section 4.3.2.2.

48 5.2. Snort pcap readback, standard rulesets and policies

5.2.2 MR1: ET Open alerts All 93 624 alerts from MR1 and the ET Open ruleset resulted in at least 221 new unique IP addresses. The following observations were made for the alerts:

• Only a single IP address was identiﬁed to be from trafﬁc originating from Sweden, and it was chosen to be referred to as Sweden #1.

• ET POLICY RDP connection conﬁrm was triggered 87 966 times and only by Sweden #1. No other alerts were identiﬁed from Sweden.

• Sweden #1 was identiﬁed to belong to the server provider.

• The other 5658 alerts showed IP addresses belonging to Austria, Bulgaria, Canada, China, France, Germany, Hong Kong, India, Iran, Ireland, Latvia, Netherlands, Pak- istan, Philippines, Poland, Moldova, Romania, Russia, Seychelles, Spain, Switzerland, Turkey, Ukraine, United Kingdom, United States and Vietnam.

• Another IP lookup service showed that some IP addresses from France and Germany were registered as being located in Russia.

• Alert messages included MS Remote Desktop Administrator Login Request, Behavioral Un- usually fast Terminal Server Traffic Potential Scan or Infection (Inbound), Dshield Block Listed, Spamhaus DROP Listed Traffic Inbound, Suspicious inbound to MSSQL port 1433, Sipvicious Scan, Active Threat Intelligence Poor Reputation, Microsoft Remote Desktop (RDP) Syn then Reset 30 Second DoS Attempt, Known Compromised or Hostile Host Traffic, Suspicious inbound to PostgreSQL port 5432, MVPower DVR Shell UCE, Netgear DGN Remote Command Execution, DNS Update From External net, Inbound RDP Connection with Minimal Security Protocol Requested and Suspicious Chmod Usage in URI (Inbound).

• Compared to the alerts from MR1 and the Community Max-detect result, ET Open had alerts from USA #2, China #1 and India #1 but not the other IP addresses.

5.2.3 MR1: Snort 2 built-in rules alerts All 10 115 alerts from MR1 and the Snort 2 built-in ruleset included the following messages: Consecutive TCP small segments exceeding threshold, (http_inspect) NON-RFC DEFINED CHAR, Reset outside window, (http_inspect) PROTOCOL-OTHER HTTP server response before client request, (http_inspect) LONG HEADER, TCP session without 3-way handshake, (http_inspect) NO CONTENT-LENGTH OR TRANSFER-ENCODING IN HTTP RESPONSE, (http_inspect) UN- ESCAPED SPACE IN HTTP URI and Bad segment, adjusted size <= 0. The geoip statistics showed that each alert message had been generated by at least one IP address from Sweden.

5.2.4 MR1: All rulesets The following could be stated when observing all ruleset alerts for the MR1 dataset:

• Snort 2 Community Security showed a subset of the alerts from Community Max- detect.

• The SERVER-OTHER Remote Desktop Protocol brute force attempt alerts from Snort 2 Reg- istered Max-detect showed many of the same IP addresses as ET POLICY MS Remote Desktop Administrator Login Request alerts from ET Open.

• Snort 2 Registered Max-detect showed alert messages (http_inspect) UNESCAPED SPACE IN HTTP URI, (http_inspect) NO CONTENT-LENGTH OR TRANSFER- ENCODING IN HTTP RESPONSE and Reset outside window, among others, which were triggered by Swedish IP addresses.

49 5.3. Snort IDS real-time performance

Snort 2 IDS real-time network trafﬁc tests on HP-i7 with Windows 10 Treatments Fixed Outcome variables Snort Trafﬁc Snort drop Packets Max RTT Avg. RTT Bitrate No D-ITG #1 - 959986 1.006 s 0.003 s 254 Mbit/s Yes D-ITG #1 0.000% 959982 1.003 s 0.001 s 254 Mbit/s No D-ITG #2 - 611979 1.004 s 0.002 s 626 Mbit/s Yes D-ITG #2 1.648% 611980 1.283 s 0.014 s 626 Mbit/s

Table 5.6: Snort 2 IDS test results from Section 4.3.3.

Snort 3 IDS packet threads and real-time network trafﬁc tests on HP-i7 with Ubuntu 20.04 Treatm. Fixed Outcome variables Threads Trafﬁc Snort drop Max RTT Avg. RTT Bitrate Alerts 0 D-ITG #2 - 1.004 s 0.002 s 626 Mbit/s - 1 D-ITG #2 8.966% 1.005 s 0.007 s 625 Mbit/s 4/5 4 D-ITG #2 4.936% 1.006 s 0.003 s 626 Mbit/s 5/5 8 D-ITG #2 1.091% 1.007 s 0.003 s 626 Mbit/s 5/5

Table 5.7: Snort 3 IDS with DAQ afpacket test results from Section 4.3.3.

5.3 Snort IDS real-time performance

All tests in this section were speciﬁed in Section 4.3.3.2. D-ITG did not drop a single packet in any test but was observed to have less throughput when not enough CPU utilization could be made. Thus, in this section the drop rate will refer to the number of packets dropped by Snort IDS. All packets that were sent ended up at the receiver. When the bitrate output variable showed a lower value for the same D-ITG conﬁguration, D-ITG sent less packets.

5.3.1 Generated network traffic with and without Snort Snort 2 was tested with two generated network traffic configurations according to Section 4.2.6. The test results were summarized in Table 5.6. For the first configuration, running Snort did not negatively affect RTT or bitrate in the network. With the second configuration, there was a small increase in RTT but no significant change in bitrate. It was also identified that Snort dropped 1.648% of all packets for a 626 Mbit/s bitrate but had zero dropped packets for a 254 Mbit/s bitrate. The D-ITG statistics showed 0% dropped packets for all generated network traffic, though the bitrate was slightly lower than what was expected from the configuration. D-ITG was expected to have 256 Mbit/s bitrate for the first configuration and 629 Mbit/s bitrate for the second. During the testing, D-ITG was observed to have 15- 20% CPU utilization for the D-ITG #2 configuration, both at the sender and receiver. Another observation was that D-ITG only calculated the bitrate from the sender to the receiver. The throughput was determined to be double the reported bitrate according to Windows 10 task manager and Metricbeat OSS.

5.3.2 Snort 3 packet threads Snort 3 was configured to use packet threads and load balancing for a single network interface (Ethernet) according to sections 2.5.6 and 4.2.9. For all tests in Table 5.7 DAQ afpacket was used, except for the first row which shows the output when Snort was not running at all. Both the actual number of alerts and the expected number of alerts, based on five test runs, are shown in the Alerts column. The following observations were made when testing Snort 3 packet threads with afpacket:

50 5.3. Snort IDS real-time performance

Snort 2 IDS rulesets and real-time network trafﬁc tests on HP-i7 with Windows 10 Treatments Outcome variables Rulesets Snort drop Max RTT Avg. RTT Bitrate Alerts Community Default 1.648% 1.283 s 0.014 s 626 Mbit/s 5/5 Community M-d 1.047% 1.103 s 0.022 s 626 Mbit/s 5/5 ET Open 32.133% 1.054 s 0.012 s 625 Mbit/s 4/5 ET Open & Community M-d 31.062% 1.262 s 0.013 s 626 Mbit/s 3/5

Table 5.8: Snort 2 IDS rulesets test results from Section 4.3.3. Max-detect has been shortened to M-d.

• RTT and throughput were not signiﬁcantly affected by Snort or the number of packet threads in any of the tests.

• Snort 3’s single-thread conﬁguration with DAQ afpacket and the Snort 3 Community Default ruleset resulted in an 8.966% drop rate. In comparison, Snort 2 with the Snort 2 Community Default ruleset had 1.648% dropped packets according to Table 5.6.

• When using a single packet thread, a malicious packet passed without triggering any alert.

• Changing from one to four packet threads decreased Snort’s drop rate from 8.966% to 4.936%.

• Changing from one to eight packet threads decreased Snort’s drop rate from 8.966% to 1.091%.

5.3.3 Snort 2 open-source rulesets Snort 2 IDS was tested with real-time network trafﬁc and four different open-source ruleset conﬁgurations. The following observations were made from the test results summarized in Table 5.8:

• All rulesets had the number of rules speciﬁed in Table 5.4. The last treatment, ET Open & Community Max-detect, had 22 757 rules in total.

• RTT and throughput were not affected signiﬁcantly by increasing the number of Snort rules from 504 to 22 757.

• Changing Snort ruleset from Community Default to Community Max-detect did not decrease Snort’s performance in terms of dropped packets. (This result is discussed in Section 6.1.7.)

• Activating the ET Open ruleset signiﬁcantly affected Snort’s drop rate. When D-ITG reported a bitrate of 625 Mbit/s, Snort dropped more than 32% of all packets.

• When Snort had a drop rate of 31.062%, only 3/5 malicious packets caused an alert.

• Activating both ET Open and Community Max-detect resulted in approximately a 1% difference in dropped packets, compared to using ET Open alone.

5.3.4 SIEM integration Snort 2 IDS was tested with real-time network trafﬁc and a SIEM integration using Metricbeat OSS and Filebeat OSS services on HP-i7 with Windows 10. During all tests, Open Distro for Elasticsearch and Logstash OSS services were running on Asus-i5 with Ubuntu 20.04. The

51 5.3. Snort IDS real-time performance

Snort 2 IDS and SIEM real-time network trafﬁc tests on HP-i7 with Windows 10 Treatments Outcome variables SIEM Snort drop Max RTT Avg. RTT Bitrate Alerts No 1.648% 1.283 s 0.014 s 626 Mbit/s 5/5 Yes 0.579% 1.291 s 0.015 s 626 Mbit/s 5/5

Table 5.9: Snort 2 IDS and SIEM integration test results from Section 4.3.3.

Snort 3 pcap vs afpacket real-time network trafﬁc tests on HP-i7 with Ubuntu 20.04 Treatm. Fixed Outcome variables DAQ Threads Snort drop Max RTT Avg. RTT Bitrate Alerts - 0 - 1.004 s 0.002 s 626 Mbit/s - pcap 1 25.766% 1.004 s 0.003 s 626 Mbit/s 3/5 afpacket 1 8.966% 1.005 s 0.007 s 625 Mbit/s 4/5

Table 5.10: Snort 3 IDS pcap vs afpacket test results from Section 4.3.3.

RTT, throughput and Snort drop rate were not affected negatively by using a SIEM integration which was set up to display system metrics and Snort alerts in real-time. Results for the SIEM integration tests were summarized in Table 5.9. The Alerts column shows both the actual number of alerts and the expected number of alerts based on ﬁve test runs. Turning on/off Metricbeat OSS and Filebeat OSS services was determined to be the SIEM integration treatment, which is represented by the ﬁrst column in Table 5.9.

5.3.5 Snort 3 DAQs: pcap vs afpacket Snort 3’s single thread performance was tested with DAQ pcap, afpacket and D-ITG #2 network trafﬁc. Test results were noted in Table 5.10 and the following observations were made:

• There was no signiﬁcant difference in RTT or bitrate from the D-ITG output, regardless of which DAQ was used. Based on the output variables, the network trafﬁc was not affected at all compared to when Snort was not running.

• Changing from the default DAQ pcap to afpacket decreased Snort’s drop rate from 25.766% to 8.966%.

• When Snort’s drop rate was 25.766%, only 3/5 malicious packets caused an alert.

The Alerts column in Table 5.10 shows both the actual number of alerts and the expected number of alerts based on ﬁve test runs.

52 6 Discussion

This chapter regards interesting ﬁndings from Chapter 5, criticizing the chosen methods and discussing the work in a wider context.

6.1 Results

In this section, interesting results from Chapter 5 are analyzed and discussed. The hypotheses from Section 4.3 are included as part of the discussion and analyzed informally based on test results.

6.1.1 Snort pcap readback and packet threads Based on Snort’s runtime for all Snort 2 and Snort 3 pcap readback tests, Asus-i5 was identified to have better single-thread performance than HP-i7. This was not entirely unexpected when considering the processor release dates Q2 2014 and Q1 2011 for Asus-i5 and HP-i7, respectively. Snort 3’s max packet threads feature allowed HP-i7 to outperform Asus-i5 in two interesting ways. First, with HP-i7 using 4/8 threads versus Asus-i5 using 4/4 threads. The runtime was significantly better with only 45% average and 50% maximum CPU utilization, compared to 89% and 97%. Second, by utilizing 8/8 threads, the runtime was improved further. This was observed in Table 5.3. Both Snort 2 and Snort 3 showed a significantly faster pcap readback speed (seconds/GB) for the MR1 dataset, compared to DARPA. This was most likely due to MR1 containing 88% encrypted bytes, based on Wireshark’s protocol hierarchy from Section 4.2.3. DARPA did not contain any SSL/TLS traffic. With the Snort 2 and Snort 3 default configurations, and according to the ssl preprocessor theory in Section 2.5.7, encrypted traffic was not inspected. It was not scientifically proven that the faster pcap readback speed was only due to the amount of encrypted traffic in each dataset, but it is a significant factor according to the theory. When choosing a suitable environment or system for running Snort pcap readback to analyze datasets and Snort rules, there are three important considerations to make. Firstly, if Snort 2 must be used for any reason, only the system’s single-thread performance should be considered. It is possible to start multiple Snort 2 processes to use more than one thread simultaneously, but multiple terminal outputs and load balancing pcap files between processes must be solved manually. Secondly, when choosing a system to use for Snort 3 pcap

53 6.1. Results readback, the system’s multithreading performance is highly relevant. Finally, Snort 3 does not use multithreading by default and does not load balance individual pcap files between threads. For best performance, make sure that the recorded network traffic is split between multiple files and use the -z flag to use the maximum number of threads reported by the system. All hypotheses from Section 4.3.1.3, except one, could be accepted based on test results. Snort 3 did not perform better than Snort 2, in terms of runtime, without activating max packet threads. This could be interpreted as Snort 2 having better single-thread performance than Snort 3 with the default configuration and default community ruleset. The default community ruleset in this case refers to downloading the applicable community ruleset for the selected Snort version and using it without making any changes to the rules. A better comparison between Snort 2 and Snort 3 could be achieved if both versions were installed on the same operating system and were using the same rules. Due to the limitations in Section 1.4, Snort 2 was installed on Windows 10 and no changes were made to the default rulesets.

6.1.2 Snort pcap readback and rulesets Based on test results from Section 5.2, Snort 2 showed almost no difference in runtime between Community Default, Balanced and Security rulesets. They included 504, 381 and 550 rules, respectively. Thus, choosing a stricter policy with more rules does not necessarily result in a longer runtime. It did however result in a larger number of alerts, both for Community and Registered rulesets. Another interesting observation was that Registered Max-detect with 33 687 rules had 22.68% less runtime than ET Open with 21 404 rules. It was concluded that the ﬁrst hypothesis could be accepted and the second one refuted, in Section 4.3.2.3.

6.1.3 Snort rulesets and detection of attacks Detection of attacks was not the focus of this thesis, but some analysis of Snort alerts was still required to validate that Snort was configured and functioned correctly. A simple configuration error that drastically affected detection results was to set the wrong value for the HOME_NET variable. When testing Snort in a local area network, 192.168.1.0/24 could be a correct value for the HOME_NET. If the same configuration was used for pcap files recorded from another network, which was the case with the MR1 dataset, many Snort rules would ignore all network traffic based on the destination address not being part of the HOME_NET. In most cases, using any as HOME_NET can force Snort to analyze all network traffic, regardless of destination. However, some ET Open rules included !HOME_NET which resulted in !any, an illegal configuration preventing Snort from starting up. (Negating any results in nothing and a rule with nothing as destination or source is a configuration error.) No official Snort rulesets from Cisco caused any issues when using any as HOME_NET. A limitation with the ET Open ruleset is that it has no Snort 3 support. It could possibly be converted manually with the Snort2lua tool, but some errors were encountered when this was tried. This means that not all Snort 2 rules can be converted successfully, which requires investigation by the Snort user. A manual ruleset download and update process allows the ruleset to become outdated, unless repeated daily. An automated process, like the one Pulled- Pork provides, is preferred to protect against the latest attacks and allow less time spent to maintain Snort. The Snort 3 Community ruleset could not be configured successfully with PulledPork 0.8.0, which was tried in Section 4.2.4. If a Snort user decided to use only open-source rules, this would be a big limitation with using Snort 3. On the other hand, Snort 3 Registered and Subscriber rulesets were found to be supported. Some interesting observations about rulesets and alerts were made in Section 5.2. Based on the number of alerts, 93 624, from MR1 and the ET Open ruleset, a lot of false alerts were expected. However, 87 966 alerts could have been avoided by adding a single IP address to

54 6.1. Results the Snort IP whitelist. Based on the geoip location not being Sweden, the rest of the alerts showed potential malicious trafﬁc but was not investigated any further. The built-in, Registered Max-detect and Registered Default rulesets were determined to include many false positives due to a lot of alert IP addresses located in Sweden. In the ET Open and Snort Community/Registered results, some alert message types and a set of IP addresses intersected. Combining both rulesets would therefore result in duplicated alerts. Some IP addresses from potential malicious trafﬁc were not found for both rulesets, which indicates that combining ET Open with Snort Community/Registered rules could improve the detection of attacks.

6.1.4 Snort IDS real-time performance According to sections 4.2.6 and 5.3, Snort 2 could analyze generated network traffic with approximately 16 concurrent TCP flows, 64 000 pps and 254 Mbit/s bitrate without dropping any packets. The second network traffic configuration caused Snort 2 to drop 1.648% of all packets and consisted of approximately 16 concurrent TCP flows, 40 800 pps and 629 Mbit/s bitrate. Because the second configuration, D-ITG #2, caused Snort 2 to drop packets, it was considered suitable for Snort 3 tests. Higher bitrates were not used for testing because D- ITG could not deliver a consistent result on the chosen system, even when Snort and other services were not running. Snort 3 with afpacket and a single thread dropped 8.966% of all packets, compared to 1.648% for Snort 2. Thus, Snort 2 was determined to have a better single-thread performance than Snort 3 with the default configuration for each version and the default community ruleset. A probable reason for this unexpected result can be found in Section 6.2.4. With eight packet threads on an octa-thread system, Snort 3 showed performance similar to Snort 2 with 1.091% dropped packets. It should be noted that some factors make a Snort 2 versus Snort 3 comparison in this thesis invalid, e.g., the difference in number of rules in default rulesets and different OSs used.

6.1.5 SIEM integration The tests from Section 5.3.4 measured network performance and Snort drop rate with Met- ricbeat OSS and Filebeat OSS services running. There was no negative change in performance, which confirmed that the services are lightweight both in theory (Section 2.7.4) and in practice. The SIEM integration did allow near real-time visualizations of Snort alerts and system metrics. With a small time interval, for example showing the last five minutes, new data was displayed every second. Using a large interval resulted in an automatic scaling to for example five second intervals. With a very large number of alerts from Snort during a short amount of time, which was tested in Section 4.2.7, the log data would be queued by Filebeat OSS and ship continuously but in a delayed manner. In that regard, the SIEM integration would not be considered real-time at all.

6.1.6 Snort 3 DAQs: pcap vs afpacket Based on the test results from sections 5.3.5 and 5.3.2, afpacket was observed to overwhelm- ingly outperform DAQ pcap. Due to DAQ pcap missing proper load balancing for Snort packet threads, it was not a valid choice for Snort 3 multithreading. This was mentioned in Section 4.2.9. When testing single-thread performance, afpacket had an 8.966% Snort drop rate compared to 25.766% for DAQ pcap. It should be noted that DAQ pcap was still the default DAQ for Snort 3 when this thesis was written. Based on performance it is recommended to switch to afpacket and not use the default DAQ when using Snort 3 to analyze real-time network trafﬁc.

55 6.2. Method

6.1.7 Snort IDS real-time performance hypotheses About the hypotheses in Section 4.3.3.3, the following was observed based on test results from Section 5.3:

1. Running Snort IDS did not affect throughput negatively in any test. The average RTT was increased with 12 ms according to Table 5.6, but it is unclear whether it was because of the treatment or a random error. To accept or refute the hypothesis, statistical analysis of variability in test data was determined to be needed.

2. Snort’s packet drop rate was decreased when changing from one to four and four to eight threads. The improvement was considered signiﬁcant enough to accept the hypothesis.

3. Metricbeat OSS and Filebeat OSS services did not affect throughput or RTT negatively. Thus, the hypothesis can be conﬁdently supported. However, a problem with the recorded test result is that the average Snort drop rate was approximately 1% less when services were running. This indicates that statistical analysis of variability is needed to scientiﬁcally accept the hypothesis.

4. Changing from Snort 2 Community Default to Community Max-detect did not have any signiﬁcant impact on Snort’s packet drop rate. Neither did adding Community Max- detect rules to the ET Open ruleset. However, ET Open rules alone increased Snort’s drop rate with approximately 30%. It was expected that ET Open with 21 404 rules would have a larger impact on performance than Community Max-detect with 1353 rules. Furthermore, because the drop rate decreased when changing policy from De- fault to Max-detect, statistical analysis was determined to be recommended to accept or refute the hypothesis.

5. Changing from DAQ pcap to afpacket showed a large decrease in Snort’s drop rate, which was enough to accept the hypothesis.

6. Snort 3 did not report a lower drop rate than Snort 2 in tests with similar number of rules. However, this thesis was determined to be unﬁt to make a comparison between Snort 2 and 3 according to Section 6.2.3. Section 6.2.4 must also be considered.

The validity of results and hypotheses is based on how well the experimentation method was followed. Problems with validity are discussed further in Section 6.2.1.

6.2 Method

This section will regard the validity of the study based on chosen methods.

6.2.1 Experimentation The selected method used to generate Snort and network performance results was based on experimentation in software engineering which was described brieﬂy in Section 4.1.1. All data generated by experiments in this thesis was not evaluated statistically. Results and hypotheses were only discussed in Section 6.1. To increase the validity of thesis results, Wilcoxon and ANOVA (ANalysis Of VAriance) tests could have been performed [90, p. 136].

6.2.2 Datasets As mentioned in Section 4.1.3, DARPA is considered to be outdated and missing the latest attacks [91]. It is however shared and available to everyone, which was found to be an important factor for research [87].

56 6.2. Method

DARPA was used to test Snort’s pcap readback performance in terms of runtime. Detec- tion of attacks in the dataset was not considered at all because of its old age. The output from DARPA tests were used in a comparison to discuss the MR1 results. MR1 is not a shared dataset, which is a negative aspect for research purposes and reproducibility. It could still be argued that the research results can be validated by others, both in terms of malicious and general traffic in MR1. To record malicious network traffic, a Snort sensor could be set up to log all network traffic for any public IP address. A general description of the MR1 recorded network traffic was included in Section 4.2.3. For example, protocol hierarchy, number of files, file size and packet sizes were shared. Since MR1 was recorded during February 2021, the dataset was not considered to be outdated. Using a larger dataset with traffic from more than one day could have increased the validity of the study.

6.2.3 Snort IDS and environment The Snort environment was set up with only three machines, a single traffic generator, a single traffic receiver and a SIEM server. Snort was running on the same system as the traffic receiver, which has been the case in previous studies in Section 3.2. It would have been interesting to include experiments with Snort installed as an inline IDS or IPS, without any traffic receiver running on the same system. That setup would have required more hardware, for example with two Ethernet NICs on a single machine and two switches. At the start of the study, the latest available Snort versions for Windows 10 and Ubuntu 20.04 were chosen. Thus, this study could be considered a comparison between Snort’s performance on Windows and Linux. Running Snort 2 on Ubuntu 20.04 would have resulted in a much better Snort 2 versus Snort 3 performance comparison. A problem with comparing Snort 2 and Snort 3 performance was the difference in default rulesets found in Section 4.2.4. Using the exact same Snort rule IDs (SIDs) for all performance tests could have increased the validity of the study. It was not investigated why the Snort Community default rulesets were different in terms of enabled SIDs. A better performance comparison between Snort 2 and Snort 3 could have been achieved by using the Snort2lua tool to convert rulesets. That process would have resulted in the study no longer using default rulesets configurable with PulledPork. Snort IDS could improve the security of any computer network environment. However, it should be noted that smart grid system [56] and industrial internet of things (IIoT) [14] environments have different solutions suggested by researchers compared to the solution presented in this thesis.

6.2.4 Snort, NIC and truncation According to the Snort 3 installation guide [37] it is recommended to disable large receive offload (LRO) and generic receive offload (GRO) in the NIC settings. Otherwise, longer packets can be truncated for Snort. These settings were not found in Windows 10 on Asus-i5 and HP-i7, which may have caused truncation of packets when testing Snort 2. That is a probable cause of why Snort 2 on Windows 10 showed a lower drop rate than Snort 3 on Ubuntu Linux for the same network traffic treatment. Furthermore, Snort will truncate packets to 1518 bytes [27, p. 13], unless the snaplen (snapshot length) flag is specified according to Ap- pendix A.7. The snaplen flag did not affect Snort 2’s drop rate for the D-ITG #2 treatment. On the other hand, Snort 3’s drop rate was significantly affected. With only two packet threads on an octa-thread system, Snort 3 reported zero dropped packets with default packet truncation. Instructing Snort 3 to use the maximum snaplen, 65 535 bytes, caused the drop rate to increase significantly for all treatments and zero dropped packets could not be achieved. This indicates that packet truncation and NIC settings need further investigation for Snort 2.9.17.1-WIN64 on HP-i7 and Asus-i5. It is also unclear which type of attacks can go unde- tected when the default packet truncation is used, and it was not investigated in this thesis.

57 6.2. Method

The test attack from Section 4.2.5 used a payload with approximately 500 bytes which is not large enough to be truncated with Snort’s default settings. Because the aim was to analyze 100% of the payload content, and the Snort 3 installation guide was followed, all network trafﬁc tests used Snort commands with the maximum snaplen according to Appendix A.7.

6.2.5 SIEM Security information and event management systems can be hosted with cloud services or installed with Docker or Kubernetes. Instead, the chosen SIEM software (from Section 4.2.1) was installed on a local machine and accessed only from a local network. In the case that Metricbeat OSS and Filebeat OSS are used, changing the host that they ship data to is not likely to affect system performance significantly. It was not considered or tested in the study. Management of alerts and data storage for the chosen SIEM solution was completely left out of the study. It was not explored what happens when large quantities of data are shipped without any management. Adding automated monitoring, triggers, alerts and index management to the theory chapter could have made the study more complete. How to configure visualizations and dashboards was also left out. Visualizations and dashboards used in the study was only shared as exported files. The study was more focused on how Snort and the system it runs on are affected by adding data shipping services, i.e., a SIEM integration.

6.2.6 Network traffic generation A limitation in the study was that D-ITG could only achieve a maximum consistent bitrate of 626 Mbit/s with the chosen hardware. It is a possibility that using other network traffic generation tools could have improved the bitrate which would have in turn allowed for more interesting tests and results. D-ITG has been able to achieve much higher throughput with other hardware, so it could be considered a hardware limitation [81]. As previously mentioned in Section 5.3.1, when D-ITG reported a 626 Mbit/s bitrate, the actual throughput was approximately 1252 Mbit/s when considering network traffic in both directions. No tool was found that could generate large quantities of TLS traffic. It would have been interesting if an experiment could show the impact that the amount of TLS traffic has on Snort’s performance.

6.2.7 Hardware The hardware listed in Section 1.4 can be considered old. An argument could be made that using newer hardware would make the study more valid. In the end, the hardware was considered adequate to answer the research questions in Section 1.3. A maximum of 20% system CPU utilization for the D-ITG #2 network trafﬁc, without running Snort, was observed on HP-i7. Thus, the CPU was not considered a bottleneck for increasing generated network trafﬁc throughput.

6.2.8 Source criticism White papers, reports and other documents from companies Cisco, Hewlett Packard Enter- prise, Huntsman Security, Identity Theft Resource Center, AlienVault and Enlyft were used as sources in Chapter 1. These sources might not be peer reviewed but were made and shared by serious companies. A book [20] published 2007 was used for general theory about Snort and intrusion detection. It can be considered outdated and many new Snort versions have been released since, but it was still the latest published Snort book when the thesis work was started. To com- plement outdated information, Snort 3 User Manual [73] and SNORT Users Manual 2.9.17 [27] were used.

58 6.3. Validity

It can be considered unreliable to include blog posts as sources in a thesis. However, it should be noted that the ofﬁcial Snort blog includes the latest information from Snort experts that might not be available in any book or manual yet. Many web links were used to refer to different software, GitHub repositories and online manuals. This was in order to back up information in the thesis with ofﬁcial sources, even though they might not be peer reviewed. If the web links could be replaced with peer reviewed sources, that would have been preferred. The goal in Chapter 3 was to include relevant research from peer reviewed sources. Be- cause the chapter was split in many different categories, adding more research papers per category could have increased the validity of the study. A few of the referenced papers had not been published in a respected and well-known journal but were still considered relevant.

6.3 Validity

Some validity threats suggested by Wohlin [90] are discussed in this section. If researchers are trying to achieve a specific experiment conclusion, they may influence the results. This is called fishing and it is a threat to the conclusion validity [90, p. 104]. In Table 5.9, it would have been desirable if the Snort drop rate was identical for both treatments. Furthermore, in Table 5.8, a better result would have been if more rules always resulted in a larger Snort drop rate. Instead of fishing for results by repeating tests, a suspected variability problem was discussed in Section 6.1.7. Reliability of measurements is another threat to validity. In this thesis, the following measurements relied on respective software:

• Metricbeat OSS: Snort and system CPU utilization.

• Metricbeat OSS: Snort and system RAM utilization.

• Open Distro for Elasticsearch (ODFE): Max/average CPU/RAM calculations and sorted alerts.

• Windows PowerShell and Linux terminal: Test start, end and runtime measurements for setting intervals in ODFE.

• D-ITG: Average bitrate, max/average RTT, packets sent/received and drop rate.

• Snort: Alerts and Snort dropped packets.

Because software, without human judgement, was used to collect measurements, reliability of measurements was not considered to be a signiﬁcant threat to the validity in this thesis. The validity threat of low statistical power was determined to be high because no statistical analysis was performed in this thesis. Informally, the Snort drop rate was observed to vary at most 1.5% for the same treatment. For pcap readback (ofﬂine tests), the number of alerts had zero variation, but duration varied a few seconds.

6.4 The work in a wider context

6.4.1 Snort IDS and encrypted traffic In Section 4.2.3 it was discovered that the MR1 dataset contained approximately 88% encrypted bytes. Thus, several solutions for analyzing encrypted network traffic were examined in Section 3.5. Most of the methods and software that seemed ready to implement and performance test were identified to be MitM solutions. A positive side of the MitM method was the ability to combine Snort with traffic decryption software to detect attacks in encrypted traffic. On the other hand, proper end-to-end encryption would be broken between all users and

59 6.4. The work in a wider context the web-based application. Storing decrypted sensitive data in pcap files, even temporarily, could be considered highly unethical. Especially if end-to-end encryption was promised. Not inspecting encrypted traffic with an IDS could result in a lot of attempted attacks passing by completely unnoticed. Hosting a web-based application and leaving it unsecured is also an ethical problem to consider. A solution to this is applying other measures to secure the application, such as writing secure code and penetration testing. Snort could still be considered useful even if 100% of the traffic was encrypted. Snort rules and the ssl preprocessor, described in sections 2.5.4 and 2.5.7, could make sure that encrypted traffic is only allowed from trusted servers and with a specific SSL/TLS version. To summarize, encrypted traffic is an obstacle for Snort users and researchers but in a wider context necessary to ensure security and integrity for all internet users.

6.4.2 Confidentiality Confidentiality of sensitive data must be maintained by researchers even if it goes against research purposes [90, p. 33]. To share all valuable datasets is suggested by Verendel [87] to improve security research, which is a research purpose. Arbitrarily recording internet traffic and sharing it could result in invasion of privacy if the data happened to contain any personal information. Snort can obfuscate IP addresses when in logger mode, but that would not ensure the integrity of all network users.

6.4.3 Power consumption Deploying Snort sensors with a lot of rules will result in increased CPU utilization for analyzing packets, and thus the power consumption will also increase. Power consumption was completely left out of the thesis, but it is something that is worth considering. It could be for economic reasons, having a portable Snort sensor running on a battery or large power consumption having a negative effect on society in general. It is not recommended to use a low power machine for Snort [43] but it has been researched for deploying portable IDS sensors [67]. In order to decrease power consumption, a Snort user can disable rules that do not aid in detecting attacks for a specific network or environment. Deploying a lot of Snort sensors in different locations could aid in reducing malicious network traffic in the whole world and thus decrease power consumption because of less throughput. This is only possible if Snort is deployed in inline mode and configured to drop packets, or if Snort IDS is used in combination with efforts to block malicious traffic with firewalls or other tools.

60 7 Conclusion

This thesis has contributed open-source Snort IDS solutions with security information and event management features. All setups and configurations that were tested showed no significant impact on connectivity. Install instructions were shared in Section 4.2.1 and performance test results were shared in Chapter 5. An Open Distro for Elasticsearch integration for Snort was suggested to be installed according to sections 4.2.7 and 4.3.3, and it showed no noticeable impact on connectivity or Snort’s drop rate. The following findings could be considered useful for researchers, Snort users and companies. Firstly, to utilize all suggested open-source rulesets with PulledPork automatic updates, Snort 2 must be used. However, the PulledPork and ET Open projects should be monitored for future updates that include full Snort 3 support. Secondly, on multi-thread systems, Snort 3 was shown to significantly outperform Snort 2 in pcap readback and Snort 2 had very low CPU utilization. Therefore, future research should focus on Snort 3. Suggestions for Snort 3 include:

• Always use the -z ﬂag, without any speciﬁed number, to enable all available system threads for pcap readback.

• For best real-time network trafﬁc performance, do not use the default DAQ pcap. Set up DAQ afpacket and packet threads according to sections 2.5.6 and 4.2.9.

Finally, according to Section 4.2.9, changing DAQ for Snort 2 on Windows is not supported. If a company has decided to use Snort 2 on Linux, all available DAQs should be considered for best performance. However, DAQs for Snort 2 on Linux were not included in this thesis due to Snort version research delimitations in Section 1.4. To maximize the thesis contribution, research questions were chosen to focus on Snort’s multithreading feature, Snort’s packet drop rate, throughput and network delay. To strictly answer the questions from Section 1.3, the following could be stated. Test results showed that throughput and network delay were not affected significantly by Snort IDS, regardless of configuration. Thus, Snort multithreading does not improve throughput or network delay. Snort’s packet drop rate decreased from 8.966% to 1.091% when packet threads were increased from one to eight, with an average throughput of approximately 1252 Mbit/s and with an octa-thread system. Furthermore, on an octa-thread system, the Snort offline analysis runtime was 68% and 74% less for two different datasets. On a quad-thread system, the

61 runtime was 42% and 43% less for the same datasets. Thus, Snort can benefit significantly from multithreading depending on how many threads are available in the system. Lastly, if Snort IDS and Open Distro for Elasticsearch are installed according to sections 4.2.7 and 4.3.3, throughput, network delay and Snort’s packet drop rate are not affected negatively. This thesis tested Snort’s performance with default rulesets and settings. No attempt was made to optimize Snort preprocessor settings or rulesets in order to increase performance, without impairing detection of attacks. For future research, it would be interesting to find out how much of a performance gain could be made by disabling individual rules in the default rulesets. Specifically rules that would not aid detection for a certain network or server. Computational intelligence (CI) and machine learning (ML) research has shown great detection accuracy with a low rate of false positive alerts [56] but was outside the scope of this thesis. Thus, implementing and performance testing intrusion detection based on CI and ML should also be considered for future work. It was determined that PulledPork 0.8.0 does not support managing Snort 3 Community and ET Open rulesets. To deploy Snort 3 with only open-source rules, automatic ruleset updates are recommended and most likely need to be solved to ensure maximum protection. The selected hardware in this thesis allowed a maximum of 626 Mbit/s average bitrate (from sender to receiver) with approximately 1252 Mbit/s average throughput in total. On an octa-thread system, Snort 3 with eight packet threads and the Snort 3 Community default ruleset could not analyze all packets. Additionally, it did not show any significant performance advantage, in terms of Snort drop rate, compared to Snort 2 on Windows. For future research, it would be interesting to test Snort 2 and Snort 3 on the same OS to make a proper performance comparison between the versions. Snort 3 configurations could also be explored to optimize performance. This thesis tested Snort’s performance when data shipping services were running on the same system, to enable a near real-time integration with SIEM software. No performance decrease in terms of network delay, throughput or Snort drop rate could be observed. To help a company make decisions about installing Snort together with ODFE to enable SIEM, the usability of ODFE should be researched. Alerts from Snort were discussed informally in Section 6.1.3, i.e., they were not investigated using a scientific method. A scientific evaluation of Snort alerts from a present time dataset and actions to prevent alerts need further investigation.

62 A Appendix

A.1 Terminology

• AF_PACKET: AF_PACKET is a network socket included in Linux [50]. It can be utilized with Snort’s afpacket DAQ module.

• ALv2: Apache license, version 2.0. An open-source license.

• Attack surface: In the context of a software environment, the attack surface is the sum of all attack vectors.

• Attack vector: An attack vector is a way or a path that an attacker can use to attack a software application. An attack includes gaining unauthorized access to, delivering malware or any action with a malicious outcome.

• D-ITG: Distributed Internet Trafﬁc Generator (open-source software).

• DAQ: Data AcQusition. In the context of this thesis, it is the method or software for network packet I/O.

• DDoS: Distributed denial-of-service.

• DoS: Denial-of-service.

• DIDS: Distributed intrusion detection system.

• Elastic: Elastic is the company behind Elastic Stack [5].

• Elastic Stack: Elastic Stack includes the software components Elasticsearch, Kibana, Logstash and Beats. It can be used to create SIEM solutions and is formerly known as the ELK Stack.

• ELv2: Elastic License v2. It is not an open-source license, but it enables free use in some cases [12].

• geoip: A Logstash plugin that adds geographical coordinates and information based on a given IP address and if it exists in a locally stored copy of the Maxmind GeoLite2 database.

63 A.1. Terminology

• HIDS: Host-based intrusion detection system.

• ICMP attack: Also known as Internet Control Message Protocol ﬂood DDoS attack or Ping ﬂood attack. ICMP echo-requests are normally used as pings to test the connection of a host.

• IDPS: Intrusion detection and prevention system.

• IDS: Intrusion detection system.

• IPS: Intrusion prevention system.

• NIC: Network interface card.

• NIDS: Network-based intrusion detection system.

• ODFE: Open Distro for Elasticsearch. Software that enables security information and event management (SIEM).

• Ofﬂine analysis: Refers to the Snort pcap readback feature described in sections 2.5.3 and 4.3.1.1.

• Online analysis: Refers to real-time network trafﬁc analysis with Snort NIDS mode, which is described in sections 2.5.2 and 2.5.5.

• OSS: Open-source software.

• OWD: End-to-end delay is a time measurement for a network packet to travel from source to destination [59, p. 69]. One-way delay (OWD) is another term to describe end-to-end delay in a network [3].

• PF_RING: A Linux kernel module and framework for high-speed packet processing [64]. PF_RING can be used as a DAQ module for Snort [66].

• POP3/IMAP: Post Ofﬁce Protocol and Internet Message Access Protocol are application layer protocols used to fetch emails from a server.

• pps: Packets per second sent or received in a network. It can be considered a measurement of throughput.

• pytbull: Pytbull is a free, open-source and Python based IDPS testing framework. It uses the GPLv3 license [35].

• RTT: Round-trip time (RTT) is a time measurement for a network packet to travel from client to server and back to the client again [59, p. 130].

• Scan attack: Also known as a scanning attack or a port scan attack.

• Security Onion: A cybersecurity Linux distribution.

• SIM: Security information management.

• SIEM: Security information and event management.

• SNMP: Simple Network Management Protocol.

• Snort++: Snort 3 is also known as Snort++.

64 A.2. Excluded results

Snort 2 ruleset tests on Asus-i5 with Windows 10 Treatments Objects Outcome variables Ruleset # Rules Dataset Duration [mm:ss.fff] # Alerts Built-in 451 MR1 00:17.117 10155 Community Default 504 MR1 00:18.525 2 Community Connectivity 15 MR1 00:17.638 0 Community Balanced 381 MR1 00:19.301 2 Community Security 550 MR1 00:18.143 4 Community Max-detect 1353 MR1 00:18.850 11 Registered Default 10005 MR1 00:30.847 10158 Registered Connectivity 468 MR1 00:18.039 0 Registered Balanced 8524 MR1 00:28.660 2 Registered Security 15347 MR1 00:40.109 4 Registered Max-detect 33687 MR1 01:19.173 12270 ET Open 21404 MR1 01:27.685 93624 Built-in 451 DARPA 08:28.215 8201260 Community Default 504 DARPA 04:29.886 0 Community Connectivity 15 DARPA 03:13.980 0 Community Balanced 381 DARPA 04:28.130 0 Community Security 550 DARPA 04:32.020 10 Community Max-detect 1353 DARPA 05:05.405 46502 Registered Default 10005 DARPA 16:26.480 8206085 Registered Connectivity 468 DARPA 04:37.763 0 Registered Balanced 8524 DARPA 07:45.627 11 Registered Security 15347 DARPA 09:18.751 167 Registered Max-detect 33687 DARPA 23:02.055 1289265 ET Open 21404 DARPA 36:30.810 127381

Table A.1: Snort 2 pcap readback and rulesets test results for Asus-i5. Tests were planned in Section 4.3.2.2.

A.2 Excluded results

This appendix shows results, Table A.1, that were excluded from Chapter 5.

65 A.3. Snort 2.9.17.1 on Windows 10

A.3 Snort 2.9.17.1 on Windows 10

The following instructions can be used to install Snort 2.9.17.1 on Windows 10 without any rulesets:

• Download and run the latest Npcap [61] installer, e.g. npcap-1.30.exe. Follow the setup instructions.

• Download and run Snort_2.9.17.1_Installer.x64.exe from snort.org. Follow the setup instructions and install all components.

• Open the Command Prompt (cmd) in Windows.

• Navigate to the Snort bin folder and run Snort with the following commands:

cd C:\Snort\bin §snort −V ¤

• ¦The Snort output should include Version 2.9.17.1-WIN64 GRE (Build 1013) if installed ¥ correctly.

• Make sure the following paths and ﬁles exist, create them if needed:

C:\Snort\rules\pulledpork.rules §C:\Snort\rules\local.rules ¤ C:\Snort\iplists\your.allowlist C:\Snort\iplists\default.blocklist

• ¦The following rule can be added to local.rules for testing: ¥

alert icmp any any −> any any ( msg:"ICMP test rule triggered"; sid:10000001; ) § ¤

• ¦Edit the system environment variables in Windows. Add C:\Snort\bin to the Path vari- ¥ able, which enables the snort command to be used from any directory.

• Test the rule with the following commands:

snort −W §cd C:\Snort ¤ snort −c rules\local.rules −i

• ¦The ﬁrst command will show all available interfaces and their indexes. Use the correct ¥ index, an integer, for the third command. Ping any device to trigger an alert.

• Alerts will be logged in the following default path:

C:\Snort\log\alert.ids § ¤

Snort¦ default Cisco Community Ruleset conﬁguration instructions for Windows 10: ¥

• Download the Snort 2.9 Community Ruleset:

https://snort.org/downloads/community/community−rules.tar.gz § ¤

• ¦Extract the community-rules.tar.gz ﬁle. ¥

• Copy or move the snort.conf and sid-msg.map ﬁles to the following locations:

C:\Snort\etc\snort.conf §C:\Snort\etc\sid−msg.map ¤

¦ 66 ¥ A.3. Snort 2.9.17.1 on Windows 10

• Copy or move the community.rules ﬁle to the following folder:

C:\Snort\rules § ¤ ¦ ¥ • Note that the community.rules ﬁle is not needed later when PulledPork is installed and conﬁgured.

• Change etc\snort.conf using the following example conﬁguration and the comments in the ﬁle:

Line 45: §ipvar HOME_NET any ¤

Line 48: ipvar EXTERNAL_NET any

Lines 104−106: var RULE_PATH c:\snort\rules #var SO_RULE_PATH ../so_rules var PREPROC_RULE_PATH c:\snort\preproc_rules

Lines 109−110: var WHITE_LIST_PATH c:\snort\iplists var BLACK_LIST_PATH c:\snort\iplists

Line 182: conﬁg logdir: c:\snort\log

Line 243: dynamicpreprocessor directory C:\Snort\lib\snort_dynamicpreprocessor

Line 246: dynamicengine C:\Snort\lib\snort_dynamicengine\sf_engine.dll

Line 249: #dynamicdetection directory /usr/local/lib/snort_dynamicrules

Lines 262−266: #preprocessor normalize_ip4 #preprocessor normalize_tcp: block, rsv, pad, urp, req_urg, req_pay, req_urp, ips, ecn stream #preprocessor normalize_icmp4 #preprocessor normalize_ip6 #preprocessor normalize_icmp6

Line 322: decompress_swf { deﬂate } \

Lines 501−506: preprocessor reputation: \ memcap 500, \ priority whitelist, \ nested_ip inner, \ whitelist $WHITE_LIST_PATH/your.allowlist, \ blacklist $BLACK_LIST_PATH/default.blocklist

Line 530, add: output alert_csv: alert.csv default

Line 539, add: include $RULE_PATH/community.rules

Lines 541−656: Comment out all rule includes, except local and community, if there are any.

Lines 664−667 (this enables built−in rules):

67 A.3. Snort 2.9.17.1 on Windows 10

include $PREPROC_RULE_PATH/preprocessor.rules include $PREPROC_RULE_PATH/decoder.rules include $PREPROC_RULE_PATH/sensitive−data.rules

• ¦Note that Snort does not support Shared Object rules (so_rules) on Windows [43]. ¥

• Test the conﬁguration with:

snort −i 1 −c C:\Snort\etc\snort.conf −T § ¤

• ¦Verify that the output from Snort includes the following: ¥

950 Snort rules read § 503 detection rules ¤ 154 decoder rules 293 preprocessor rules [...] Snort successfully validated the conﬁguration! Snort exiting

• ¦Use the following commands to manage Snort as a service in Windows: ¥

cd C:\Snort\bin §snort.exe /SERVICE /INSTALL −i 1 −c C:\Snort\etc\snort.conf ¤ snort.exe /SERVICE /SHOW snort.exe /SERVICE /UNINSTALL services.msc

• ¦Note that when testing Snort default conﬁguration from the Community Ruleset, Snort ¥ warns about 37 ﬂowbit dependencies. For example:

WARNING: flowbits key ’file.xml’ is set but not ever checked. § ¤ ¦Cisco suggests using PulledPork to solve the flowbit dependency warnings [25], but ¥ Snort can run successfully with the warnings.

The following instructions can be used to install PulledPork on Windows 10 (for managing Snort rules):

• Download and run the latest recommended Strawberry Perl [84] installer, e.g. strawberry-perl-5.32.1.1-64bit.msi. Follow the setup instructions. Note that Perl is required to run PulledPork.

• "This is perl 5..." should be included in the output when testing Perl with the following command:

perl −v § ¤

• ¦Download and extract the latest PulledPork [34] version from GitHub, e.g. PulledPork ¥ v0.8.0 from the following link:

https://github.com/shirkdog/pulledpork/archive/master.tar.gz § ¤

•¦PulledPork v0.8.0 should be included in the output when testing PulledPork with the ¥ following command in the extracted folder:

pulledpork.pl −V § ¤

• ¦If Windows asks, select Perl Interpreter as a program to open pulledpork.pl with. ¥

68 A.3. Snort 2.9.17.1 on Windows 10

• Change etc\pulledpork.conf using the following example conﬁguration and the comments in the ﬁle:

Line 19: §#rule_url=https://www.snort.org/reg−rules/|snortrules−snapshot.tar.gz| ¤

Line 21: rule_url=https://snort.org/downloads/community/|community−rules.tar.gz|Community

Line 24: rule_url=https://snort.org/downloads/ip−block−list|IPBLOCKLIST|open

Line 27: rule_url=https://rules.emergingthreats.net/|emerging.rules.tar.gz|open−nogpl

Line 59: temp_path=C:\temp

Line 72: rule_path=C:\Snort\rules\pulledpork.rules

Line 87: local_rules=C:\Snort\rules\local.rules

Line 90: sid_msg=C:\Snort\etc\sid−msg.map

Line 94: sid_msg_version=2

Line 98: sid_changelog=C:\Snort\log\sid_changes.log

Line 110: #sorule_path=/usr/local/lib/snort_dynamicrules/

Line 113: snort_path=C:\Snort\bin\snort.exe

Line 117: conﬁg_path=C:\Snort\etc\snort.conf

Line 134: #distro=FreeBSD−12

Line 142: block_list=C:\Snort\iplists\default.blocklist

Line 151: IPRVersion=C:\Snort\iplists

Line 154: #snort_control=/usr/local/bin/snort_control

Line 209 (Only works with Snort/Cisco rules, comment out if using ET Open rules. Accepted values are connectivity/balanced/security/max−detect.): ips_policy=balanced

¦ ¥ • Note that Snort does not support reload for Windows, and thus will not be able to handle the signal sent from PulledPork when rules have been updated [23].

• Create the ﬁle: C:\Snort\rules\pulledpork.rules

• Run PulledPork to download and update Snort rules with the following command. Change the path to pulledpork.conf depending on where PulledPork was extracted:

69 A.3. Snort 2.9.17.1 on Windows 10

pulledpork.pl −c C:\software\pulledpork−0.8.0\etc\pulledpork.conf −l −P −E −T § ¤ ¦ ¥ • Verify that the output from PulledPork includes the following or similar:

Rule Stats... § New:−−−−−−−375 ¤ Deleted:−−−0 Enabled Rules:−−−−375 Dropped Rules:−−−−0 Disabled Rules:−−−0 Total Rules:−−−−−−375 IP Blocklist Stats... Total IPs:−−−−−1200

Done Please review C:\Snort\log\sid_changes.log for additional details Fly Piggy Fly!

¦ ¥ • Change C:\Snort\etc\snort.conf to use only PulledPork rules by making the following changes:

Line 539, add: §include $RULE_PATH/pulledpork.rules ¤

Lines 542−657: Comment out all rule includes, except pulledpork, if there are any.

¦ ¥ • Add the previous PulledPork command as a scheduled task that runs once daily. Use the Task Scheduler app in Windows 10 and choose a random or unusual time to avoid overloading Cisco’s servers.

The instructions in this section used some selected information from a Windows Intrusion Detection System install guide [82].

70 A.4. Open Distro for Elasticsearch install instructions

A.4 Open Distro for Elasticsearch install instructions

To install Open Distro for Elasticsearch, Kibana, Logstash OSS, Filebeat OSS and Metricbeat OSS, the following instructions can be used:

• Choose the latest available and compatible versions of Open Distro for Elasticsearch, Kibana, Logstash OSS and Beats OSS using the following links:

https://opendistro.github.io/for−elasticsearch−docs/version−history/ §https://www.elastic.co/support/matrix#matrix_compatibility ¤

¦ ¥ • Download Open Distro for Elasticsearch, Kibana, Logstash OSS, Filebeat OSS and Met- ricbeat OSS from the following links:

https://opendistro.github.io/for−elasticsearch/downloads.html §https://www.elastic.co/downloads/past−releases/ ¤

¦ ¥ • Install Open Distro for Elasticsearch and Kibana using the install instructions from the ofﬁcial website [4].

• It is recommended to change all the default passwords before starting with Open Distro for Elasticsearch and Kibana, by using the following links:

https://aws.amazon.com/blogs/opensource/change−passwords−open−distro−for−elasticsearch/ §https://github.com/opendistro/for−elasticsearch−docs/issues/5 ¤

¦ ¥ • Follow the Metricbeat OSS install instructions in Appendix A.5.

• Install Filebeat OSS and Logstash OSS, which are installed similarly to Metricbeat OSS.

• Conﬁgure Filebeat OSS and Logstash OSS according to instructions in Appendix A.6.

• Visualizations and Dashboards used in this thesis are shared on the following link. The ndjson ﬁles can be imported as objects into ODFE.

https://github.com/chrizze3/odfe § ¤ ¦ ¥

71 A.5. Metricbeat OSS install and Open Distro for Elasticsearch integration

A.5 Metricbeat OSS install and Open Distro for Elasticsearch integration

To install Metricbeat OSS (open-source software) on Windows:

• Download and install OpenJDK 11 if needed. It can be found on the following link:

https://adoptopenjdk.net § ¤ ¦ ¥ • Download the latest OSS version of Metricbeat, e.g. metricbeat-oss-7.12.0-windows- x86_64.zip, from the following link. Choose the WINDOWS ZIP 64-BIT option.

https://www.elastic.co/downloads/beats/metricbeat−oss § ¤ ¦ ¥ • Unzip/extract the ﬁle to a suitable install location.

• Run Windows PowerShell as administrator and install Metricbeat service with the following commands:

Set−ExecutionPolicy Unrestricted §.\install−service−metricbeat.ps1 ¤ Set−ExecutionPolicy Restricted

¦ ¥ • Modify the Elasticsearch Output settings on lines 92-103 in metricbeat.yml to the following. This requires an existing user in Kibana called "metricbeat". Replace localhost with the actual host if Elasticsearch runs on a different host.

output.elasticsearch: § hosts: ["localhost:9200"] ¤ protocol: "https" username: "metricbeat" password: "yoursecretpasswordhere"

¦ ¥ • If only testing Metricbeat, add the following insecure setting to metricbeat.yml:

ssl.veriﬁcation_mode: none § ¤ ¦ ¥ • Modify the modules.d\system.yml ﬁle so it includes at least the following lines:

− module: system § period: 1s ¤ metricsets: − cpu − memory − network − process − process_summary − socket_summary processes: ["snort"] cpu.metrics: ["normalized_percentages"]

¦ ¥ • Comment out the following lines in the modules.d\system.yml ﬁle:

#process.include_top_n: § # by_cpu: 5 # include top 5 processes by CPU ¤ # by_memory: 5 # include top 5 processes by memory

¦ ¥ • If needed, create a user for Metricbeat OSS in Kibana. Use the menu: Open Distro for Elasticsearch / Security / Internal users.

72 A.5. Metricbeat OSS install and Open Distro for Elasticsearch integration

• If needed, create a role for Metricbeat OSS in Kibana. Use the menu: Open Distro for Elasticsearch / Security / Roles. For testing, duplicate the logstash role and map it to the metricbeat user. For production, read more about the security module in Open Distro for Elasticsearch. • Start the Metricbeat service in the Windows Services app. • Use the menu: Management / Stack Management / Index Patterns / Create index pattern. Name the pattern metricbeat-7.12.0* and choose @timestamp as time ﬁlter. • Create the following scripted ﬁelds if needed. – Snort RAM Bytes. Replace 8535257088L with the total amount of system RAM.

if (doc["process.memory.pct"].size() == 0) return null; § ¤ else return (long) (doc["process.memory.pct"].value*8535257088L)

–¦Snort RAM GB. ¥ if (doc["process.memory.pct"].size() == 0) return null; § ¤ else return doc["process.memory.pct"].value*8.535257088

–¦Snort CPU. ¥ if (doc["process.cpu.pct"].size() == 0) return null; § ¤ else return doc["process.cpu.pct"].value*100

–¦System CPU. ¥ if (doc["system.cpu.total.norm.pct"].size() == 0) return null; § ¤ else return doc["system.cpu.total.norm.pct"].value*100

To install¦ Metricbeat OSS (open-source software) on Ubuntu: ¥ • Install OpenJDK 11 if needed with the following command:

sudo apt−get update §sudo apt−get install openjdk−11−jdk ¤

• ¦Install apt-transport-https if needed with the following command: ¥

sudo apt−get install apt−transport−https § ¤

• ¦Add and conﬁgure the Elastic OSS repository: ¥

wget −qO − https://artifacts.elastic.co/GPG−KEY−elasticsearch | sudo apt−key add − §echo "deb https://artifacts.elastic.co/packages/oss−7.x/apt stable main" | ¤ sudo tee −a /etc/apt/sources.list.d/elastic−7.x.list

• ¦Install Metricbeat OSS: ¥

sudo apt−get update §sudo apt−get install metricbeat ¤

• ¦Manage the Metricbeat service with the following commands: ¥

systemctl status metricbeat.service §sudo systemctl start metricbeat.service ¤ sudo systemctl stop metricbeat.service

• ¦Modify /etc/metricbeat/metricbeat.yml and /etc/metricbeat/modules.d/system.yml according ¥ to the Windows instruction above. Continue with the rest of the Windows install instruction.

73 A.6. Logstash OSS, Snort and Open Distro for Elasticsearch integration

A.6 Logstash OSS, Snort and Open Distro for Elasticsearch integration

This appendix contains a Logstash OSS example configuration for Snort 2 default alert_csv output, located in the logstash.conf file. Consider that Logstash OSS is installed on the same server as Open Distro for Elasticsearch but not on the same server as Filebeat OSS and Snort. The certificate files are not included in Logstash OSS but can be generated with OpenSSL on Ubuntu Linux (self-signed) or acquired from a legitimate source:

input{ § beats{ ¤ port => 5047 ssl => true ssl_certificate_authorities => ["/etc/logstash/RootCA.pem"] ssl_key =>"/etc/logstash/certificate.key" ssl_certificate =>"/etc/logstash/certificate.pem" ssl_verify_mode =>"peer" } } filter{ csv{ separator =>"," # Replace with columns from the CSV file columns => [ "timestamp","sig_generator","sig_id","sig_rev","msg","proto","src","srcport","dst","dstport", "ethsrc","ethdst","ethlen","tcpflags","tcpseq","tcpack","tcplen","tcpwindow","ttl", "tos","id","dmglen","iplen","icmptype","icmpcode","icmpid","icmpseq" ] } mutate{ convert => { "srcport" =>"integer" "dstport" =>"integer" } } date{ # Example2 March 2021: 03/02/2021 11:15:14.266 match => ["timestamp","yy/MM/dd−HH:mm:ss.SSS"] } geoip{ source =>"src"} } output{ elasticsearch{ hosts =>"https://open−distro−for−elasticsearch−host:9200" ssl => true ssl_certificate_verification => false user => logstash password => logstash index =>"logstash−snort−csv−hp" } stdout{} }

¦ Consider using ssl_verify_mode => "force_peer" and ssl_certificate_verification => true in- ¥ stead, which are more secure settings. This requires changing the test certificates included in Open Distro for Elasticsearch. Filebeat OSS input to Logstash OSS can be configured in the filebeat.yml file in the following way:

ﬁlebeat.inputs: §− type: log ¤ enabled: true paths: −c:\Snort\log\ *.csv

74 A.6. Logstash OSS, Snort and Open Distro for Elasticsearch integration

output.logstash: hosts:["logstash−oss−host:5047"] ssl.certificate_authorities:[’C:/software/filebeat−oss−7.12.0−windows−x86_64/RootCA.pem’] ssl.certificate:’C:/software/filebeat−oss−7.12.0−windows−x86_64/certificate.pem’ ssl.key:’C:/software/filebeat−oss−7.12.0−windows−x86_64/certificate.key’

¦ ¥

75 A.7. Snort commands

A.7 Snort commands

A.7.1 Snort 2.9.17.1-WIN64 commands Pcap readback example command:

snort −c C:\Snort\etc\snort.conf −−pcap−file C:\darpa\pcap_list.txt −y −k none −P 65535 § ¤ ¦ Snort 2 NIDS example command: ¥ snort −i 4 −c C:\snort\etc\snort.conf −k none −P 65535 § ¤ ¦ Explanation of settings: ¥ • -c specifies the configuration file • --pcap-file specifies a file which includes a space separated list of pcap files to read, but it can also specify a single pcap file • -i specifies the network interface Snort listens on • -k none instructs Snort to analyze all packets, including those with bad checksums •-P 65535 instructs Snort to not truncate packets below the given size in bytes • -y includes year in timestamps

A.7.2 Snort 3.1.0.0 commands Pcap readback example command:

snort −c /usr/local/etc/snort/snort.lua −−plugin−path=/usr/local/lib/snort_extra −−pcap−dir ~/darpa §−l /var/log/snort −k none −s 65535 −y −z 8 > snortoutput.txt ¤

¦Explanation of settings: ¥ • -c specifies the configuration file • --plugin-path specifies directories or libraries for plugins and can also include a colon separated list • --pcap-dir specifies a directory for pcap files • -l specifies log directory • -k none instructs Snort to analyze all packets, including those with bad checksums • -s 65535 instructs Snort to not truncate packets below the given size in bytes • -y includes year in timestamps • -z specifies the number of packet threads, or if no number is specified the number of available threads reported by the system will be used Snort 3 NIDS with four packet threads example command:

sudo snort −c /usr/local/etc/snort/snort.lua −l /var/log/snort −k none −s 65535 −y §−−plugin−path=/usr/local/lib/snort_extra −−daq−dir /usr/local/lib/daq ¤ −−daq afpacket −−daq−var lb_total=4 −−daq−var fanout_type=hash −i eno1 −−daq−var lb_id=1 −i eno1 −−daq−var lb_id=2 −i eno1 −−daq−var lb_id=3 −i eno1 −−daq−var lb_id=4 −z 4

¦Explanation of DAQ settings for packet threads can be found in the following link [62]: ¥ https://www.snort.org/documents/snort−3−multiple−packet−threads−processing § ¤ ¦ ¥ 76 A.8. Open Distro for Elasticsearch screenshots

A.8 Open Distro for Elasticsearch screenshots

This appendix includes screenshots, ﬁgures A.1, A.2 and A.3, showing the dashboards used in this thesis. The DARPA 1999 Intrusion Detection Dataset was used to populate the visualizations shown. Images were also shared on the following link:

https://github.com/chrizze3/odfe § ¤ ¦ To generate the visualizations and dashboard in Figure A.2, the following process can be ¥ used:

• Install Snort 2 on Windows according to Section A.3. It includes conﬁguring default csv alert output.

• Install ODFE according to Section A.4. It includes Logstash OSS and Filebeat OSS con- ﬁgurations from Section A.6, which match the default csv alert output from Snort 2.

• Run Snort 2 with pcap readback or online analysis to generate alerts.

• Import visualizations with indexes from the shared GitHub repository above.

To generate the visualizations and dashboard in Figure A.3, the following process can be used:

• Install Snort 3 on Ubuntu 18/20 using the ofﬁcial install instructions [37].

• Install ODFE according to Section A.4. Use the Snort 3 Logstash OSS conﬁgurations from the GitHub repository.

• The ODFE server runs two Logstash OSS processes, one for OpenAppID data and one for alerts.

• The Snort server runs two Filebeat OSS services, one for OpenAppID logs and one for json alert ﬁles.

• Run Snort 3 with pcap readback or online analysis to generate alerts and OpenAppID data.

• Import visualizations with indexes from the shared GitHub repository above.

77 A.8. Open Distro for Elasticsearch screenshots

Figure A.1: A system and Snort process dashboard populated by Metricbeat OSS.

Figure A.2: A Snort 2 alert dashboard populated by Filebeat OSS and Logstash OSS.

Figure A.3: A Snort 3 alert dashboard populated by Filebeat OSS and Logstash OSS.

78 Bibliography

[1] Identity Theft Resource Center 2021. ITRC 2020 Data Breach Report. URL: https:// notified.idtheftcenter.org/s/2020- data- breach- report (visited on 03/22/2021). [2] AlienVault. The SIEM Evaluator’s Guide. URL: https://cdn-cybersecurity.att. com/docs/guides/The-SIEM-Evaluators-Guide.pdf (visited on 03/22/2021). [3] Guy Almes, Sunil Kalidindi, and Matthew Zekauskas. A one-way delay metric for IPPM. Tech. rep. RFC 2679, September, 1999. [4] Amazon Web Services, Inc. Open Distro for Elasticsearch. URL: https://opendistro. github.io/for-elasticsearch/ (visited on 04/13/2021). [5] Elasticsearch B.V. About | Elastic. URL: https://www.elastic.co/about/ (visited on 05/09/2021). [6] Elasticsearch B.V. Elastic Stack: Elasticsearch, Kibana, Beats & Logstash | Elastic. URL: https://www.elastic.co/elastic-stack (visited on 03/11/2021). [7] Elasticsearch B.V. File input plugin | Logstash Reference [7.11] | Elastic. URL: https : //www.elastic.co/guide/en/logstash/current/plugins-inputs-file. html (visited on 03/11/2021). [8] Elasticsearch B.V. Filter plugins | Logstash Reference [7.11] | Elastic. URL: https://www. elastic.co/guide/en/logstash/current/filter-plugins.html (visited on 03/11/2021). [9] Elasticsearch B.V. Input plugins | Logstash Reference [7.11] | Elastic. URL: https://www. elastic.co/guide/en/logstash/current/input-plugins.html (visited on 03/11/2021). [10] Elasticsearch B.V. Metricbeat overview | Metricbeat Reference [7.11] | Elastic. URL: https: //www.elastic.co/guide/en/beats/metricbeat/current/metricbeat- overview.html (visited on 03/11/2021). [11] Elasticsearch B.V. Output plugins | Logstash Reference [7.11] | Elastic. URL: https:// www.elastic.co/guide/en/logstash/current/output- plugins.html (visited on 03/11/2021). [12] Shay Banon. Introducing Elastic License v2, simpliﬁed and more permissive; SSPL remains an option | Elastic Blog. URL: https://www.elastic.co/blog/elastic-license- v2 (visited on 04/13/2021).

79 Bibliography

[13] Philippe Biondi and the Scapy community. Scapy. URL: https://scapy.net/ (visited on 02/22/2021). [14] Mohammad Borhani, Madhusanka Liyanage, Ali Hassan Sodhro, Pardeep Kumar, Anca Delia Jurcut, and Andrei Gurtov. “Secure and Resilient Communications in the Industrial Internet”. In: Guide to Disaster-Resilient Communication Networks. Springer In- ternational Publishing, 2020, pp. 219–242. [15] Alessio Botta, Alberto Dainotti, and Antonio Pescapè. “A tool for the generation of realistic network workload for emerging networking scenarios”. In: Computer Networks 56.15 (2012), pp. 3531–3547. [16] Alessio Botta, Walter de Donato, Alberto Dainotti, Stefano Avallone, and Antonio Pescapé. D-ITG 2.8.1 Manual. URL: http : / / traffic . comics . unina . it / software/ITG/manual/D-ITG-2.8.1-manual.pdf (visited on 03/31/2021). [17] Broadcom. Encrypted Trafﬁc Management. URL: https : / / www . broadcom . com / products / cyber - security / network / encrypted - traffic - management (visited on 02/24/2021). [18] Boštjan Brumen and Jernej Legvart. “Performance analysis of two open source intrusion detection systems”. In: 2016 39th International Convention on Information and Com- munication Technology, Electronics and Microelectronics (MIPRO). IEEE. 2016, pp. 1387– 1392. [19] Brian Carpenter and Scott Brim. Middleboxes: Taxonomy and issues. Tech. rep. RFC 3234, February, 2002. [20] Brian Caswell, Jay Beale, and Andrew Baker. Snort intrusion detection and prevention toolkit. Syngress, 2007. [21] Cisco. Cisco Annual Internet Report (2018–2023) White Paper. URL: https : / / www . cisco.com/c/en/us/solutions/collateral/executive-perspectives/ annual - internet - report / white - paper - c11 - 741490 . html (visited on 02/22/2021). [22] Cisco. README.counts. URL: https://snort.org/faq/readme-counts (visited on 05/13/2021). [23] Cisco. README.reload. URL: https://snort.org/faq/readme-reload (visited on 04/11/2021). [24] Cisco. README.ssl. URL: https : / / snort . org / faq / readme - ssl (visited on 05/24/2021). [25] Cisco. Shared Object Rules. URL: https : / / snort . org / faq / shared - object - rules (visited on 04/14/2021). [26] Cisco. SNORT SUBSCRIBER RULES LICENSE AGREEMENT (v. 3.1). URL: https:// snort.org/snort_license (visited on 04/15/2021). [27] Cisco. SNORT Users Manual 2.9.17. Oct. 2020. URL: https : / / snort . org / #documents (visited on 03/25/2021). [28] Cisco. What are Community Rules? URL: https://snort.org/faq/what- are- community-rules (visited on 02/25/2021). [29] Cisco. What are Snort Subscriber Rule Set? URL: https://snort.org/faq/what- are-snort-subscriber-rule-set (visited on 02/25/2021). [30] Cisco. What are the differences in the rule sets? URL: https://snort.org/faq/what- are-the-differences-in-the-rule-sets (visited on 02/22/2021). [31] Cisco. Why are rules commented out by default? URL: https://snort.org/faq/why- are-rules-commented-out-by-default (visited on 05/13/2021).

80 Bibliography

[32] Aldo Cortesi, Maximilian Hils, Thomas Kriechbaumer, and contributors. mitmproxy: A free and open source interactive HTTPS proxy. URL: https://mitmproxy.org/. [33] Victor Costan and Srinivas Devadas. “Intel SGX Explained.” In: IACR Cryptol. ePrint Arch. 2016.86 (2016), pp. 1–118. [34] JJ Cummings, Michael Shirk, and the PulledPork Team. GitHub - shirkdog/pulledpork: Pulled Pork for Snort and Suricata rule management (from Google code). URL: https:// github.com/shirkdog/pulledpork (visited on 03/26/2021). [35] Sébastien Damaye. pytbull - IDS/IPS Testing Framework - home. URL: http://pytbull. sourceforge.net/ (visited on 04/04/2021). [36] Dell. Dell OptiPlex 7010. URL: https : / / www . dell . com / downloads / global / products / optix / en / dell _ optiplex _ 7010 _ spec _ sheet . pdf (visited on 03/12/2021). [37] Noah Dietrich. Snort 3.1.0.0 on Ubuntu 18 & 20. URL: https : / / snort . org / documents (visited on 03/26/2021). [38] Jon Dugan, Seth Elliott, Bruce A. Mah, Jeff Poskanzer, and Kaustubh Prabhu. iPerf - The TCP, UDP and SCTP network bandwidth measurement tool. URL: https://iperf.fr/ (visited on 04/04/2021). [39] Enlyft. Companies using snort and its marketshare. URL: https://enlyft.com/tech/ products/snort (visited on 03/22/2021). [40] Hewlett Packard Enterprise. The Business of Hacking. URL: https : / / www . thehaguesecuritydelta . com / images / HPE _ Security _ Research _ Business_of_Hacking_May_2016.pdf (visited on 03/22/2021). [41] Felix Erlacher, Simon Woertz, and Falko Dressler. “A tls interception proxy with real- time libpcap export”. In: 41st IEEE Conference on Local Computer Networks (LCN 2016), Demo Session. 2016. [42] Joel Esler. Registered vs. Subscriber. URL: https : / / snort . org / documents / registered-vs-subscriber (visited on 02/25/2021). [43] Joel Esler. Snort Supported OSes. URL: https://snort.org/documents/snort- supported-oses (visited on 04/07/2021). [44] Dede Fadhilah and Marza Ihsan Marzuki. “Performance Analysis of IDS Snort and IDS Suricata with Many-Core Processor in Virtual Machines Against Dos/DDoS Attacks”. In: 2020 2nd International Conference on Broadband Communications, Wireless Sensors and Powering (BCWSP). IEEE. 2020, pp. 157–162. [45] Wireshark Foundation. Wireshark User’s Guide Version 3.5.0. URL: https : / / www . wireshark.org/docs/wsug_html_chunked/ChStatHierarchy.html (visited on 05/08/2021). [46] Free Software Foundation, Inc. Frequently Asked Questions about the GNU GPL v2.0 - GNU Project - Free Software Foundation. URL: https://www.gnu.org/licenses/ old-licenses/gpl-2.0-faq.html (visited on 02/25/2021). [47] Alka Gupta and Lalitsen Sharma. “Mitigation of DoS and Port Scan Attacks Using Snort”. In: International Journal of Computer Sciences and Engineering 7 (2019), pp. 248– 258. [48] Alka Gupta and Lalitsen Sharma. “Performance Analysis and Comparison of Snort on Various Platforms”. In: International Journal of Computer Information Systems and Indus- trial Management Applications 10 (Feb. 2020), pp. 23–32. [49] Qinwen Hu, Muhammad Rizwan Asghar, and Nevil Brownlee. “Evaluating network intrusion detection systems for high-speed networks”. In: 2017 27th International Telecommunication Networks and Applications Conference (ITNAC). IEEE. 2017, pp. 1–6.

81 Bibliography

[50] Qinwen Hu, Se-Young Yu, and Muhammad Rizwan Asghar. “Analysing performance issues of open-source intrusion detection systems in high-speed networks”. In: Journal of Information Security and Applications 51 (2020), p. 102426. [51] Proofpoint Inc. ET OPEN Ruleset Download Instructions. URL: https : / / rules . emergingthreats . net / OPEN _ download _ instructions . html (visited on 05/13/2021). [52] Proofpoint Inc. Proofpoint Emerging Threats Rules. URL: https : / / rules . emergingthreats.net (visited on 02/22/2021). [53] Intel. Intel® Xeon® Gold 6126 Processor. URL: https://www.intel.com/content/ www/us/en/products/processors/xeon/scalable/gold- processors/ gold-6126.html (visited on 02/22/2021). [54] Imdadul Karim, Quoc-Tuan Vien, Tuan Anh Le, and Glenford Mapp. “A Compara- tive Experimental Design and Performance Analysis of Snort-Based Intrusion Detec- tion System in Practical Computer Networks”. In: Computers 6.1 (2017). [55] Nattawat Khamphakdee, Nunnapus Benjamas, and Saiyan Saiyod. “Improving Intru- sion Detection System based on Snort rules for network probe attack detection”. In: 2014 2nd International Conference on Information and Communication Technology (ICoICT). 2014, pp. 69–74. [56] Suleman Khan, Kashif Kifayat, Ali Kashif Bashir, Andrei Gurtov, and Mehdi Has- san. “Intelligent intrusion detection system in smart grid using computational intelligence and machine learning”. In: Transactions on Emerging Telecommunications Technolo- gies (2020), e4062. [57] Igor Kotenko, Artem Kuleshov, and Igor Ushakov. “Aggregation of elastic stack instruments for collecting, storing and processing of security information and events”. In: 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Com- puted, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI). IEEE. 2017, pp. 1–8. [58] Daniel E Krych, Joshua Edwards, and Tracy Braun. Generating Artiﬁcial Snort Alerts and Implementing SELK: The Snort-Elasticsearch-Logstash-Kibana Stack. Tech. rep. US Army Research Laboratory Adelphi United States, 2017. [59] James Kurose and Keith Ross. Computer Networking: A Top-Down Approach, 7th edition. Pearson, 2017. [60] MIT Lincoln Laboratory. 1999 DARPA Intrusion Detection Evaluation Dataset. URL: https : / / www . ll . mit . edu / r - d / datasets / 1999 - darpa - intrusion - detection-evaluation-dataset (visited on 02/22/2021). [61] Gordon Lyon. Npcap: Windows Packet Capture Library & Driver. URL: https://nmap. org/npcap/ (visited on 03/25/2021). [62] Yaser Mansour. Snort 3 Multiple Packet Processing Threads. URL: https : / / snort . org/documents/snort-3-multiple-packet-threads-processing (visited on 05/25/2021). [63] Jon Munshaw. Snort Blog: Snort 3 ofﬁcially released. URL: https://blog.snort.org/ 2021/01/snort-3-officially-released.html (visited on 02/26/2021). [64] ntop. GitHub - ntop/PF_RING: High-speed packet processing framework. URL: https:// github.com/ntop/PF%5C_RING (visited on 04/01/2021). [65] ntop. Using Snort with PF_RING. URL: https://www.ntop.org/guides/pf%5C_ ring/thirdparty/snort-daq.html (visited on 05/26/2021).

82 Bibliography

[66] ntop. Using Snort with PF_RING — PF_RING dev documentation. URL: https://www. ntop.org/guides/pf%5C_ring/thirdparty/snort- daq.html (visited on 04/01/2021). [67] Carl Nykvist, Martin Larsson, Ali Hassan Sodhro, and Andrei Gurtov. “A lightweight portable intrusion detection communication system for auditing applications”. In: In- ternational Journal of Communication Systems 33.7 (2020), e4327. [68] Catherine Paquet. Implementing Cisco IOS Network Security (IINS). Cisco Systems, Inc. Apr. 2009. URL: https://community.cisco.com/legacyfs/online/legacy/ 3/6/8/15374863-Ch%5C%206_Network_Security_Using_Cisco_IOS_IPS. pdf (visited on 04/04/2021). [69] Geong Sen Poh, Dinil Mon Divakaran, Hoon Wei Lim, Jianting Ning, and Achintya Desai. “A Survey of Privacy-Preserving Techniques for Encrypted Traffic Inspection over Network Middleboxes”. In: arXiv preprint arXiv:2101.04338 (2021). [70] Inc Proofpoint. Proofpoint Joins Open Information Security Foundation, Reinforces Open Source Commitment | Proofpoint US. URL: https : / / www . proofpoint . com / us / proofpoint - joins - open - information - security - foundation - reinforces-open-source-commitment (visited on 02/25/2021). [71] Proofpoint Inc. Emerging Threats Pro Ruleset | Proofpoint. URL: https : / / www . proofpoint . com / us / threat - insight / et - pro - ruleset (visited on 02/26/2021). [72] Tamara Radivilova, Lyudmyla Kirichenko, Dmytro Ageyev, Maxim Tawalbeh, and Vi- talii Bulakh. “Decrypting SSL/TLS traffic for hidden threats detection”. In: 2018 IEEE 9th International Conference on Dependable Systems, Services and Technologies (DESSERT). IEEE. 2018, pp. 143–146. [73] Martin Roesch and The Snort Team. Snort 3 User Manual. URL: https : / / usermanual . wiki / Document / snortmanual . 760997111 (visited on 03/25/2021). [74] Daniel Roethlisberger and contributors. SSLsplit - transparent SSL/TLS interception. URL: https://www.roe.ch/SSLsplit (visited on 02/24/2021). [75] Amtul Saboor, Monis Akhlaq, and Baber Aslam. “Experimental evaluation of Snort against DDoS attacks under different hardware configurations”. In: 2013 2nd National Conference on Information Assurance (NCIA). IEEE. 2013, pp. 31–37. [76] Phil Schroeder. Emerging Threats FAQ. URL: https://doc.emergingthreats.net/ bin/view/Main/EmergingFAQ (visited on 02/25/2021). [77] Huntsman Security. White Paper | The Essential Guide to SIEM. URL: https://www. huntsmansecurity . com / wp - content / uploads / 2018 / 11 / Essential _ Guide_to_SIEM.pdf (visited on 03/24/2021). [78] Pranav Shukla and Sharath Kumar M N. Learning Elastic Stack 7.0: Distributed search, analytics, and visualization using Elasticsearch, Logstash, Beats, and Kibana, 2nd Edition. Packt Publishing, 2019. [79] Abraham Silberschatz, Peter B. Galvin, and Greg Gagne. Operating System Concepts Ninth Edition. John Wiley & Sons, Inc., 2013. [80] Security Onion Solutions. PCAPs for Testing — Security Onion 2.3 documentation. URL: https : / / docs . securityonion . net / en / 2 . 3 / pcaps . html (visited on 03/25/2021). [81] Shalvi Srivastava, Sweta Anmulwar, AM Sapkal, Tarun Batra, Anil Kumar Gupta, and Vinodh Kumar. “Comparative study of various traffic generator tools”. In: 2014 Recent Advances in Engineering and Computational Sciences (RAECS). IEEE. 2014, pp. 1–6.

83 Bibliography

[82] Michael Steele. How to Install a Windows Intrusion Detection System (WinIDS). URL: http : / / www . winsnort . com / tutorials / article / 4 - installing - an - apache2- webserver- logging- events- to- a- mysql- database/ (visited on 06/24/2021). [83] Daniel Steinvall. Utvärdering av signaturdatabaser i systemet Snort [Evaluation of Signature Databases in the System Snort]. Swedish Defence Research Agency, 2019. [84] Strawberry Perl for Windows. URL: https : / / strawberryperl . com (visited on 03/26/2021). [85] TRex Team. GitHub - cisco-system-trafﬁc-generator/trex-core: trex-core site. URL: https: //github.com/cisco-system-traffic-generator/trex-core (visited on 04/04/2021). [86] Bhagya Tholpady. Snort Blog: How to use Snort2lua. URL: https://blog.snort. org/2020/10/how-to-use-snort2lua.html (visited on 02/25/2021). [87] Vilhelm Verendel. “Quantiﬁed security is a weak hypothesis: a critical survey of results and assumptions”. In: Proceedings of the 2009 workshop on New security paradigms workshop. 2009, pp. 37–50. [88] Louis Waked, Mohammad Mannan, and Amr Youssef. “To intercept or not to intercept: Analyzing tls interception in network appliances”. In: Proceedings of the 2018 on Asia Conference on Computer and Communications Security. 2018, pp. 399–412. [89] Simon Woertz. swoertz / master-project — Bitbucket. URL: https://bitbucket.org/ swoertz/master-project (visited on 02/22/2021). [90] Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and An- ders Wesslén. Experimentation in software engineering. Springer Science & Business Me- dia, 2012. [91] Bruno Bogaz Zarpelão, Rodrigo Sanches Miani, Cláudio Toshio Kawakani, and Sean Carlisto de Alvarenga. “A survey of intrusion detection in Internet of Things”. In: Jour- nal of Network and Computer Applications 84 (2017), pp. 25–37.