Royal Military College of Canada Department of Electrical and Computer Engineering
EEE404 - Cyber Defence
Dr G.S. Knight [email protected] Rm S5108 6470
Capt Stephen McKeon [email protected] Rm S4206 3802
Exercise 3 A Basic Suricata Primer
Introduction
This exercise is an introduction to the basic usage and configuration of the open-source intrusion detection tool, Suricata.
"For years, Snort (developed and maintained by SourceFire) has been the de facto standard for open source Intrusion Detection/Prevention Systems (IDS/IPS). Its engine combines thebenefits of signatures, protocols, and anomaly-based inspection and has become the most widely deployed IDS/IPS in the world.
Suricata, a new and less widespread product developed by the Open Information Security Foundation (OISF), has recentlyappeared, and seems really promising. It is also based on signatures but integrates revolutionary techniques. This engine embeds a HTTP normalizer andparser (HTP library) that provides very advanced processing of HTTP streams, enabling the understanding of traffic on the 7th level of the OSI model." [Reference: http://www.aldeid.com/wiki/Suricata-vs-snort , 23 November 2013.]
In this exercise you will complete the following tasks:
• discover the location of the configuration and documentation files • configure the network address variables • confirm the alert and log output settings • confirm the rule selection • test your configuration files • discover the location and format of alert and log outputs
You will run Suricata on your Linux VM. No network setup is required.
Part 1 - Discovering Suricata Files
There are many different types of files associated with Suricata and initially it may be overwhelming. In this part of the exercise you will simply explore your directory trees to familiarize yourself with the location of the essential Suricata files. For this you will need either to open a file manager (GUI access) or to open a terminal window (command line access) - your choice.
Explore the Suricata files listed below. Do not make any changes, just explore.
• Executable: o The Suricata executable file is located in directory /usr/bin/ • Scripts: o Suricata related scripts for the tools Barnyard2 and BASE are located in directory /usr/local/bin/ o Each script’s usage will be described in the next lab. • Configuration files: o Local copies of the Suricata related configuration files that you will require are: § /home/eee404/suricata.yaml § /home/eee404/threshold.config § /home/eee404/barnyard2.conf • Alert and Log files: o The Suricata alert and log files will be written to /home/eee404/log/ as specified in the configuration file. • Rules: o For the EEE404 installation: § The Emerging Threat rules used by Suricata are located in the directory /etc/suricata/rules/ § The generator message mapping is located in /etc/suricata/rules/gen-msg.map § This file provides valuable information about the mapping of rules to their respective generator identifiers (gen-id) and signature identifiers (sig-id). § Signature Identifier Description (SID) mappings for existing rules are located in /etc/suricata/rules/sid-msg.map § In order to support any new rules that you create two local files are provided: § A local rules file /home/eee404/local.rules § A local mapping file has been provided at /home/eee404/local.map. § The procedure to add new local rules is described in the next lab.
Part 2 - Configuring Network Address Variables in Suricata
In this part of the exercise you will begin creating your own customized copy of a Suricata configuration file.
• Ensure that you have a copy of the following four files in your home directory: § suricata.yaml § threshold.config § local.rules § local.map
• Open suricata.yaml with text editor Leafpad. o Note that this is a rather lengthy file (approximately 1328 lines).
o It will be helpful if you display line numbers. Select Options -> Line Numbers.
o Briefly scroll through the configuration file and become loosely familiar with the relative location of some of the configuration options. § An easy way to find particular configuration sections is to use Ctrl-F to find "some parameter" and if necessary Ctrl-G to find the next occurrence.
• Go to the section in the configuration file for setting the network address variables. The parameters changed here apply to network variables as they relate to the IDS sensor location (~ line 985) o Configure the network variables based upon the information below. § Change the home network to the following subnetwork address: § HOME_NET: “[10.31.0.0/16]” § For the external network, assume that it is everything BUT your network. Therefore confirm it is set as follows: § EXTERNAL_NET: “!$HOME_NET” § For now leave all other network servers and port lists as per the default settings. § Save your changes but leave the configuration file open and continue to the next part.
Part 3 - Suricata Output
In this part of the exercise you will confirm the types of alert logging that have been selected for Suricata. Suricata provides a rich set of plug-ins for supporting various alert and log data formats including various types of databases. In this exercise Suricata has been configured to output alerts in a text format, alerts in a unified format, and event logs for http traffic.
• Confirm that the default logging will be to a location in your home directory. (~ line 56) o default-log-dir: /home/eee404/log
• Go to the alert logging section of the configuration file and confirm that the following types of alerts have been enabled: o - fast: Alerts logged in a text readable format similar to Snort’s fast.log output. (~ line 75) § Filename: fast.log o – unified2-alert: Alerts logged in a unified2 output format for later use with Barnyard2. (~ line 254). § Filename: unified2.alert o – http-log: A log of HTTP requests. (~ line 296) § Filename: http.log o – stats: A log of various Suricata engine counters such as the number of packets, the number of TCP sessions, the number of alerts, … (~ line 409) § Filename: stats.log
• Note that the logging of pcap files is disabled (~ line 361). Since we are running Suricata against previously captured pcaps, enabling this logging option would merely create a local copy of the entire pcap file, and it would do so every time Suricata is run. Clearly this is neither warranted nor efficient.
Part 4 – Suricata Rules
In this part of the exercise you will review the current rules configuration. Literally thousands of rules are available for use with Suricata. We will use only the rules provided by the Emerging Threat creators and the local rules that we create ourselves.
• In order for Suricata to know where to find the rules it uses a default rule path variable (~ line 1865). o Confirm that the following has been set: § default-rule-path: /etc/suricata/rules
• Suricata must also know which rule files to use including any custom or local rules (~ line 1867). o Review the list of rule files and confirm that your local rules are included. § - /home/eee404/local.rules
• Finally, confirm that your threshold configuration file is included (~ line 1933). o Thresholding will be explained in the next lab. § threshold-file: /home/eee404/threshold.config
Part 5 – Suricata Application-Layer Protocols
Suricata supports several application-layer protocols. (~ line 772).
• All application-layer protocols are enabled in the default Suricata configuration file. Confirm that, as a minimum, the following protocols are enabled in your configuration file:
o TLS o FTP o SSH o DNS (TCP and UDP) o HTTP
Part 6 – Testing the Suricata Configuration File and Your Rules
Suricata provides an easy way to check the syntactic correctness of your configuration file along with any included rules. As you are making future changes to the configuration file you should save frequently and test it. Similarly, you should also test any new rules you create.
• To test your configuration file, use switches -c (specify the rules/configuration file) and -T (test configuration file). o suricata -c ~/suricata.yaml -T • If there are no errors the output should look similar to that below, with potentially multiple warnings:
11/10/2014 -- 16:14:06 -
11/10/2014 -- 16:14:06 -
11/10/2014 -- 16:14:08 -
Part 7 - Investigating Alerts (the hard way)
In this part you will run Suricata against a previously captured pcap file and investigate one of the generated alerts using the text based alert output and the original pcap file.
• Run Suricata as follows: o suricata –c ~/suricata.yaml \
–r ~/Resources/Traces/Exercise3.pcap
• Once Suricata completes, observe that you have 4 files in your log directory: fast.log, http.log, stats.log, unified2.alert.timestamp. Ignore the unified2 file for now. • Open the stats.log file to observe the logged statistics. o You should see that there are a total of 61,696 decoded IPv4 packets consisting of 60,445 TCP packets, 1,120 UDP packets and 131 ICMP packets. § How many TCP SYN, SYN-ACK and RST packets are there?
§ What might explain the relatively high number of RST packets? o You should see that there are 2,052 TCP sessions. § How many of these are HTTP requests? Check your output log files as one contains the answer. You should observe that there are 738 HTTP requests.
o You should also see that there are a total of 139 alerts. § How many different types of alerts are there? How many of each type? From the fast.log file you might observe that there are 10 different types of alerts, ranging in frequency from 106 occurrences to only 1 occurrence, but doing so manually is very tedious. One of the following options may help: § Option 1: Open the fast.log file in LibreOffice Calc using a space delimiter. Then sort and subtotal the data on the unique alert signature column. § This may seem like a good option, but try option 2 first, then decide. § Option 2: Use the following combination of command line programs; use the man pages for any command you are unfamiliar with. § cut ~/log/fast.log -d" " -f4 | sort | \
uniq -c | sort –n
o If you did not get the same results as indicated above see an instructor for assistance!
• Let's investigate one specific type of alert, “ET POLICY Inbound Frequent Emails - Possible Spambot Inbound”. o Use grep to isolate this category of alert: § grep 'Spambot' ~/log/fast.log § You should see several instances of Possible Spambot Inbound alerts, all originating from host 10.27.0.8. o Let's focus on just those alerts directed at target 10.31.2.4. § grep 'Spambot' ~/log/fast.log | grep '10.31.2.4' § Now you should see 4 instances of Possible Spambot Inbound alerts against this target. o To simply generate a summary of how many times this alert fired against this target run the following:
§ grep 'Spambot' ~/log/fast.log | grep -c '10.31.2.4'
§ Again the answer should be 4 instances. o How many unique internal hosts were there associated with these alerts? Recall how you answered a similar question above. § You should determine that there are 5 different targeted mail servers on your network. o Let's now follow up on this possible spam directed at your network by further investigating one of the incidents that generated the alert. § To do so let's look at the original pcap file and filter for just the packets associated with one of the target hosts and only mail traffic. § tcpdump -n \
-r ~/Resources/Traces/Exercise3.pcap \
'host 10.31.2.4 and port 25' | less
§ You should see that beyond the suspected spammer there is at least one other host communicating with this mail server. § To make this clearer modify the filter to see just the mail packets destined to our target host. § tcpdump -n \
-r ~/Resources/Traces/Exercise3.pcap \
'dst host 10.31.2.4 and port 25' | less
§ You should now see that in addition to our suspect spammer 10.27.0.8, there is also traffic coming from an internal host 10.31.16.4.
§ Now let’s write the results of our filter to a small pcap file for use with Wireshark to see if the content of the suspect traffic yields any new evidence. §
§ tcpdump -n \
-r ~/Resources/Traces/Exercise3.pcap \
–w ~/log/mail_10.31.2.4.pcap \
'host 10.31.2.4 and port 25'
§ wireshark –r ~/log/mail_10.31.2.4.pcap & § Do you think this looks like spam? If you are not sure, it may be more obvious using the Follow TCP Stream feature.
o Does the evidence of your investigation support the assertion that your network has been spammed? Or is it more likely that the rule fired due to the unusually high number of mail servers on your network? The answer is most likely the latter. In fact, one of the biggest problems with intrusion detection is the high volume of false alarms that are generated using community detection rules that have not been “pruned and tuned” to fit your network. We explore this aspect in more detail in the next lab.
o This investigation raises one other very important aspect of intrusion detection and investigation. Unlike in the fabricated exercise scenario above, it is not standard practice to include mail traffic in packet captures due to the great risk of privacy violations.
• Rerun Suricata against the Exercise3.pcap file.
o suricata –c ~/suricata.yaml \
–r ~/Resources/Traces/Exercise3.pcap
o Note that Suricata will continue to accumulate alerts and continue to create new entries in the statistics log and the HTTP requests log. You should also note that Suricata creates a separate unified2 file with a different time stamp each time it is run. o While you are testing and configuring Suricata it will be very helpful if you clean up these files regularly using the GUI file manager or from the command line using: § rm ~/log/filename
Conclusion
In this exercise you have learned the location of the essential files associated with a typical Suricata installation, have completed a very basic configuration review of Suricata for use as a network intrusion detection system, and have learned a few basic techniques involved in investigating alerts. These skills will be necessary in the next lab.