Malicious Traffic Detection in Local Networks with Snort Lo¨ıcEtienne / EPFL - SSC Abstract personal computers. CERN hosts many conferences, and many students from universities all around the world. Snort is an open source Network Intrusion Detection System With scientists representing more than 80 countries, combining the benefits of signature, protocol and anomaly network and web traffic span across the whole Internet. based inspection and is considered to be the most widely de- The CERN Public Network is liberal, and there are ployed IDS/IPS technology worldwide. However, Snort's de- few restrictions on the network usage. The network is ployment in a large corporate network poses different prob- heterogenous, and contains mainly Windows (XP, Vista, lems in terms of performance or rule selection. This pa- 2000, Seven), Linux (Scientific Linux CERN, Ubuntu, per proposes different improvements to the Snort Security Debian, Redhat), and Mac OS (9, 10.4, 10.5) computers, Platform: the use of another library is proposed to signifi- but also hosts more exotic devices such as tablet PCs, cantly improve the amount of traffic that can be analyzed, mobile phones, and other various devices. As all the de- and Snort's multithreading possibilities are explored. A new vices are registered, the CERN public network provides rule classification has been devised, and rulesets suited to an easy way to interact and cross-check data with the large corporate networks are proposed. The use of Oinkmas- corresponding users. ter has been tested and documented to seamlessly update Table 1 shows a typical distribution of network traffic Snort's rules. on a sample of 1.5 million packets. This table provides interesting results, because they significantly differ from what could be found in a typical enterprise, where most 1 Introduction people have never heard the term of SSH for example. This paper will explore how Snort, an open source Net- #Protocol % of traffic TCP 93 % work Intrusion Detection System, can be used to secure UDP 6 % and monitor such a very large corporate network. Other 1 % SSH 26 % This thesis will start by a brief tour of horizon of net- HTTP 12 % work security in Chapters 3 and 4; Chapters 5 and 6 will SSL 2 % X11 1 % present Snort's functionalities and rule system. It will SMTP < 1 % show that Snort's default classification is inadequate for Other 59 % large network rule selection. Chapter 7 will present findings concerning commonly Tab. 1: Protocol breakdown of traffic sample available rules, starting with rules related to CERN poli- cies. A solution to improve Emule detection when Skype is present is proposed. Rules that significantly improve The IDS typically sees 600Mbits/s on average, and the detection of infected and compromized devices at more than 1Gbit/s during peak hours. CERN are also proposed. With such a broad variety of devices, origins, and Chapter 8 will introduce a tool to handle rule up- amount of traffic, this network offers a perfect frame- dates, and Chapter 9 will suggest possible performance work for IDS tuning. Clearly, naive approaches such as improvements to the Snort platform. blocking all non-http traffic would be unacceptable in this context. 2 About CERN 3 Network Traffic Analysis as Part of an 2.1 CERN IDS System CERN, the European Organization for Nuclear Research, 3.1 IDS is the largest particle physics laboratory in the world. Commonly referred to as the birthplace of the world- An Intrusion Detection System (or IDS) is composed of wide web, it currently hosts scientists from some 580 in- software and/or hardware designed to detect unwanted stitutes and counts 20 European member states. attempts of accessing, manipulating, and/or disabling of CERN's missions are Research, Technology, Collabo- computer systems. An IDS is used to detect several types ration, and Education [1]. of malicious behaviors that can compromize the security and trust of a computers system. These threats are vari- 2.2 CERN Public Network ous, and include network attacks against vulnerable ser- vices, data driven attacks on applications, host based at- The CERN Public Network offers a great playground for tacks such as privilege escalation, unauthorized accesses, any IDS system. or malware (viruses, worms) [2]. CERN hosts more than 10'000 visitors each year, most of whom bring their own, unmanaged and/or unpatched, 1 4 State of the Art - From Raw Packet Capture to Advanced Detection Mechanisms 2 Terminology • Source & Destination IPs: Provide the source and destination addresses of every packet. False positive: A false positive is defined by an incorrect result of a test which erroneously detects something • Protocol: The transport protocol. Typically TCP when in fact it is not present. In an IDS, it typically or UDP. consists in detecting a network threat which is in • Source & Destination Ports: Complete the source fact non-existent. and destination addresses. Intrusion: Any set of actions that compromise the in- • Size: The size of the packets. tegrity, confidentiality or availability of a resource. • Flags: Whether the packet has some flag bits set. Attack: An attempt to bypass security controls on a These could be: urgent, SYN, ACK, FIN, ... computer. May precede an intrusion. • Payload: The data itself, that will be delivered to Signature: Network traffic is examined for preconfig- the application running on destination address and ured and predetermined patterns known as signa- port. tures. Many attacks or threats today have dis- tinct signatures. Good security practice requires a Each of these features can provide valuable informa- database of known signatures to be constantly up- tion for a NIDS. Today, many corporate switches can dated to mitigate emerging threats. export raw data, NetFlow, sFlow or similar data. Net- Flow data contains Source and Destination IP and port, Alert: Event generated when a signature matches traffic and the amount of traffic transferred per flow. activity. On a higher level, it is also possible to analyze the payload of every packet. However this requires a full Noise: Amount of unneeded, unwanted, or false-positive understanding of the protocols by the analyzer, as well alerts; masking or reducing the number of real alerts. as a full access to the traffic, which is not easily scalable. Structure of an IDS An IDS is typically composed of NetFlow is an embedded instrumentation within Cisco three main parts: a sensor, an engine, and a console. IOS Software to characterize network operation. It gives The sensor's main task is to analyze all the data ac- the administrators the tools to understand who, what, cording to some configuration data, and create events when, where, and how network traffic is flowing. accordingly. The engine records events logged by the Data is collected directly by switches, and can be ex- sensors in a database, and uses a system of rules to gen- ported to a reporting server. A flow is a n-tuple1, usually erate alerts from security events received. The console identified by the source and destination IPs and ports, monitors events and alerts, and allows to interact with the protocol, and the unilateral number of bytes trans- the latter. ferred. In many cases, the three components are combined in a This data can then be aggregated and analyzed to de- simple device or appliance. A more detailed explanation tect unwanted or malicious behavior. One could, for of the components can be found in [3]. example, count the number of SMTP servers (destina- tion port 25), the number of peers, or the number of SSH servers contacted by any host to detect misbehav- Different Types of IDS There are two main types of ing hosts (sending spam mail, using P2P, or doing ssh IDS working at different points in the infrastructure: scans). Network IDS (NIDS): The NIDS scans all network traf- A worm detection system based on netflow data is pro- fic that is fed to it. The NIDS typically functions in posed in [5]. the same way as an antivirus software: every single packet is scanned for patterns which may indicate a High Level Analysis With this technique, the payload problem. of every single packet is inspected. This requires much more resources than NetFlow based inspection tech- Host-based IDS (HIDS): HIDS are typically installed on niques, as all packets have to be opened up to the 7th every host. HIDS are more platform specific, and are layer of the OSI model to be analyzed. However, this focused on the target computer. HIDS can capture is obviously much more useful, as it can detect proto- all the traffic generated by a host, which a NIDS cols running on any ports, as well as any byte pattern typically cannot do in a switched network. HIDS regardless of the underlying protocol. are not necessarily traffic based, but also look at the system's state. Snort is an open source NIDS software [6]. Combining the benefits of signature, protocol and anomaly based 4 State of the Art - From Raw Packet inspection Snort is the most widely deployed IDS/IPS Capture to Advanced Detection technology worldwide. It is able to perform \high level analysis" on all the traffic flowing through its sensor. Mechanisms Snort is available in two different version: Snort itself, and the Snort Security Platform (Snort SP). Snort SP Network Traffic Analysis can be performed in many is an extension of Snort, using the same engine, but al- different ways. Here is a list of the features that charac- lowing much more options, such as multithreading, an terizes network traffic. Each of these features is part of the OSI model [4]. 1 n depending on the version 5 Snort in a Large Corporate Network 3 interactive shell, and performance improvements.