Intrusion Detection Using Open Source Tools

Revista Informatica Economică nr.2(46)/2008 75

Intrusion Detection using Open Source Tools

We have witnessed in the recent years that open source tools have gained popularity among all types of users, from individuals or small businesses to large organizations and enterprises. In this paper we will present three open source IDS tools: OSSEC, Prelude and SNORT. Keywords: Network security, IDS, IPS, intrusion detection, intrusion prevention, open source.

ntroduction OSSEC (www.ossec.net) is an open source I The traditional form of securing the net- host-based intrusion detection system work, the firewall proved to be insufficient. (HIDS). Given the nature and the complexity of the According to their own website, "OSSEC is a attacks, new ways of protecting the network scalable, multi-platform, open source Host- had to be developed. Intrusion Detection Sys- based Intrusion Detection System (HIDS). It tems, or shortly IDS are tools to monitor the has a powerful correlation and analysis en- events occurring in a computer system or gine, integrating log analysis, file integrity network and detect signs of possible inci- checking, Windows registry monitoring, cen- dents, such as violation of computer security tralized policy enforcement, rootkit detection, policies, acceptable use policies or standard real-time alerting and active response. It security practices. An Intrusion Prevention runs on most operating systems, including System, or IPS in addition to detecting inci- Linux, OpenBSD, FreeBSD, MacOS, Solaris dents, can play an active role by attempting and Windows." to stop the possible incidents which are de- In the recent years, OSSEC has received sev- tected. The Intrusion Detection/Prevention eral awards. In 2007, it was placed on the Systems (IDPS) can be classified according first position in the Top 5 Open Security to different criteria. The NIST Guide to In- Tools in the enterprise by LinuxWorld. In trusion Detection and Prevention Systems 2006, OSSEC was voted as the second best (NIST 800-94), lists four main types: IDS tool by the survey conducted by the sec- - Network-Based, which monitor the net- tools.org website. Recently, OSSEC has work; been acquired by Third Brigade, but accord- - Wireless - monitor wireless network traffic; ing to the press release, OSSEC will remain - Network Behavior Analysis (NBA) - which open source. examine the network traffic to identify The OSSEC HIDS can be installed as a threats like denial of service attacks and stand-alone tool to monitor one host or can malware; be deployed in a multi-host scenario, one in- - Host-Based - monitor a single host and the stallation being the server and the others as events occurring within that host. agents. The server and agents communicate According to the way an IDPS detect the in- securely using encryption. cidents, they can be classified into three cat- OSSEC also has intrusion prevention fea- egories: signature-based, anomaly-based and tures, being able to react to specific events or stateful protocol analysis, but most IDPS sys- set of events by using commands and active tems use multiple detection methodologies, responses. The system allows the creation of either separately or integrated, to improve the new commands which can be bound to performance. events. The system comes with some prede- Below we will briefly present a selection of fined active response tools, but the adminis- three open source tools: OSSEC, Prelude and trator can add others. SNORT. OSSEC was designed initially for Linux, but 76 Revista Informatica Economică nr.2(46)/2008 it evolved and since version 0.8 it also fea- them into the database, having several features a Windows agent, which can monitor tures such as relaying or filtering the events. the event log and other files. The Prelude LML is a signature-based log PRELUDE. Prelude (www.prelude- analyzer monitoring log files and received ids.com) is more than an open source IDS syslog messages for suspicious activity. It system - it is a framework which enables can handle events generated by a large set of other security applications to report to a cen- components, such as: Cisco PIX, Clamav, tralized system. ipchains, Netfilter, ipfw, Nokia ipso, Ms- It makes use of the IDMEF (Intrusion Detec- SQL, Nagios, NTsyslog, Pam, Portsentry, tion Message Exchange Format) standard Postfix, Proftpd, ssh etc. proposed by IETF, which allows defining the The PreludeDB library allows the developers events recorded by different sensors using a to use the Prelude IDMEF database, provid- single language. Prelude is considered a hy- ing an abstraction layer based upon the type brid system, since it allows the coexistence and the format of the database used to store of the event data from host-based, network- IDMEF alerts. based or wireless IDS agents, or simply any The last component, Prewikka is a console other security application. Existing security providing advanced features like contextual applications can be modified to use the Pre- filtering, aggregation, etc. Below we can see lude system, using the provided C, Python a simplified architecture with three sensors. and Perl frameworks. An interesting feature of Prelude is the possi- The main components of a Prelude system bility to “relay” the events between manag- are: the Prelude library (framework), the ers: in the following example, Branch A is Prelude Manager, the Prelude-LML, Prelu- relaying all the events to the Prelude Manag- deDB library and Prewikka (the console). er located in the Network Operation Center The Prelude library, Libprelude, is used to of the organization. Using this setup, Branch access the Prelude system and provides an A can only access the events recorded by API (Application Programming Interface) to sensors D, E and F, while the NOC (Network create events in the IDMEF (Intrusion Detec- Operation Center) can access the events gen- tion Message Exchange Format) format. It erated from all branches. can be used for failover and allows the crea- An interesting thing related to the three open tion of sensors that read the events received source tools presented in this article, is that by one or a set of prelude managers. they can be integrated: Prelude IDS frame- The Prelude Manager is a high-availability work has native support for both Snort and server which collects and normalizes infor- OSSEC. A list with all the external sensors mation from distributed sensors and stores which are supported natively is presented in Table 1.

Fig.1. Prelude Simple Architecture

Using the IDMEF API, additional own sen- SNORT. Snort (www.snort.org) is, without sors can be defined and programmed. In the doubt, one of the most popular open source PRELUDE documentation there is such an security tools. With millions of downloads, it example written in C. is used by individuals as well as large corpo- Revista Informatica Economică nr.2(46)/2008 77 rations or government organizations. The be faster than for most commercial IDS tools. first version was written in 1998 by Martin According to Jennifer Albornoz Mulligan, Roesch, who later founded Sourcefire. Since security researcher at Forrester Research, then, the product evolved both as features “Once the community writes those signa- and as portability: currently Snort is available tures, Sourcefire takes a little extra time to for most major platforms including Win- test them before it puts them out there, but dows, BSD, Solaris or Mac OS X. It is worth the process is still generally more responsive mentioning that Snort has an excellent sup- than others". This community ruleset comes port from the user community. This can be in addition to the official ruleset, released by considered as a big advantage since the the Sourcefire Vulnerability Research Team. availability of signatures for new attacks can

Fig.2. Prelude Relaying Architecture

Security Tool Description AuditD audit system Nepenthes detection and collection of worms and malware NuFW authenticating firewall OSSEC HIDS PAM authentication tasks PFLogger reports alerts from OpenBSD firewall Sancp information collection on network activity Samhain file integrity checker Snort NIDS Table 1. Native Support in Prelude IDS

Snort can run in different modes, ranging and rules. from a simple sniffer to a IPS system: Snort Preprocessors allow the functionality - Sniffer mode; of Snort to be extended by allowing users - Packet Logger mode - logs packets to disk; and programmers add modular plug-ins. Pre- - NIDS mode - allows Snort to analyze net- processor code is run before the detection work traffic for matches against a user- engine is called, but after the packet has been defined rule set and to perform several ac- decoded. Such preprocessors exist for IP de- tions based on these matches; fragmentation (Frag3), TCP stream reassem- - Inline (IPS) mode - allows Snort to drop or bly (Stream4), HTTP, FTP, SMTP, SSH etc. pass packets based on the specific Snort Snort rules are used by the system to detect rules. incidents. Snort rules are divided into two Working as an IDS, Snort uses preprocessors logical sections, the rule header and the rule 78 Revista Informatica Economică nr.2(46)/2008 options. The rule header contains the rule's notification and logging when the system action, protocol, source and destination IP fires events. The supported types are text addresses and netmasks, and the source and (console), syslog and Unified 2 (a serialized destination ports information. The rule op- binary stream format). tion section contains alert messages and in- - The TCP Stream Reassembler – provide formation on which parts of the packet target-based services for reassembling TCP should be inspected to determine if the rule segments into normalized streams. action should be taken. The Data Source API is an interface between At this time, the latest stable version is the Data Source component and the Dis- 2.8.2.2. Currently a new beta version is under patcher. testing, Snort Security Platform (SnortSP) v The Dispatcher coordinates the information 3.0, which comes with a new architecture flow between the different components of (see Figure 3 below). The primary compo- Snort 3.0. nents are the Snort Security Platform (SnortSP) and traffic analysis engine modules that plug into SnortSP. The current beta version includes one engine module contain- ing the Snort 2.8.2 detection engine. Its major features are: - Shell-based user interface with embedded scripting language; - Native IPv6, MPLS and GRE support; - Native support for inline operation; - More subsystem plugin types such as data acquisition modules, decoders and traffic analyzers; - Multithreaded execution model - multiple analysis engines may operate simultaneously on the same traffic; - Better performance. The main components of the new architecture will be briefly presented below.

The Data Source component encapsulates the Fig.3. Snort 3 Architecture common functionality needed by any network traffic before the analysis tasks and in- The Attribute Management System (AMS) is corporates several modules: used to store network data about the opera- - The Data Acquisition Module (DAQ) – gets tional environment in which the Snort system the packets from the underlying hardware – is deployed. Snort 3.0 can incorporate arbitrary packet in- The Analytics System contains the detection terfaces like libpcap, IPQ etc; engines. The idea is to have the detection in - The Decoder – which performs the same analytics modules that run as separate tasks like in Snort 2.x: validate the packets, threads. Using this architecture, several detect protocol anomalies and provide a refe- threads can operate simultaneously on the da- rential structure; ta coming from the dispatcher. Currently - The Flow Manager – can help tracking the among the analytics modules under devel- conversations on the network; opment are the Snort 2.x engine, RNA (for The IP Defragmenter – can provide IPv4 and Sourcefire implementation) and a Lua (a IPv6 services for putting the packets back to- scripting language) traffic analysis module. gether and for fragment reassembly. Snortd and command shell. Snortd is the The Action System handles event queuing, daemon process, and the command shell pro- Revista Informatica Economică nr.2(46)/2008 79 vides management services for different homepage has different sections by industry software modules, processes and threads, type, like Financial, Manufacturing, Public health management and includes the Lua etc, only briefly presenting one real case scripting language. study per section, but without naming the re- While Snort does not offer a GUI, there are spective organizations. many complementary open-source tools like However a question arises: are open source ACID (PHP-based) or BASE (Basic Analysis IDS products good enough for corporate use? and Security Engine) which provide this GUI According to a study from Gartner [9], some functionality for Snort. Or it can be inte- commercial IDS providers use Snort and/or grated into an IDS framework, like Prelude Snort signatures for their appliances, which IDS. There are also other tools, a simple gives Snort credibility in this direction. But query on freshmeat.net resulting in a listing different organization have different needs. of 68 projects, at the time of the writing of While for big enterprises open source IDS this article. Of course, we do not bring into systems might not be enough, since they are discussion the quality of all these projects, the target of the most sophisticated attacks, not all of them being developed to a ‘mature’ the smaller and medium-sized businesses can level. protect their intranet for free or at a fraction Sourcefire, the company which maintains of the price of a commercial system. Howev- SNORT, offers also commercial solutions er, even for the companies choosing com- based on the Snort Engine – Sourcefire IPS mercial IDS, tools like Snort, OSSEC or Pre- and Sourcefire RNA . It is worth to be noted lude can be used for 'a second opinion’. that in 2006, the Israel based company Checkpoint intended to acquire Sourcefire, References the owner of Snort, but, according to the me- 1. Karen Scarfone, Peter Mell, NIST 800-94, dia [7], “FBI and Pentagon expressed strong Guide to Intrusion Detection and Prevention Sys- reservations about the deal because Snort is tems (IDPS), Feb 2007 used to safeguard classified U.S. military and 2. Open Source Intrusion Detection: No-cost Sys- intelligence data.” As a consequence, the tem Lockdown – http://itmanagement.earthweb.com/secu/article.p deal was off, but we can deduct the impor- hp/11076_3673721_1 tance given to Snort. 3. OSSEC Homepage - www.ossec.net 4. SNORT Homepage - www.snort.org Conclusion 5. Prelude IDS Homepage - www.prelude- The three tools presented above are open ids.com source, but commercial support is available 6. The Intrusion Detection Message Exchange from the companies maintaining the prod- Format (IDMEF) ucts. The fact that commercial companies are http://tools.ietf.org/rfc/rfc4765.txt behind these projects, while the products re- 7. The Case for Open Source IDS, main open source, can be seen as a good http://www.itsecurity.com/features/the-case-for- sign, allowing the tools to be further devel- open-source-ids-022607/ 8. Check Point drops plans to acquire Sourcefire - oped in an organized manner and providing http://searchsecurity.techtarget.com/news/article/ the necessary funding. The intended users for 0,289142,sid14_gci1175561,00.html the tools listed here are not only individuals, 9. Magic Quadrant for Network Intrusion Preven- but also businesses and other organizations. tion System Appliances, 1H08 - Getting a list with the companies using OS- http://mediaproducts.gartner.com/reprints/tipping SEC, Prelude or Snort is not an easy task, point/154849.html since most organizations prefer not to expose 10. Snort 3.0 Architecture Series Part 1: Over- their security architecture. For example, OS- view, http://securitysauce.blogspot.com SEC lists only two companies from Brazil: 11. Snort 3.0 Architecture Series Part 2: Changes Embratel and Resenet ISP. The Prelude IDS and Betas, http://securitysauce.blogspot.com