Logging and Monitoring to Detect Network Intrusions and Compliance Violations in the Environment
Total Page:16
File Type:pdf, Size:1020Kb
Interested in learning more about security? SANS Institute InfoSec Reading Room This paper is from the SANS Institute Reading Room site. Reposting is not permitted without express written permission. Logging and Monitoring to Detect Network Intrusions and Compliance Violations in the Environment Log Management and Intrusion Detection solutions have been evolving for years. Yet, it remains a challenge for organizations of all sizes to meet the operational, audit and security needs using these solutions. This paper presents a solution to bridge logging, log based intrusion detection and network based intrusion detection using well known free open source tools available on the Security Onion Linux Distribution. It walks through the logging, monitoring and alerting approach necessary for security, compliance and q... Copyright SANS Institute Author Retains Full Rights AD Logging and Monitoring to Detect Network Intrusions and Compliance Violations in the Environment . ts h g ri GIAC (GCIA) Gold Certification ll u f Author: Sunil Gupta, [email protected] s Advisor: Dr. Kees Leune in ta re r Accepted: July 4, 2012o th u , A te tu ti Abstract s n Log Management and Intrusion I Detection solutions have been evolving for years. Yet, it remains a challenge for organizationsS of all sizes to meet the operational, audit and security needs using these solutions. ThisN paper presents a solution to bridge logging, log based intrusion detection A and network Sbased intrusion detection using well known free open source tools available on the Security Onion Linux Distribution. It walks through the logging, monitoring and alerting 12 approach0 necessary for security, compliance and quality of service. In the process it provides for cost 2 effective, customizable and scalable solution alternative to vendor based Security ©Information & Event Management (SIEM) solutions. © 2012 The SANS Institute Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46 Author retains full rights. Logging and Monitoring to Detect Network Intrusions and Compliance Violations | 2 1. Introduction Intrusion detection is the process of monitoring the events occurring in a computer system or network and analyzing them for signs of possible incidents, which are violations or . imminent threats of violation of computer security policies, acceptable use policies, or standardts security practices. An intrusion detection system (IDS) is software that automates the intrusionh detection process. Network-Based IDS (NIDS) monitors network traffic for particular networkg ri segments or devices and analyzes the network and application protocol activity to lidentifyl suspicious activity (Scarfone & Mell, 2007). fu s Security Log Analysis Systems are also known as Log-based Intrusionin Detection Systems (LIDS). Log Analysis For Intrusion Detection is the process or techniquesta used to detect attacks on a specific environment using logs as the primary source of information.e LIDS is also used to r detect computer misuse, policy violations and other forms ofr inappropriate activities (Cid, 2007). o The main thesis of this paper is that NIDS andth LIDS are necessary for effectively u monitoring the security posture of an organization.A Both techniques, network-based detection and log-based detection, complement each other, in the identification and reporting of security te incidents. u tit 1.1. Outline s n This paper describes I how to build a system that combines Network Based Intrusion Detection with Log BasedS Intrusion Detection to create a comprehensive security monitoring N platform. ChapterA 2 provides an overview of essential terminology in the field of Security Information EventS Monitoring and Log Management. Chapter 3 builds on the terminology by proposing12 a technical architecture and by providing configuration guidance. Chapter 4 discusses Log0 Analysis and Correlation and the paper concludes by discussing Alerting and Reporting in Chapter 2 5. © This paper describes a fictional scenario in which an intrusion is detected using NIDS and LIDS alerts on the monitor console. The example demonstrates the value of this approach by following an intruder performing a network scan, connect to a system and control gain, followed by privilege escalation on the target system. Sunil Gupta, [email protected] © 2012 The SANS Institute Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46 Author retains full rights. Logging and Monitoring to Detect Network Intrusions and Compliance Violations | 3 1.2. Problem Addressed In an organization, there are many possible signs of incidents which may go unnoticed each day. These events can be studied mainly by analyzing network behavior or by reviewing computer security event logs. In order to avoid or minimize the losses from an incident outcome, . ts the events need to be analyzed as close to real-time as possible. Logging and intrusion detectionh systems have the potential to produce very large amount of data, and all that data must beg ri managed, filtered and analyzed. Having a single approach and a unified platform helpll s with this very difficult and challenging task to monitor and report in near-real time. fu s Automation is needed to perform an initial analysis of the data iandn to alert on select events of interest for human review. Event correlation software andt acentralized logging can be of great value in automating the analysis process. However, the effectivenessre of the process depends on the quality of the data and the data rules that goesr into it. o th 2. Log Management and SIEM Overviewu A The NIST Guide to Computer Security, Log Management (Kent & Souppaya, 2006) states that information regarding an incident mayte be recorded in several places, such as firewalls, u routers, network IDS, host IDS, andtit application logs. Organizations should deploy one or more centralized logging servers ands configure logging devices throughout the organization to send duplicates of their log entriesn to the centralized logging servers. A log management infrastructure I consists of the hardware,S software, networks and media used to generate, transmit, store, analyze, and dispose of log data.N This section describes the typical architecture and functions of a log A management.S 2 2.1.0 1Log Management Architecture 2 The NIST Guide to Computer Security Log Management (Kent & Souppaya, 2006) ©explains that a log management infrastructure typically comprises of three tiers: log generation, log analysis and storage, and log monitoring. Sunil Gupta, [email protected] © 2012 The SANS Institute Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46 Author retains full rights. Logging and Monitoring to Detect Network Intrusions and Compliance Violations | 4 The log generation tier involves hosts making their logs available to log servers in the second tier. This is performed in two different ways. The exact method depends on the log type, and, on the host and network controls. In one way hosts run some services to send their log data over the network to log collection servers. Alternatively, hosts allow the log servers to pull the . log data from them. The logs are often transferred to the log receivers either in a real-time or ts near-real-time manner, or in occasional batches based on a schedule. h g ri The log analysis and storage tier is composed of one or more log servers receivll ing log data from the hosts. These log receivers are also called collectors or aggregators. Tou facilitate log f analysis, automated methods of converting logs from multiple formats to as single standard format needs to be implemented. Syslog format of logging is often usedin for this purpose. ta The log monitoring tier contains consoles that are used for emonitoring and reviewing of log r data and the results of automated analysis. Report generation,r management dashboards and log o baselines may also be done using consoles as part of tthish tier. u The scope of log management infrastructure can be dictated by many factors, including the , A organization’s internal structure, systemt typese (e.g., a separate infrastructure for enterprise security systems), log types (e.g., a separateu infrastructure for application audit logs), and facility locations. it st 2.2. Log Management In Functions S Log managementN infrastructures typically perform several functions that assist in the storage, analysis,A and disposal of log data. These functions are normally performed in such a way that they do notS alter the original logs (Kent & Souppaya, 2006). 12 0 General functions of log management infrastructure include log parsing, event filtering and 2 event aggregation. On the storage side, log management has to provide for log rotation, log ©archival, log compression, log reduction, log conversion, log normalization and log file integrity. Event correlation, log viewing and log reporting are some of the analysis functions of a log management infrastructure. Sunil Gupta, [email protected] © 2012 The SANS Institute Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46 Author retains full rights. Logging and Monitoring to Detect Network Intrusions and Compliance Violations | 5 Kent & Souppaya (2006) also explain that a log management infrastructure usually encompasses most or all of the functions described in this section. The placement of some of the log functions among the three tiers of the log management infrastructure depends primarily on the type of log management software used. It is in the best interest of organizations to have . appropriate auditing in place that allows for effective and efficient log management. ts h 2.3. SIEM ig r Security information and event management (SIEM) software provides thel logl management infrastructure encompassing log analysis, log storage and log monitoringu tiers. f What sets SIEM products apart from traditional log management software sis the ability to perform event correlation, alerting, incident management, reporting andin forensic investigation ta based on event analysis. There are many SIEM solutions commerciallyre available today and these solutions provide different set of these features and additionalr add-ons.