Bachelor Thesis Summer Semester 2013 at Fachhochschule Frankfurt am Main University of Applied Sciences Department of Computer Science and Engineering towards Bachelor of Science Computer Science submitted by Jens Kühnel Centralized and structured log file analysis with Open Source and Free Software tools 1. Supervisor: Prof. Dr. Jörg Schäfer 2. Supervisor: Prof. Dr. Matthias Schubert topic received: 11. 07. 2013 thesis delivered: 30. 08. 2013 Abstract This thesis gives an overview on the Open Source and Free Software tools available for a centralized and structured log file analysis. This includes the tools to convert unstructured logs into structured log and different possibilities to transport this log to a central analyzing and storage station. The different storage and analyzing tools will be introduced, as well as the different web front ends to be used by the system administrator. At the end different tool chains will be introduced, that are well tested in this field. Revisions Rev. 269: Official Bachelor these sent to FH Rev. 273: Removal of Affidavit, fix of Pagenumber left/right II Table of Contents 1 Introduction.......................................................................................................................................1 1.1 Selection criteria........................................................................................................................1 1.2 Programs that are included in this thesis...................................................................................2 1.3 What this thesis is not covering.................................................................................................4 1.3.1 Hadoop...............................................................................................................................4 1.3.2 Programs that are not included in this thesis......................................................................4 1.4 Structure of this thesis................................................................................................................6 1.5 History of log files.....................................................................................................................6 2 Definitions.........................................................................................................................................7 2.1 Log file.......................................................................................................................................7 2.2 Centralized log file.....................................................................................................................7 2.3 Definition structured log files....................................................................................................7 2.4 Definition Open Source and Free Software...............................................................................8 2.5 Definition Log File Analysis......................................................................................................9 3 Components and Functions.............................................................................................................10 3.1 Formats....................................................................................................................................11 3.1.1 Semi structured logs.........................................................................................................11 3.1.1.1 BSD syslog (RFC3164)............................................................................................11 3.1.1.2 Modern syslog (RFC 5424)......................................................................................11 3.1.2 Structured logs.................................................................................................................12 3.1.2.1 CEE..........................................................................................................................12 3.1.2.2 GELF........................................................................................................................13 3.1.2.3 JSON-logstash..........................................................................................................14 3.1.2.4 Systemd journal........................................................................................................15 3.1.2.5 Windows Event Log.................................................................................................16 3.1.2.6 Auditlog....................................................................................................................17 3.1.2.7 Intrusion Detection Message Exchange Format (IDMEF).......................................18 3.1.3 Other formats...................................................................................................................18 3.2 Collector/Shipper.....................................................................................................................19 3.2.1 File...................................................................................................................................19 3.2.2 Sockets, named pipes and STDIN....................................................................................19 3.2.3 Local Windows Eventlog.................................................................................................19 3.2.4 Compare collector / shipper.............................................................................................19 3.3 Transport..................................................................................................................................20 3.3.1 Syslog...............................................................................................................................20 3.3.2 AMQP..............................................................................................................................21 3.3.3 STOMP............................................................................................................................21 3.3.4 Ømq/ZMTP......................................................................................................................21 3.3.5 Redis.................................................................................................................................21 3.3.6 Lumberjack......................................................................................................................22 3.3.7 Remote Windows Eventlog..............................................................................................22 3.3.8 Compare Transports.........................................................................................................22 3.4 Transformation/Normalization.................................................................................................23 3.4.1 Pattern-DB.......................................................................................................................24 3.4.2 Liblognorm.......................................................................................................................24 3.4.3 Octopussy.........................................................................................................................24 3.4.4 Grok.................................................................................................................................25 III 3.4.5 Heka.................................................................................................................................25 3.4.6 Filter_regex......................................................................................................................25 3.4.7 nxlog.................................................................................................................................26 3.5 Storage.....................................................................................................................................26 3.5.1 Log files...........................................................................................................................26 3.5.2 SQL..................................................................................................................................26 3.5.3 NoSQL.............................................................................................................................27 3.5.4 Compare Storage..............................................................................................................27 3.6 Analysis....................................................................................................................................27 3.6.1 nxlog.................................................................................................................................27 3.6.2 SEC..................................................................................................................................28 3.6.3 Sagan................................................................................................................................28 3.6.4 Logstash and metrics........................................................................................................29 3.6.5 Graylog2...........................................................................................................................29 3.7 Visual output............................................................................................................................29