DOCSLIB.ORG

Centralized and Structured Log File Analysis with Open Source and Free Software Tools

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Centralized and Structured Log File Analysis with Open Source and Free Software Tools Bachelor Thesis Summer Semester 2013 at Fachhochschule Frankfurt am Main University of Applied Sciences Department of Computer Science and Engineering towards Bachelor of Science Computer Science submitted by Jens Kühnel Centralized and structured log file analysis with Open Source and Free Software tools 1. Supervisor: Prof. Dr. Jörg Schäfer 2. Supervisor: Prof. Dr. Matthias Schubert topic received: 11. 07. 2013 thesis delivered: 30. 08. 2013 Abstract This thesis gives an overview on the Open Source and Free Software tools available for a centralized and structured log file analysis. This includes the tools to convert unstructured logs into structured log and different possibilities to transport this log to a central analyzing and storage station. The different storage and analyzing tools will be introduced, as well as the different web front ends to be used by the system administrator. At the end different tool chains will be introduced, that are well tested in this field. Revisions Rev. 269: Official Bachelor these sent to FH Rev. 273: Removal of Affidavit, fix of Pagenumber left/right II Table of Contents 1 Introduction.......................................................................................................................................1 1.1 Selection criteria........................................................................................................................1 1.2 Programs that are included in this thesis...................................................................................2 1.3 What this thesis is not covering.................................................................................................4 1.3.1 Hadoop...............................................................................................................................4 1.3.2 Programs that are not included in this thesis......................................................................4 1.4 Structure of this thesis................................................................................................................6 1.5 History of log files.....................................................................................................................6 2 Definitions.........................................................................................................................................7 2.1 Log file.......................................................................................................................................7 2.2 Centralized log file.....................................................................................................................7 2.3 Definition structured log files....................................................................................................7 2.4 Definition Open Source and Free Software...............................................................................8 2.5 Definition Log File Analysis......................................................................................................9 3 Components and Functions.............................................................................................................10 3.1 Formats....................................................................................................................................11 3.1.1 Semi structured logs.........................................................................................................11 3.1.1.1 BSD syslog (RFC3164)............................................................................................11 3.1.1.2 Modern syslog (RFC 5424)......................................................................................11 3.1.2 Structured logs.................................................................................................................12 3.1.2.1 CEE..........................................................................................................................12 3.1.2.2 GELF........................................................................................................................13 3.1.2.3 JSON-logstash..........................................................................................................14 3.1.2.4 Systemd journal........................................................................................................15 3.1.2.5 Windows Event Log.................................................................................................16 3.1.2.6 Auditlog....................................................................................................................17 3.1.2.7 Intrusion Detection Message Exchange Format (IDMEF).......................................18 3.1.3 Other formats...................................................................................................................18 3.2 Collector/Shipper.....................................................................................................................19 3.2.1 File...................................................................................................................................19 3.2.2 Sockets, named pipes and STDIN....................................................................................19 3.2.3 Local Windows Eventlog.................................................................................................19 3.2.4 Compare collector / shipper.............................................................................................19 3.3 Transport..................................................................................................................................20 3.3.1 Syslog...............................................................................................................................20 3.3.2 AMQP..............................................................................................................................21 3.3.3 STOMP............................................................................................................................21 3.3.4 Ømq/ZMTP......................................................................................................................21 3.3.5 Redis.................................................................................................................................21 3.3.6 Lumberjack......................................................................................................................22 3.3.7 Remote Windows Eventlog..............................................................................................22 3.3.8 Compare Transports.........................................................................................................22 3.4 Transformation/Normalization.................................................................................................23 3.4.1 Pattern-DB.......................................................................................................................24 3.4.2 Liblognorm.......................................................................................................................24 3.4.3 Octopussy.........................................................................................................................24 3.4.4 Grok.................................................................................................................................25 III 3.4.5 Heka.................................................................................................................................25 3.4.6 Filter_regex......................................................................................................................25 3.4.7 nxlog.................................................................................................................................26 3.5 Storage.....................................................................................................................................26 3.5.1 Log files...........................................................................................................................26 3.5.2 SQL..................................................................................................................................26 3.5.3 NoSQL.............................................................................................................................27 3.5.4 Compare Storage..............................................................................................................27 3.6 Analysis....................................................................................................................................27 3.6.1 nxlog.................................................................................................................................27 3.6.2 SEC..................................................................................................................................28 3.6.3 Sagan................................................................................................................................28 3.6.4 Logstash and metrics........................................................................................................29 3.6.5 Graylog2...........................................................................................................................29 3.7 Visual output............................................................................................................................29
Load more

Recommended publications
  • Naemonbox Manual Documentation Release 0.0.7
    NaemonBox Manual Documentation Release 0.0.7 NaemonBox Team September 16, 2016 Contents 1 Introduction 3 1.1 Target audience..............................................3 1.2 Prerequisite................................................3 2 About Naemonbox 5 2.1 Project..................................................5 2.2 Features..................................................6 3 Installation Guide 7 3.1 System requirements...........................................7 3.2 Recommended system requirements...................................7 3.3 Client Operating Systems........................................7 3.4 Openvz VPS installation.........................................8 3.5 GNU/Linux Debian 7 (or later) Installation...............................8 3.6 Installing Naemonbox..........................................8 4 Getting Started 9 4.1 Step one.................................................9 4.2 Step two................................................. 10 4.3 Step three................................................. 10 4.4 Step four................................................. 10 5 Configuring Naemon 11 5.1 Introduction............................................... 11 5.2 Actions.................................................. 11 5.3 Hosts Definition............................................. 12 5.4 Services.................................................. 13 5.5 Commands................................................ 14 5.6 Time periods............................................... 15 5.7 Contacts................................................
    [Show full text]
  • Josh Malone Systems Administrator National Radio Astronomy Observatory Charlottesville, VA
    heck What the #%!@ is wrong ^ with my server?!? Josh Malone Systems Administrator National Radio Astronomy Observatory Charlottesville, VA 1 Agenda • Intro to Monitoring • Internet protocols 101 • • Nagios SMTP • IMAP • Install/Config • HTTP • Usage • Custom plugins • Packet sniffing for dummies • Intro to Troubleshooting • Tools • telnet, openssl • grep, sed • ps, lsof, netstat 2 MONITORING 3 Automated Monitoring Workflow 4 Monitoring Packages: Open Source • • Pandora FMS • Opsview Core • Naemon • • • • • • Captialware ServerStatus • Core • Sensu All Trademarks and Logos are property of their respective trademark or copyright holders and are used by permission or fair use for education. Neither the presenter nor the conference organizers are affiliated in any way with any companies mentioned here. 5 Monitoring Packages: Commercial • Nagios XI • Groundwork • PRTG network monitor • CopperEgg • WhatsUp Gold • PRTG network monitor • op5 (Naemon) All Trademarks and Logos are property of their respective trademark or copyright holders and are used by permission or fair use for education. Neither the presenter nor the conference organizers are affiliated in any way with any companies mentioned here. 6 Why Automatic Service Monitoring? • Spot small problems before they become big ones • Learn about outages before your users do • Checklist when restoring from a power outage • Gives you better problem reports than users • Problems you might never spot otherwise • Failed HDDs in RAIDs • Full /var partitions • Logs not rotating • System temperature rising 7 Why Automatic Service Monitoring? • Capacity planning • Performance data can generate graphs of utilization • RAM, Disk, etc. • Availability reports - CAUTION • Easy to generate -- even easier to generate wrong • Make sure your configurations actually catch problems • Will also include problems with Nagios itself :( • If you’re going to quote your availability numbers (SLAs, etc.) make sure you understand what you’re actually monitoring.
    [Show full text]
  • Monitoring Bareos with Icinga 2 Version: 1.0
    Monitoring Bareos with Icinga 2 Version: 1.0 We love Open Source 1 © NETWAYS Table of Contents 1 Environment 2 Introduction 3 Host 4 Active Checks 5 Passive Events 6 Graphite 2 © NETWAYS 1 Environment 3 © NETWAYS Pre-installed Software Bareos Bareos Database (PostgreSQL) Bareos WebUI Icinga 2 IDO (MariaDB) Icinga Web 2 Graphite 4 © NETWAYS 2 Introduction 5 © NETWAYS 2.1 Bareos 6 © NETWAYS What is Bareos? Backup Archiving Recovery Open Sourced Backup, archiving and recovery of current operating systems Open Source Fork of Bacula (http://bacula.org) Forked 2010 (http://bareos.org) AGPL v3 License (https://github.com/bareos/bareos) A lot of new features: LTO Hardware encryption Bandwidth limitation Cloud storage connection New console commands Many more 7 © NETWAYS Bareos Structure 8 © NETWAYS 2.2 Icinga 2 9 © NETWAYS Icinga - Open Source Enterprise Monitoring Icinga is a scalable and extensible monitoring system which checks the availability of your resources, notifies users of outages and provides extensive BI data. International community project Everything developed by the Icinga Project is Open Source Originally forked from Nagios in 2009 Independent version Icinga 2 since 2014 10 © NETWAYS Icinga - Availability Monitoring Monitors everything Gathering status Collect performance data Notifies using any channel Considers dependencies Handles events Checks and forwards logs Deals with performance data Provides SLA data 11 © NETWAYS What is Icinga 2? Core based on C++ and Boost Supports all major *NIX and Windows platforms Powerful configuration
    [Show full text]
  • Network Monitoring Using Nagios and Autoconfiguration for Cyber Defense Competitions
    NETWORK MONITORING USING NAGIOS AND AUTOCONFIGURATION FOR CYBER DEFENSE COMPETITIONS Jaipaul Vasireddy B.Tech, A.I.E.T, Jawaharlal Nehru Technological University, India, 2006 PROJECT Submitted in partial satisfaction of the requirements for the degree of MASTER OF SCIENCE in COMPUTER SCIENCE at CALIFORNIA STATE UNIVERSITY, SACRAMENTO FALL 2009 NETWORK MONITORING USING NAGIOS AND AUTOCONFIGURATION FOR CYBER DEFENSE COMPETITIONS A Project by Jaipaul Vasireddy Approved by: __________________________________, Committee Chair Dr. Isaac Ghansah __________________________________, Second Reader Prof. Richard Smith __________________________ Date ii Student: Jaipaul Vasireddy I certify that this student has met the requirements for format contained in the University format manual, and that this Project is suitable for shelving in the Library and credit is to be awarded for the Project. __________________________, Graduate Coordinator ________________ Dr. Cui Zhang Date Department of Computer Science iii Abstract of NETWORK MONITORING USING NAGIOS AND AUTOCONFIGURATION FOR CYBER DEFENSE COMPETITIONS by Jaipaul Vasireddy The goal of the project is to monitor the services running on the CCDC (College Cyber Defense Competition) network, using Nagios which uses plugins to monitor the services running on a network. Nagios is configured by building configuration files for each machine which is usually done to monitor small number of systems. The configuration of Nagios can also be automated by using shell scripting which is generally done in an industry, where the numbers of systems to be monitored are large. Both the above methods of configuration have been implemented in this project. The project has been successfully used to know the status of each service running on the defending team’s network.
    [Show full text]
  • Performance Monitoring Using Nagios Core Hpc4e-Comcidis Vin´Icius P
    Performance Monitoring Using Nagios Core HPC4e-ComCiDis Vin´ıcius P. Kl^oh Mariza Ferro Gabrieli D. Silva Bruno Schulze LNCC { Petr´opolis,RJ Abstract The High Performance Computing for Energy (HPC4e) project aims to apply\new exascale HPC techniques to energy industry simulations, customizing them if necessary, and going beyond the state-of-the-art in the required HPC exascale simulations for different energy sources that are the present and the future of energy like, wind energy production and design, efficient combustion systems for biomass-derived fuels (biogas), and exploration geophysics for hydrocarbon reservoirs". Beyond the general objective, there are specific technical objectives that will be developed to enhance the final results. Our objective is study the mapping and optimization of the codes proposed for simulations in energy domain (atmosphere, biomass and geophysics for energy), analysing all the aspects related with the performance of these simulations' codes. Trying to meet all these objectives, we are investigating performance tools that would help our research. We investigated at first tools that enable online measurement of performance (online approaches are without code instrumentation). More specifically, in this work we present our initial work with Nagios and the hard begin to put this performance tool on work. In this work we present the steps and instructions, on how to install and configure Nagios Core to enhance it monitoring your local and remote host. July 2016 Contents 1 Introduction 2 2 Nagios Core 3 3 Install and Configure Nagios Core and Basic Plugins 4 4 Plugins 6 4.1 Install and Configure NRPE (Nagios Remote Plugin Executor) .
    [Show full text]
  • Log-Management-Tenshi.Pdf
    Network Monitoring and Management Log Management Network Startup Resource Center www.ws.nsrc.org These materials are licensed under the Creative Commons Attribution-NonCommercial 4.0 International license (http://creativecommons.org/licenses/by-nc/4.0/) Log Management & Monitoring • Keep your logs in a secure place • Where they can be easily inspected • Watch your log file • They contain important information – Many things happen – Someone needs to review them – It’s not practical to do this manually Log Management & Monitoring On your routers and switches And, on your servers Log Management • Centralize and consolidate log files • Send all log messages from your routers, switches and servers to a single node – a log server. • All network hardware and UNIX/Linux servers can be monitored using some version of syslog (we use either syslog-ng or rsyslog for this workshop). • Windows can, also, use syslog with extra tools. • Save a copy of the logs locally, but, also, save them to a central log server. Syslog Basics Uses UDP protocol, port 514 Syslog messages have two attributes (in addition to the message itself): Facility Level Auth Security | Emergency (0) Authpriv User | Alert (1) Console Syslog | Critical (2) Cron UUCP | Error (3) Daemon Mail | Warning (4) Ftp Ntp | Notice (5) Kern News | Info (6) Lpr | Debug (7) Local0 ...Local7 | Centralized Logging Configuring Centralized Logging Cisco hardware – At a minimum: logging ip.of.logging.host Unix and Linux nodes – In syslogd.conf, or in rsyslog.conf, add: *.* @ip.of.log.host – Restart syslogd, rsyslog or syslog-ng Other equipment have similar options – Options to control facility and level Receiving Messages – syslog-ng • Identify the facility that the equipment is going to use to send its messages.
    [Show full text]
  • NXLOG Community Edition Reference Manual for V2.9.1716 I
    Ed. v2.9.1716 NXLOG Community Edition Reference Manual for v2.9.1716 i NXLOG Community Edition Reference Manual for v2.9.1716 Ed. v2.9.1716 Ed. v2.9.1716 NXLOG Community Edition Reference Manual for v2.9.1716 ii Copyright © 2009-2014 NXLog Ltd. Ed. v2.9.1716 NXLOG Community Edition Reference Manual for v2.9.1716 iii Contents 1 Introduction 1 1.1 Overview . .1 1.2 Features . .1 1.2.1 Multiplatform . .1 1.2.2 Modular architecture . .1 1.2.3 Client-server mode . .2 1.2.4 Log message sources and destinations . .2 1.2.5 Importance of security . .2 1.2.6 Scalable multi-threaded architecture . .2 1.2.7 High performance I/O . .2 1.2.8 Message buffering . .2 1.2.9 Prioritized processing . .3 1.2.10 Avoiding lost messages . .3 1.2.11 Apache-style configuration syntax . .3 1.2.12 Built-in config language . .3 1.2.13 Scheduled tasks . .3 1.2.14 Log rotation . .3 1.2.15 Different log message formats . .4 1.2.16 Advanced message processing capabilites . .4 1.2.17 Offline processing mode . .4 1.2.18 Character set and i18n support . .4 2 Installation and quickstart 5 2.1 Microsoft Windows . .5 2.2 GNU/Linux . .6 2.2.1 Installing from DEB packages (Debian, Ubuntu) . .6 2.2.2 Installing from RPM packages (CentOS, RedHat) . .6 2.2.3 Configuring nxlog on GNU/Linux . .6 Ed. v2.9.1716 NXLOG Community Edition Reference Manual for v2.9.1716 iv 3 Architecture and concepts 7 3.1 History .
    [Show full text]
  • Fedora 16 System Administrator's Guide
    Fedora 16 System Administrator's Guide Deployment, Configuration, and Administration of Fedora 16 Jaromír Hradílek Douglas Silas Martin Prpič Eva Kopalová Eliška Slobodová Tomáš Čapek Petr Kovář Miroslav Svoboda System Administrator's Guide John Ha David O'Brien Michael Hideo Don Domingo Fedora 16 System Administrator's Guide Deployment, Configuration, and Administration of Fedora 16 Edition 1 Author Jaromír Hradílek [email protected] Author Douglas Silas [email protected] Author Martin Prpič [email protected] Author Eva Kopalová [email protected] Author Eliška Slobodová [email protected] Author Tomáš Čapek [email protected] Author Petr Kovář [email protected] Author Miroslav Svoboda [email protected] Author John Ha Author David O'Brien Author Michael Hideo Author Don Domingo Copyright © 2011 Red Hat, Inc. and others. The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. The original authors of this document, and Red Hat, designate the Fedora Project as the "Attribution Party" for purposes of CC-BY-SA. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
    [Show full text]
  • Ubuntu Server Guide Basic Installation Preparing to Install
    Ubuntu Server Guide Welcome to the Ubuntu Server Guide! This site includes information on using Ubuntu Server for the latest LTS release, Ubuntu 20.04 LTS (Focal Fossa). For an offline version as well as versions for previous releases see below. Improving the Documentation If you find any errors or have suggestions for improvements to pages, please use the link at thebottomof each topic titled: “Help improve this document in the forum.” This link will take you to the Server Discourse forum for the specific page you are viewing. There you can share your comments or let us know aboutbugs with any page. PDFs and Previous Releases Below are links to the previous Ubuntu Server release server guides as well as an offline copy of the current version of this site: Ubuntu 20.04 LTS (Focal Fossa): PDF Ubuntu 18.04 LTS (Bionic Beaver): Web and PDF Ubuntu 16.04 LTS (Xenial Xerus): Web and PDF Support There are a couple of different ways that the Ubuntu Server edition is supported: commercial support and community support. The main commercial support (and development funding) is available from Canonical, Ltd. They supply reasonably- priced support contracts on a per desktop or per-server basis. For more information see the Ubuntu Advantage page. Community support is also provided by dedicated individuals and companies that wish to make Ubuntu the best distribution possible. Support is provided through multiple mailing lists, IRC channels, forums, blogs, wikis, etc. The large amount of information available can be overwhelming, but a good search engine query can usually provide an answer to your questions.
    [Show full text]
  • Zabbix 4.2 Data Processing and More Sponsors Gold Sponsors Co-Organizer
    Zabbix 4.2 data processing and more Sponsors Gold sponsors Co-organizer 2 What is Zabbix? Zabbix is a universal Open Source enterprise level monitoring solution 3 4 We have found a good balance between giving away free software and having a sustainable growth 5 Free Software Services Services that save your time and money Free Software Services Services that save your time and money Customers in 75+ countries 8 Some of our users 9 Zabbix Real-time Alerting & Data collection problem Visualization Remediation detection 10 Where we are currently? 3.0 LTS 3.2 3.4 4.0 LTS 11 Zabbix 4.0 LTS 12 Data preprocessing { “host": { "name": "Linux #2356", "vms": [{ “name”: “vm034”, “network": { "read": “0xfa673”, "write": “0x45b30” } }, { “name”: “vm076”, “network": { "read": “0x76ab”, “write": “0xff3a” } }] } } 13 Data preprocessing "write": “0x45b30” } }, { “name”: “vm076”, “network": { "read": “0x76ab”, 0x76ab “write": “0xff3a” } }] JSON Path: .host.vms[1].network.read } } 14 Data preprocessing 0x76ab JSON Path: .host.vms[1].network.read 15 Data preprocessing 0x76ab 76ab Regexp: 0x(.*) 16 Data preprocessing 0x76ab 76ab 330379 Hex to Decimal 17 Data preprocessing 0x76ab 76ab 330379 338308096 KB -> bytes 18 Data preprocessing 19 Efficiency Zabbix server mysql[status] 20 Efficiency Master item 21 Efficiency Dependent items 22 Efficiency mysql[questions] mysql[status] Zabbix server mysql[reads] 23 Efficiency mysql[questions] mysql[status] Zabbix server mysql[reads] Performance, less user parameters, all logic in templates 24 Limitations 4.0 LTS 4.2 Not flexible
    [Show full text]
  • Linux and Open Source for (Almost) Zero Cost PCI Compliance
    Linux and Open Source for (Almost) Zero Cost PCI Compliance Rafeeq Rehman 2 Some Introductory Notes ¡ Payment Card Industry (PCI) standard is not a government regulaon. ¡ Who needs to comply with PCI? ¡ Twelve major requirements covering policy, processes, and technology to protect Credit Card Data. ¡ What is Credit Card Data? ¡ Few Clarificaons ¡ Payment Card Industry (PCI) requires some tasks to be performed by external vendors depending upon merchant level. There is no other way around, unfortunately. ¡ Open Source soluCons do need people. That is why it is almost free but not totally free. 9/10/11 3 What the Auditors Look For? ¡ Is PCI just a checklist? ¡ Are auditors genuinely interested in securing the PCI data? ¡ Does it maer if you use an open source or commercial product to meet PCI requirements? ¡ What if you meet PCI requirements while improving security and spending less money? 9/10/11 4 Is it viable to use Open Source for PCI Compliance? ¡ Is there a real company who uses Open Source soQware to achieve PCI compliance? Is it even possible? ¡ PCI 2.0 focuses more on Risk based approach. ¡ PCI (or any compliance) is boring! Make it interesCng by using Open Source. 9/10/11 5 PCI Biggest Expenses 1. Log Management (Storage and archiving, Monitoring and Alerng) 2. Vulnerability Scanning 3. Network Firewalls and Network Segmentaon 4. Intrusion DetecCon System 5. EncrypCon for data-at-rest 6. File Integrity Monitoring 7. IdenCty Management (Password controls, Two factor for remote access, Role based access) 9/10/11 6 AddiConal PCI
    [Show full text]
  • 5. Zabbix Appliance 5
    2021/07/12 12:09 1/8 5. Zabbix appliance 5. Zabbix appliance As an alternative to setting up manually or reusing existing server for Zabbix, users may download Zabbix appliance. To get started, boot the appliance and point your browser at the IP it has received over DHCP. Booting Zabbix appliance Zabbix appliance versions are based upon the following OpenSUSE versions: Zabbix appliance version OpenSUSE version 2.2.0 12.3 It is available in the following formats: vmdk (VMware/Virtualbox) OVF (Open Virtualisation Format) KVM CD ISO HDD/flash image Preload ISO Xen guest Microsoft VHD Preload USB Zabbix Documentation 2.2 - https://www.zabbix.com/documentation/2.2/ Last update: 2018/09/04 11:39 manual:appliance https://www.zabbix.com/documentation/2.2/manual/appliance It has Zabbix server configured and running on MySQL, as well as frontend available. The appliance has been built using SUSE Studio. 1 Changes to SUSE configuration There are some changed applied to the base OpenSUSE configuration. 1.1 MySQL configuration changes Binary log is disabled; InnoDB is configured to store data for each table in a separate file. 1.2 Using a static IP address By default the appliance uses DHCP to obtain IP address. To specify a static IP address: Log in as root user; Open file /etc/sysconfig/network/ifcfg-eth0 in your favourite editor; Set BOOTPROTO variable to static; Set IPADDR, NETMASK and any other parameters as required for your network; Create file /etc/sysconfig/network/routes. For the default route, use default 192.168.1.1 - - (replacing with your gateway address).
    [Show full text]