Centralized log management for complex computer networks Marcus Hanikat KTH ROYAL INSTITUTE OF TECHNOLOGY ELEKTROTEKNIK OCH DATAVETENSKAP Abstract In modern computer networks log messages produced on different devices throughout the network is collected and analyzed. The data from these log messages gives the network administrators an overview of the networks operation, allows them to detect problems with the network and block security breaches. In this thesis several different centralized log management systems are analyzed and evaluated to see if they match the requirements for security, performance and cost which was established. These requirements are designed to meet the stakeholder’s requirements of log management and allow for scaling along with the growth of their network. To prove that the selected system meets the requirements, a small-scale implementation of the system will be created as a “proof of concept”. The conclusion reached was that the best solution for the centralized log management system was the ELK Stack system which is based upon the three open source software Elasticsearch, Logstash and Kibana. In the small-scale implementation of the ELK Stack system it was shown that it meets all the requirements placed on the system. The goal of this thesis is to help develop a greater understanding of some well-known centralized log management systems and why the usage of them is important for computer networks. This will be done by describing, comparing and evaluating some of the functionalities of the selected centralized log management systems. This thesis will also be able to provide people and entities with guidance and recommendations for the choice and implementation of a centralized log management system. Keywords Logging ; Log management ; Computer networks ; Centralization ; Security Abstrakt I moderna datornätverk så produceras loggar på olika enheter i nätverket för att sedan samlas in och analyseras. Den data som finns i dessa loggar hjälper nätverksadministratörerna att få en överblick av hur nätverket fungerar, tillåter dem att upptäcka problem i nätverket samt blockera säkerhetshål. I detta projekt så analyseras flertalet relevanta system för centraliserad loggning utifrån de krav för säkerhet, prestanda och kostnad som är uppsatta. Dessa krav är uppsatta för att möta intressentens krav på loghantering och även tillåta för skalning jämsides med tillväxten av deras nätverk. För att bevisa att det valda systemet även fyller de uppsatta kraven så upprättades även en småskalig implementation av det valda systemet som ett ”proof of concept”. Slutsatsen som drogs var att det bästa centraliserade loggningssystemet utifrån de krav som ställs var ELK Stack som är baserat på tre olika mjukvarusystem med öppen källkod som heter Elasticsearch, Logstash och Kibana. I den småskaliga implementationen av detta system så påvisades även att det valda loggningssystemet uppnår samtliga krav som ställdes på systemet. Målet med detta projekt är att hjälpa till att utveckla kunskapen kring några välkända system för centraliserad loggning och varför användning av dessa är av stor betydelse för datornätverk. Detta kommer att göras genom att beskriva, jämföra och utvärdera de utvalda systemen för centraliserad loggning. Projektet kan även att hjälpa personer och organisationer med vägledning och rekommendationer inför val och implementation av ett centraliserat loggningssystem. Nyckelord Loggning ; Log hantering; Datornätverk ; Centralisering ; Säkerhet Table of contents 1 Introduction ............................................................................................................. 1 1.1 Background .................................................................................................................... 2 1.2 Problem ............................................................................................................................ 3 1.3 Purpose ............................................................................................................................. 4 1.4 Goals ................................................................................................................................... 4 1.4.1 Benefits for society, ethics and sustainability .......................................................... 5 1.5 Research Methodology ............................................................................................ 5 1.6 Stakeholder .................................................................................................................... 7 1.7 Delimitations ................................................................................................................ 7 1.8 Disposition ..................................................................................................................... 8 2 Log management ................................................................................................. 10 2.1 Log source groups ................................................................................................... 10 2.2 Log severity levels ................................................................................................... 11 2.3 Log processing pipeline ....................................................................................... 12 2.4 Centralized log management structures ................................................... 13 2.5 Logging policy ............................................................................................................ 16 2.6 Related work .............................................................................................................. 16 3 Research methodologies and methods ................................................... 19 3.1 System development methodologies ........................................................... 19 3.2 Research phases ....................................................................................................... 20 3.3 Data collection........................................................................................................... 20 3.4 Setting up system requirements .................................................................... 21 3.5 System selection process .................................................................................... 21 4 System requirements ........................................................................................ 24 4.1 Logging policy ............................................................................................................ 24 4.1.1 Log generation....................................................................................................................... 24 4.1.2 Log transmission .................................................................................................................. 25 4.1.3 Log storage and disposal .................................................................................................. 26 4.1.4 Log analysis ............................................................................................................................ 27 4.2 Requirements ............................................................................................................ 28 5 System selection ................................................................................................... 30 5.1 Splunk ............................................................................................................................ 30 5.2 ELK Stack ..................................................................................................................... 32 5.3 Graylog ........................................................................................................................... 37 5.4 System selection ....................................................................................................... 40 5.4.1 Stack Overflow ...................................................................................................................... 40 5.4.2 Cost of implementation .................................................................................................... 41 5.4.3 Scalability ................................................................................................................................. 42 5.4.4 Open source ............................................................................................................................ 44 5.4.5 Criterion summarization.................................................................................................. 44 6 System implementation ................................................................................... 46 6.1 Network topology .................................................................................................... 46 6.2 Parsing log data ........................................................................................................ 49 6.3 Encryption, Authentication and Integrity ................................................ 51 i 6.4 Data persistency and availability ................................................................... 53 6.5 Scaling the system ................................................................................................... 55 6.6 Generating alerts and X-Pack .......................................................................... 56 6.7 Kibana visualization .............................................................................................