DOCSLIB.ORG

Centralized Log Management for Complex Computer Networks

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Centralized Log Management for Complex Computer Networks Centralized log management for complex computer networks Marcus Hanikat KTH ROYAL INSTITUTE OF TECHNOLOGY ELEKTROTEKNIK OCH DATAVETENSKAP Abstract In modern computer networks log messages produced on different devices throughout the network is collected and analyzed. The data from these log messages gives the network administrators an overview of the networks operation, allows them to detect problems with the network and block security breaches. In this thesis several different centralized log management systems are analyzed and evaluated to see if they match the requirements for security, performance and cost which was established. These requirements are designed to meet the stakeholder’s requirements of log management and allow for scaling along with the growth of their network. To prove that the selected system meets the requirements, a small-scale implementation of the system will be created as a “proof of concept”. The conclusion reached was that the best solution for the centralized log management system was the ELK Stack system which is based upon the three open source software Elasticsearch, Logstash and Kibana. In the small-scale implementation of the ELK Stack system it was shown that it meets all the requirements placed on the system. The goal of this thesis is to help develop a greater understanding of some well-known centralized log management systems and why the usage of them is important for computer networks. This will be done by describing, comparing and evaluating some of the functionalities of the selected centralized log management systems. This thesis will also be able to provide people and entities with guidance and recommendations for the choice and implementation of a centralized log management system. Keywords Logging ; Log management ; Computer networks ; Centralization ; Security Abstrakt I moderna datornätverk så produceras loggar på olika enheter i nätverket för att sedan samlas in och analyseras. Den data som finns i dessa loggar hjälper nätverksadministratörerna att få en överblick av hur nätverket fungerar, tillåter dem att upptäcka problem i nätverket samt blockera säkerhetshål. I detta projekt så analyseras flertalet relevanta system för centraliserad loggning utifrån de krav för säkerhet, prestanda och kostnad som är uppsatta. Dessa krav är uppsatta för att möta intressentens krav på loghantering och även tillåta för skalning jämsides med tillväxten av deras nätverk. För att bevisa att det valda systemet även fyller de uppsatta kraven så upprättades även en småskalig implementation av det valda systemet som ett ”proof of concept”. Slutsatsen som drogs var att det bästa centraliserade loggningssystemet utifrån de krav som ställs var ELK Stack som är baserat på tre olika mjukvarusystem med öppen källkod som heter Elasticsearch, Logstash och Kibana. I den småskaliga implementationen av detta system så påvisades även att det valda loggningssystemet uppnår samtliga krav som ställdes på systemet. Målet med detta projekt är att hjälpa till att utveckla kunskapen kring några välkända system för centraliserad loggning och varför användning av dessa är av stor betydelse för datornätverk. Detta kommer att göras genom att beskriva, jämföra och utvärdera de utvalda systemen för centraliserad loggning. Projektet kan även att hjälpa personer och organisationer med vägledning och rekommendationer inför val och implementation av ett centraliserat loggningssystem. Nyckelord Loggning ; Log hantering; Datornätverk ; Centralisering ; Säkerhet Table of contents 1 Introduction ............................................................................................................. 1 1.1 Background .................................................................................................................... 2 1.2 Problem ............................................................................................................................ 3 1.3 Purpose ............................................................................................................................. 4 1.4 Goals ................................................................................................................................... 4 1.4.1 Benefits for society, ethics and sustainability .......................................................... 5 1.5 Research Methodology ............................................................................................ 5 1.6 Stakeholder .................................................................................................................... 7 1.7 Delimitations ................................................................................................................ 7 1.8 Disposition ..................................................................................................................... 8 2 Log management ................................................................................................. 10 2.1 Log source groups ................................................................................................... 10 2.2 Log severity levels ................................................................................................... 11 2.3 Log processing pipeline ....................................................................................... 12 2.4 Centralized log management structures ................................................... 13 2.5 Logging policy ............................................................................................................ 16 2.6 Related work .............................................................................................................. 16 3 Research methodologies and methods ................................................... 19 3.1 System development methodologies ........................................................... 19 3.2 Research phases ....................................................................................................... 20 3.3 Data collection........................................................................................................... 20 3.4 Setting up system requirements .................................................................... 21 3.5 System selection process .................................................................................... 21 4 System requirements ........................................................................................ 24 4.1 Logging policy ............................................................................................................ 24 4.1.1 Log generation....................................................................................................................... 24 4.1.2 Log transmission .................................................................................................................. 25 4.1.3 Log storage and disposal .................................................................................................. 26 4.1.4 Log analysis ............................................................................................................................ 27 4.2 Requirements ............................................................................................................ 28 5 System selection ................................................................................................... 30 5.1 Splunk ............................................................................................................................ 30 5.2 ELK Stack ..................................................................................................................... 32 5.3 Graylog ........................................................................................................................... 37 5.4 System selection ....................................................................................................... 40 5.4.1 Stack Overflow ...................................................................................................................... 40 5.4.2 Cost of implementation .................................................................................................... 41 5.4.3 Scalability ................................................................................................................................. 42 5.4.4 Open source ............................................................................................................................ 44 5.4.5 Criterion summarization.................................................................................................. 44 6 System implementation ................................................................................... 46 6.1 Network topology .................................................................................................... 46 6.2 Parsing log data ........................................................................................................ 49 6.3 Encryption, Authentication and Integrity ................................................ 51 i 6.4 Data persistency and availability ................................................................... 53 6.5 Scaling the system ................................................................................................... 55 6.6 Generating alerts and X-Pack .......................................................................... 56 6.7 Kibana visualization .............................................................................................
Load more

Recommended publications
  • Guide to Computer Security Log Management
    Special Publication 800-92 Guide to Computer Security Log Management Recommendations of the National Institute of Standards and Technology Karen Kent Murugiah Souppaya GUIDE TO COMPUTER SECURITY LOG MANAGEMENT Reports on Computer Systems Technology The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analysis to advance the development and productive use of information technology. ITL’s responsibilities include the development of technical, physical, administrative, and management standards and guidelines for the cost-effective security and privacy of sensitive unclassified information in Federal computer systems. This Special Publication 800-series reports on ITL’s research, guidance, and outreach efforts in computer security and its collaborative activities with industry, government, and academic organizations. National Institute of Standards and Technology Special Publication 800-92 Natl. Inst. Stand. Technol. Spec. Publ. 800-92, 72 pages (September 2006) Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended
    [Show full text]
  • Guide to Computer Security Log Management
    Special Publication 800-92 Guide to Computer Security Log Management Recommendations of the National Institute of Standards and Technology Karen Kent Murugiah Souppaya NIST Special Publication 800-92 Guide to Computer Security Log Management Recommendations of the National Institute of Standards and Technology Karen Kent Murugiah Souppaya C O M P U T E R S E C U R I T Y Computer Security Division Information Technology Laboratory National Institute of Standards and Technology Gaithersburg, MD 20899-8930 September 2006 U.S. Department of Commerce Carlos M. Gutierrez, Secretary Technology Administration Robert C. Cresanti, Under Secretary of Commerce for Technology National Institute of Standards and Technology William Jeffrey, Director GUIDE TO COMPUTER SECURITY LOG MANAGEMENT Reports on Computer Systems Technology The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analysis to advance the development and productive use of information technology. ITL’s responsibilities include the development of technical, physical, administrative, and management standards and guidelines for the cost-effective security and privacy of sensitive unclassified information in Federal computer systems. This Special Publication 800-series reports on ITL’s research, guidance, and outreach efforts in computer security and its collaborative activities with industry, government, and academic organizations. National Institute of Standards and Technology Special Publication 800-92 Natl. Inst. Stand. Technol. Spec. Pub l. 800-92, 72 pages (September 2006) Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experim ental procedure or concept adequately.
    [Show full text]
  • Analysis of Security Information and Event Management (Siem) Evasion and Detection Methods
    TALLINN UNIVERSITY OF TECHNOLOGY Faculty of Information Technology Department of Computer Engineering ITC70LT Seyed Morteza Zeinali (IVCM131121) ANALYSIS OF SECURITY INFORMATION AND EVENT MANAGEMENT (SIEM) EVASION AND DETECTION METHODS (Master Thesis) Supervisor: Bernhards Blumbergs Master’s Degree Ph.D. Student at Tallinn University of Technology Tallinn 2016 Declaration I hereby certify that I am the sole author of this thesis. All the used materials, references to the literature and the work of others have been referred to. This thesis has not been presented for examination anywhere else. .................................. .................................. (Author's signature) (Date) 2 Abstract: Security Information and Event Management (SIEM) systems have become today a crucial and essential component of complex enterprise networks. They typically aggregate and correlate incidents from different systems and platforms, and carry out a rule-based analysis to detect advanced threats. The latest reports show that in spite of the fact SIEMs are significantly efficient, but there are still shortcomings and evasion methods that can compromise the integrity of data and forge the data stored and need to improve over prior solutions. This paper evaluates and analyze the SIEM evasion detections, SIEM evasion methods, expresses approaches and the tools that evade security appliances. An attack simulation experiment is performed using multiple Advanced Evasion Techniques (AETs) to demonstrate the capabilities of SIEM in detecting any suspicious behaviour of event logs and alerting them in near real-time. The tested SIEM was able to collect, filter, normalize, correlate, alert, and report network attacks within minutes after attack incidents. Keywords: Evasion Detection, Event Management, Security Information, Evasion Techniques, SIEM, Outlier Detection, Information Management, Advanced Threats, AET, incident, Anomaly Behaviour.
    [Show full text]
  • Regulation for the Digital Logs Management in Public Administration
    PRIME MINISTER’S OFFICE NATIONAL AGENCY FOR CYBER SECURITY (ALCIRT) REGULATION FOR THE DIGITAL LOGS MANAGEMENT IN PUBLIC ADMINISTRATION Approved with Order no. 109 date 10.06. 2016 of the Director of the National Agency for Cyber Security (ALCIRT). REGULATION FOR THE DIGITAL LOGS MANAGEMENT IN PUBLIC ADMINISTRATION Content 1. Introduction ....................................................................................................................................... 4 2. Purpose .............................................................................................................................................. 4 3. Definitions .......................................................................................................................................... 5 4. General ............................................................................................................................................... 6 5. Activities for which logs will be held .................................................................................................. 7 5.1 Log elements ..................................................................................................................................... 7 5.2 Logs Management Infrastructure and Tasks of Responsible Staff for Log Management ................ 8 5.2.1 Log Management Infrastructure ................................................................................................... 8 Depending on the resources and specifics of the institution, log infrastructure
    [Show full text]
  • Cyber Security Monitoring and Logging Guide
    Cyber Security Monitoring and Logging Guide Version 1 Cyber Security Monitoring and Logging Guide Published by: CREST Tel: 0845 686-5542 Email: [email protected] Web: http://www.crest-approved.org Principal Author Principal reviewer Jason Creasey, Ian Glover, President, CREST Managing Director, Jerakano Limited DTP notes For ease of reference, the following DTP devices have been used throughout the Guide. Acknowledgements CREST would like to extend its special thanks to those CREST member organisations and third parties who took part in interviews, participated in the workshop and completed questionnaires. Warning This Guide has been produced with care and to the best of our ability. However, CREST accepts no responsibility for any problems or incidents arising from its use. A Good Tip ! A Timely Warning Quotes are presented in a box like this. © Copyright 2015. All rights reserved. CREST (GB). 3 Cyber Security Monitoring and Logging Guide Contents Part 1 - Introduction and overview • About this Guide .........................................................................................................................................6 • Audience .....................................................................................................................................................6 • Purpose and scope ......................................................................................................................................7 • A practical solution ......................................................................................................................................7
    [Show full text]