NXLOG Community Edition Reference Manual for V2.9.1716 I

Total Page：16

File Type：pdf, Size：1020Kb

Ed. v2.9.1716 NXLOG Community Edition Reference Manual for v2.9.1716 i NXLOG Community Edition Reference Manual for v2.9.1716 Ed. v2.9.1716 Ed. v2.9.1716 NXLOG Community Edition Reference Manual for v2.9.1716 ii Copyright © 2009-2014 NXLog Ltd. Ed. v2.9.1716 NXLOG Community Edition Reference Manual for v2.9.1716 iii Contents 1 Introduction 1 1.1 Overview . .1 1.2 Features . .1 1.2.1 Multiplatform . .1 1.2.2 Modular architecture . .1 1.2.3 Client-server mode . .2 1.2.4 Log message sources and destinations . .2 1.2.5 Importance of security . .2 1.2.6 Scalable multi-threaded architecture . .2 1.2.7 High performance I/O . .2 1.2.8 Message buffering . .2 1.2.9 Prioritized processing . .3 1.2.10 Avoiding lost messages . .3 1.2.11 Apache-style configuration syntax . .3 1.2.12 Built-in config language . .3 1.2.13 Scheduled tasks . .3 1.2.14 Log rotation . .3 1.2.15 Different log message formats . .4 1.2.16 Advanced message processing capabilites . .4 1.2.17 Offline processing mode . .4 1.2.18 Character set and i18n support . .4 2 Installation and quickstart 5 2.1 Microsoft Windows . .5 2.2 GNU/Linux . .6 2.2.1 Installing from DEB packages (Debian, Ubuntu) . .6 2.2.2 Installing from RPM packages (CentOS, RedHat) . .6 2.2.3 Configuring nxlog on GNU/Linux . .6 Ed. v2.9.1716 NXLOG Community Edition Reference Manual for v2.9.1716 iv 3 Architecture and concepts 7 3.1 History . .7 3.2 Concepts . .7 3.3 Architecture . .8 4 Configuration 9 4.1 File inclusion . .9 4.2 Constant and macro definitions . .9 4.3 Global directives . 10 4.4 Modules . 11 4.4.1 Common module directives . 12 4.4.1.1 Module . 12 4.4.1.2 FlowControl . 12 4.4.1.3 Schedule . 12 4.4.1.4 Exec . 13 4.4.1.5 Processors . 14 4.4.1.6 InputType . 14 4.4.1.7 OutputType . 15 4.5 Routes . 16 4.5.1 Priority . 16 4.5.2 Path . 17 5 Language 19 5.1 Types ......................................................... 19 5.2 Expressions . 20 5.2.1 Literals . 20 5.2.2 Fields . 21 5.2.3 Operations . 21 5.2.3.1 Unary operations . 22 5.2.3.2 Binary operations . 22 5.2.4 Functions . 26 5.3 Statements . 27 5.3.1 Assignment . 27 5.3.2 Block . 27 5.3.3 Procedures . 27 5.3.4 If-Else . 27 5.4 Variables . 28 5.5 Statistical counters . 29 5.6 List of available functions and procedures . 30 5.6.1 Functions and procedures exported by core . 30 5.6.1.1 Functions exported by core . 30 5.6.1.2 Procedures exported by core . 35 5.6.2 Functions and procedures exported by modules . 38 Ed. v2.9.1716 NXLOG Community Edition Reference Manual for v2.9.1716 v 6 Modules 39 6.1 Extension modules . 39 6.1.1 CSV (xm_csv) . 39 6.1.1.1 Configuration . 39 6.1.1.1.1 Specifying characters for quote, escape and delimiter . 40 6.1.1.2 Functions and procedures exported by xm_csv . 41 6.1.1.2.1 Functions exported by xm_csv . 41 6.1.1.2.2 Procedures exported by xm_csv . 41 6.1.1.3 Configuration examples . 42 6.1.2 JSON (xm_json) . 42 6.1.2.1 Configuration . ..

174

Copy Link

Recommended publications

Implementation of Centralized Logging and Log Analysis in Cloud Transition
Implementation of Centralized Logging and Log Analysis in Cloud Transition Antti Vainio School of Science Thesis submitted for examination for the degree of Master of Science in Technology. Espoo 3.7.2018 Supervisor Prof. Jukka Suomela Advisor MSc Cyril de Vaumas Copyright ⃝c 2018 Antti Vainio Aalto University, P.O. BOX 11000, 00076 AALTO www.aalto.fi Abstract of the master’s thesis Author Antti Vainio Title Implementation of Centralized Logging and Log Analysis in Cloud Transition Degree programme Computer, Communication and Information Sciences Major Computer Science Code of major SCI3042 Supervisor Prof. Jukka Suomela Advisor MSc Cyril de Vaumas Date 3.7.2018 Number of pages 84 Language English Abstract Centralized logging can be used to collect log data from multiple log files on multiple separate server machines and transmit the data to a single centralized location. Log analysis on top of that can automatically process large amounts of logs for various different purposes including problem detection, troubleshooting, monitoring system performance, identifying security incidents, and understanding user behavior. As the volume of log data is growing when software systems, networks, and services grow in size, the log data located on multiple separate server machines can be difficult to manage. The traditional way of manually inspecting logs hasalso become too labor-intensive and error-prone when large amounts of log data need to be analyzed. Setting up centralized logging and automatic log analysis systems can be used to solve this problem. This thesis explains the concepts of log data, centralized logging, and log analysis, and also takes a look at existing software solutions to implement such a system.
[Show full text]
Log-Management-Tenshi.Pdf
Network Monitoring and Management Log Management Network Startup Resource Center www.ws.nsrc.org These materials are licensed under the Creative Commons Attribution-NonCommercial 4.0 International license (http://creativecommons.org/licenses/by-nc/4.0/) Log Management & Monitoring • Keep your logs in a secure place • Where they can be easily inspected • Watch your log file • They contain important information – Many things happen – Someone needs to review them – It’s not practical to do this manually Log Management & Monitoring On your routers and switches And, on your servers Log Management • Centralize and consolidate log files • Send all log messages from your routers, switches and servers to a single node – a log server. • All network hardware and UNIX/Linux servers can be monitored using some version of syslog (we use either syslog-ng or rsyslog for this workshop). • Windows can, also, use syslog with extra tools. • Save a copy of the logs locally, but, also, save them to a central log server. Syslog Basics Uses UDP protocol, port 514 Syslog messages have two attributes (in addition to the message itself): Facility Level Auth Security | Emergency (0) Authpriv User | Alert (1) Console Syslog | Critical (2) Cron UUCP | Error (3) Daemon Mail | Warning (4) Ftp Ntp | Notice (5) Kern News | Info (6) Lpr | Debug (7) Local0 ...Local7 | Centralized Logging Configuring Centralized Logging Cisco hardware – At a minimum: logging ip.of.logging.host Unix and Linux nodes – In syslogd.conf, or in rsyslog.conf, add: *.* @ip.of.log.host – Restart syslogd, rsyslog or syslog-ng Other equipment have similar options – Options to control facility and level Receiving Messages – syslog-ng • Identify the facility that the equipment is going to use to send its messages.
[Show full text]
Nxlog Community Edition Reference Manual
NXLog Community Edition Reference Manual NXLog Ltd. Version 2.10.2150, November 2018 Table of Contents 1. Man Pages. 1 1.1. nxlog(8) . 1 1.2. nxlog-processor(8) . 3 2. Configuration . 5 2.1. General Directives . 5 2.2. Global Directives . 6 2.3. Common Module Directives. 7 2.4. Route Directives. 12 3. Language. 15 3.1. Types . 15 3.2. Expressions. 15 3.3. Statements . 24 3.4. Variables . 26 3.5. Statistical Counters . 26 3.6. Functions. 28 3.7. Procedures . 31 4. Extension Modules . 34 4.1. Character Set Conversion (xm_charconv) . 34 4.2. Delimiter-Separated Values (xm_csv) . 35 4.3. External Programs (xm_exec) . 39 4.4. File Operations (xm_fileop) . 41 4.5. GELF (xm_gelf) . 44 4.6. JSON (xm_json). 48 4.7. Key-Value Pairs (xm_kvp) . 51 4.8. Multi-Line Parser (xm_multiline) . 60 4.9. Perl (xm_perl) . 68 4.10. Syslog (xm_syslog) . 71 4.11. WTMP (xm_wtmp) . 83 4.12. XML (xm_xml). 84 5. Input Modules . 88 5.1. Fields . 88 5.2. DBI (im_dbi) . 88 5.3. External Programs (im_exec) . 89 5.4. Files (im_file) . 90 5.5. Internal (im_internal). 93 5.6. Kernel (im_kernel) . 95 5.7. Mark (im_mark) . 96 5.8. EventLog for Windows XP/2000/2003 (im_mseventlog). 97 5.9. EventLog for Windows 2008/Vista and Later (im_msvistalog) . 100 5.10. Null (im_null) . 105 5.11. TLS/SSL (im_ssl) . 105 5.12. TCP (im_tcp) . 107 5.13. UDP (im_udp) . 108 5.14. Unix Domain Sockets (im_uds) . 109 6. Processor Modules . 111 6.1. Blocker (pm_blocker) . ..
[Show full text]
Fedora 16 System Administrator's Guide
Fedora 16 System Administrator's Guide Deployment, Configuration, and Administration of Fedora 16 Jaromír Hradílek Douglas Silas Martin Prpič Eva Kopalová Eliška Slobodová Tomáš Čapek Petr Kovář Miroslav Svoboda System Administrator's Guide John Ha David O'Brien Michael Hideo Don Domingo Fedora 16 System Administrator's Guide Deployment, Configuration, and Administration of Fedora 16 Edition 1 Author Jaromír Hradílek [email protected] Author Douglas Silas [email protected] Author Martin Prpič [email protected] Author Eva Kopalová [email protected] Author Eliška Slobodová [email protected] Author Tomáš Čapek [email protected] Author Petr Kovář [email protected] Author Miroslav Svoboda [email protected] Author John Ha Author David O'Brien Author Michael Hideo Author Don Domingo Copyright © 2011 Red Hat, Inc. and others. The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. The original authors of this document, and Red Hat, designate the Fedora Project as the "Attribution Party" for purposes of CC-BY-SA. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
[Show full text]
Ubuntu Server Guide Basic Installation Preparing to Install
Ubuntu Server Guide Welcome to the Ubuntu Server Guide! This site includes information on using Ubuntu Server for the latest LTS release, Ubuntu 20.04 LTS (Focal Fossa). For an offline version as well as versions for previous releases see below. Improving the Documentation If you find any errors or have suggestions for improvements to pages, please use the link at thebottomof each topic titled: “Help improve this document in the forum.” This link will take you to the Server Discourse forum for the specific page you are viewing. There you can share your comments or let us know aboutbugs with any page. PDFs and Previous Releases Below are links to the previous Ubuntu Server release server guides as well as an offline copy of the current version of this site: Ubuntu 20.04 LTS (Focal Fossa): PDF Ubuntu 18.04 LTS (Bionic Beaver): Web and PDF Ubuntu 16.04 LTS (Xenial Xerus): Web and PDF Support There are a couple of different ways that the Ubuntu Server edition is supported: commercial support and community support. The main commercial support (and development funding) is available from Canonical, Ltd. They supply reasonably- priced support contracts on a per desktop or per-server basis. For more information see the Ubuntu Advantage page. Community support is also provided by dedicated individuals and companies that wish to make Ubuntu the best distribution possible. Support is provided through multiple mailing lists, IRC channels, forums, blogs, wikis, etc. The large amount of information available can be overwhelming, but a good search engine query can usually provide an answer to your questions.
[Show full text]
Facing the Challenge(S) of Windows Logs Collection to Leverage Valuable Iocs
Facing the challenge(s) of Windows logs collection to leverage valuable IOCs . Michel de Crevoisier Security Analyst, Radar Cyber Security 15.10.2019, Berne © RadarServices // Classification: Public The five challenges © RadarServices // Classification: Public #1 High diversity of log sources Server Microsoft 3rd party Built-in roles software software Advanced Threat ADFS Application Analytics (ATA) Ivanti software Certification authority Exchange PowerShell Kaspersky DHCP server Skype Security DNS server SQL Server Veeam Backup System IIS web server SYSMON […] […] NPS Radius Defender © RadarServices // Classification: Public 3 #2 Different log extensions EVTX ETL TXT (standard Windows logs (analytical logs, like DNS (IIS, NPS, DHCP, in XML format) Server or PowerShell) PowerShell Transcript, former DNS logs) © RadarServices // Classification: Public 4 #3 Multiple architectural approaches Access method / Protocol (MS-EVEN6, RPC, WMI,…) Push vs Pull Agent vs Agentless Intermediate collector VS Direct sending to receiver Central file store vs Shared folder Managed agent VS Unmanaged agent © RadarServices // Classification: Public 5 #4 Disabled and restrictive event logs • Protected users (if configured, on DCs only) Valuable event • LSA (Local Security Authority) logs disabled • IIS web server • DNS client Event logs with • SMB server restrictive • SMB client access • IIS web server © RadarServices // Classification: Public 6 6 #5 Operational constraints Security Data exchange Performance Configuration Environment • Avoid usage of • Data
[Show full text]
Using Nxlog with Elasticsearch and Kibana I
Using NXLog with Elasticsearch and Kibana i Using NXLog with Elasticsearch and Kibana Using NXLog with Elasticsearch and Kibana ii Contents 1 Setting up Elasticsearch and Kibana 1 1.1 Installing Elasticsearch . .1 1.2 Installing Kibana . .1 2 Loading data into Elasticsearch with NXLog2 2.1 Loading data with om_elasticsearch . .2 2.2 Loading data with om_http . .4 2.3 Using Logstash . .5 Using NXLog with Elasticsearch and Kibana 1 / 6 Elasticsearch coupled with the Kibana frontend has become quite popular recently as a low-cost centralized log monitoring solution. This is commonly referred to as the ELK stack comprised of Elasticsearch, Logstash and Kibana. While Logstash is a great piece of software it has some disadvantages compared to NXLog: • Logstash is written in ruby and requires Java to run. Besides being a lot more hungry on system resources, many system administrators would rather not take the hassle of deploying the Java runtime onto their production servers and needing take care of the Java security updates. • The Eventlog plugin in Logstash pulls the eventlog data through the Windows WMI interface which incurs a signiﬁcant performance penalty. NXLog hooks directly into the Windows EventLog API natively and can collect logs from our highly loaded Domain Controllers also. • It’s just one more piece of software to take care about. NXLog is a small and efﬁcient log collector that can be set up to securely and reliably centralize event data from Windows and Unix platforms. As such, NXLog is recommended by many ELK users as the log collector of choice for Windows and Linux.
[Show full text]
An Exploratory Semantic Analysis of Logging Questions
Received: Added at production Revised: Added at production Accepted: Added at production DOI: xxx/xxxx An Exploratory Semantic Analysis of Logging Questions Harshit Gujral*1 | Sangeeta Lal2 | Heng Li3 1Department of Computer Science Engineering and Information Abstract Technology, Jaypee Institute of Information Technology, Noida, India. Logging is an integral part of software development. Software practitioners often face issues Email: [email protected] in software logging, and they post these issues on Q&A websites to take suggestions from 2Lecture Data Science, School of Computing and Mathematics, Keele the experts. In this study, we perform a three-level empirical analysis of logging questions University, Keele, United Kingdom. posted on six popular technical Q&A websites, namely Stack Overflow (SO), Serverfault Email: [email protected] 3Department of Computer and Software (SF), Superuser (SU), Database Administrators (DB), Software Engineering (SE), and Engineering, Polytechnique Montréal, Android Enthusiasts (AE). The findings show that logging issues are prevalent across var- Montréal, Canada. Email: [email protected] ious domains, e.g., database, networks, and mobile computing, and software practitioners from different domains face different logging issues. The semantic analysis of logging ques- Correspondence *Corresponding author. tions using Latent Dirichlet Allocation (LDA) reveals trends of several existing and new logging topics, such as logging conversion pattern, android device logging, and database logging. In addition, we observe specific logging topics for each website: DB (Log shipping, Log file growing/shrinking), SU (Event Log, Syslog configuration), SF (Log analysis, Sys- log configuration), AE (App Install, Usage tracking), SE (Client server logging, Exception logging), and SO (Log file creation/deletion, Android Emulator Logging, Logger class of log4j).
[Show full text]
Linux and Open Source for (Almost) Zero Cost PCI Compliance
Linux and Open Source for (Almost) Zero Cost PCI Compliance Rafeeq Rehman 2 Some Introductory Notes ¡ Payment Card Industry (PCI) standard is not a government regulaon. ¡ Who needs to comply with PCI? ¡ Twelve major requirements covering policy, processes, and technology to protect Credit Card Data. ¡ What is Credit Card Data? ¡ Few Clariﬁcaons ¡ Payment Card Industry (PCI) requires some tasks to be performed by external vendors depending upon merchant level. There is no other way around, unfortunately. ¡ Open Source soluCons do need people. That is why it is almost free but not totally free. 9/10/11 3 What the Auditors Look For? ¡ Is PCI just a checklist? ¡ Are auditors genuinely interested in securing the PCI data? ¡ Does it maer if you use an open source or commercial product to meet PCI requirements? ¡ What if you meet PCI requirements while improving security and spending less money? 9/10/11 4 Is it viable to use Open Source for PCI Compliance? ¡ Is there a real company who uses Open Source soQware to achieve PCI compliance? Is it even possible? ¡ PCI 2.0 focuses more on Risk based approach. ¡ PCI (or any compliance) is boring! Make it interesCng by using Open Source. 9/10/11 5 PCI Biggest Expenses 1. Log Management (Storage and archiving, Monitoring and Alerng) 2. Vulnerability Scanning 3. Network Firewalls and Network Segmentaon 4. Intrusion DetecCon System 5. EncrypCon for data-at-rest 6. File Integrity Monitoring 7. IdenCty Management (Password controls, Two factor for remote access, Role based access) 9/10/11 6 AddiConal PCI
[Show full text]
Red Hat Openstack Platform 16.1 Logging, Monitoring, and Troubleshooting Guide
Red Hat OpenStack Platform 16.1 Logging, Monitoring, and Troubleshooting Guide An In-Depth Guide to OpenStack Logging, Monitoring, and Troubleshooting Last Updated: 2021-05-13 Red Hat OpenStack Platform 16.1 Logging, Monitoring, and Troubleshooting Guide An In-Depth Guide to OpenStack Logging, Monitoring, and Troubleshooting OpenStack Team [email protected] Legal Notice Copyright © 2021 Red Hat, Inc. The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/ . In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux ® is the registered trademark of Linus Torvalds in the United States and other countries. Java ® is a registered trademark of Oracle and/or its affiliates. XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
[Show full text]
Analysis and Comparison of Log Shipment Solutions at AWS S3 for Windows 10
Department of Information Engineering and Computer Science Master in Computer Science 2020-2021 Final thesis Analysis and comparison of Log Shipment solutions at AWS S3 for Windows 10. Francisco Manuel Colmenar Lamas i SUMMARY A fundamental aspect that every company must address to start building its security infrastructure is visibil- ity. Increasing a company’s visibility raises the quality and effectiveness of all other existing security solutions. The objective was to implement an endpoint log forwarding solution for the Windows 10 devices of the com- pany About You. To accomplish the objective, several concepts and knowledge in the scope of log management solutions were studied, as well as the use of AmazonWeb Services (AWS) dedicated to these activities. After analyzing the different solutions, Kinesis Windows Agent was chosen to implement the endpoint log shipment solution. Because it provides a serverless architecture, where the agent sends logs from the endpoints to Kinesis Firehose. In addition, it does not require any heavy-weight dependencies and its configuration is straightforward. Also, since Kinesis Firehose is an AWS managed service, there is no need to handle the scaling or fault tolerance issues common in a client-server architecture and it integrates seamlessly with S3. Regarding the implementation, the code for the installation and maintenance of the Kinesis Windows Agent was mainly developed in Powershell scripts triggered remotely using Ninjarmm. And the AWS infrastructure code required for this project was developed using Terraform. In addition, through Gitlab’s CI/CD pipeline, AWS resources are automatically updated if the code is modified. As a conclusion, the deployment of the Kinesis agent for Windows on all employee Windows devices was a success.
[Show full text]
Rsyslog Doc Documentation Release 8.18.0.Master
Rsyslog Doc Documentation Release 8.18.0.master foo bar March 09, 2016 Contents 1 Manual 3 2 Reference 275 3 Sponsors and Community 317 4 Related Links 319 i ii Rsyslog Doc Documentation, Release 8.18.0.master Rsyslog is a rocket-fast system for log processing. It offers high-performance, great security features and a modular design. While it started as a regular syslogd, rsyslog has evolved into a kind of swiss army knife of logging, being able to • accept inputs from a wide variety of sources, • transform them, • and output the results to diverse destinations. Rsyslog has a strong enterprise focus but also scales down to small systems. It supports, among others, MySQL, Post- greSQL, failover log destinations, ElasticSearch, syslog/tcp transport, fine grain output format control, high precision timestamps, queued operations and the ability to filter on any message part. Contents 1 Rsyslog Doc Documentation, Release 8.18.0.master 2 Contents CHAPTER 1 Manual 1.1 Configuration Rsyslogd is configured via the rsyslog.conf file, typically found in /etc. By default, rsyslogd reads the file /etc/rsyslog.conf. This can be changed by a command line option. Note that configurations can be built interactively via the online rsyslog configuration builder tool. 1.1.1 Basic Structure This section describes how rsyslog configuration basically works. Think of rsyslog as a big logging and event pro- cessing toolset. It can be considered a framework with some basic processing that is fixed in the way data flows, but is highly customizable in the details of this message flow.
[Show full text]