BSD-based Systems Monitoring with Icinga2 and OpenSSH
BSD-based Systems Monitoring with Icinga2 and OpenSSH
Benedict Reuschling [email protected]
BSDCan 2019 Tutorial
1 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH
Monitoring
When we talk about monitoring, we talk about collecting valuable information from target systems to help system administrators and others stay informed about their ever-increasing hardware and software landscape. Problems can be detected early and proactively, without little or no visible impact to the users of these systems. It is also a tool for decision-makers about how well service-levels are provided, outdated systems and services that could be replaced, as well as new systems to be introduced and monitored the same way as done on previous systems. Typically, monitoring deals with the following three aspects: Availability - Is the host or service available on the network? Metrics - Data collected from hosts and services Logs - Messages (errors, warnings) written to log files from hosts and services We’ll cover the first two aspects in these slides. The third one warrants its own due to size and complexity of the topic. Icinga is used to demonstrate one of many monitoring solutions due to its programmability, flexibility, and scalability with the number of hosts and services to monitor.
2 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Introduction to Icinga
Overview
1 Introduction to Icinga
2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup
3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks
4 Restarting Services Automatically
3 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Introduction to Icinga
Introduction to Icinga2
In this tutorial, we’ll take a closer look at using and configuring the unofficial Nagios replacement called Icinga2 to cover our monitoring needs. Icinga is a modern system that builds on many of the concepts that made Nagios great, but adds their own, including a modern web interface called Icingaweb2. Plugins from Nagios can be used in Icinga without modifications and there is a whole ecosystem of new plugins available for Icinga. Icinga was developed due to lack of response from the Nagios authors to react to bugs and feature requests. The Icinga project was created to integrate those and has since rewritten many of the backend components, while still maintaining compatibility to Nagios plugins as much as possible. Icinga offers its own ecosystem of plugins and there is an active open source community that develops the core Icinga system and creates new plugins (https://www.monitoring-plugins.org). There is also training, consulting, and support available for large and complex Icinga2 installations.
4 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System
Overview
1 Introduction to Icinga
2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup
3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks
4 Restarting Services Automatically
5 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System
Installing Required Software Packages
Icinga2 and its Icingaweb2 graphical component is a classic LAMP stack, but can also use of some other components. In this tutorial, we’ll use Nginx as the webserver and PostgreSQL as database backend. The port is still using PostgreSQL 9.5 at the time of this writing. Newer versions of PHP should either be pulled in as dependencies automatically or can be selected based on your own experiences. There is no need to use the latest versions and teh monitoring system will run just fine with older versions. We’ll install the following packages (the rest is pulled in as dependencies): • pkg install icinga2 icingaweb2 postgresql95-server nginx ImageMagick-nox11 php72-pecl-imagick The ImageMagick ports are required for by Icingaweb2 during setup and are used for report generation. If that part is not used, omit this port.
6 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System
Adding Services to /etc/rc.conf
We need to start three components of our monitoring system: 1 The backing PostgreSQL database 2 The Icinga2 monitoring core component 3 The Nginx webserver with fastCGI support for Icingaweb2 Using sysrc, this is not difficult:
sysrc icinga2_enable=yes sysrc postgresql_enable=yes sysrc nginx_enable=yes Don’t start these services just yet, we have a bit of configuration ahead of us! But before we forget, we should rotate our log files:
# cp /usr/local/share/examples/icinga2/newsyslog/icinga2 /etc/newsyslog.conf.d/ We’ll start our configuration with the database setup.
7 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Database Setup Overview
1 Introduction to Icinga
2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup
3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks
4 Restarting Services Automatically
8 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Database Setup Database Setup
First of all, when running on OpenZFS, we’ll create a dataset for PostgreSQL and set a mountpoint. Lastly, we’ll change ownership to the pgsql user account that will run the service later.
# zfs create -o mountpoint=/usr/local/pgsql/data sys/pgdata # zfs set recordsize=8k sys/pgdata # zfs set logbias=throughput sys/pgdata # zfs set redundant_metadata=most sys/pgdata # zfs set primarycache=metadata sys/pgdata # chown pgsql:pgsql /usr/local/pgsql/data The ZFS dataset settings are typical for any kind of PostgreSQL database. There are no heavy queries to the database, but Icinga2 still needs to store a lot of information about monitoring objects, it’s internal state, user accounts, etc.
9 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Database Setup Initializing the Database Cluster We’ll switch to the pgsql user account and initialize the database cluster.
# su pgsql $ cd $ initdb -D ./data --no-locale --encoding=utf-8 --lc-collate=C $ pg_ctl start -D ./data After we started the database cluster, we’ll create a user account in the database for Icinga and set a password. We need this later when setting up the icinga tables and Icingaweb2. Then, we set up a database with the same name to hold the tables.
$ createuser -dPrs icinga Password : Re-type password: $ createdb -O icinga -E UTF8 icinga Then, we create local access entries for the Icinga user in pg_hba.conf. Make sure not to change the trust parts into ident as this will not work.
local icinga icinga md5 host icinga icinga 127.0.0.1/32 md5 host icinga icinga ::1/128 md5
10 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Database Setup Importing the Icinga2-Schema
The database tables and indexes that Icinga2 needs to store various information are stored in /usr/local/share/icinga2-ido-pgsql/schema/pgsql.sql as a schema. $ psql -U icinga -d icinga < /usr/local/share/icinga2-ido-pgsql/schema/pgsql.sql Future versions of Icinga2 might require schema upgrades when updating to a newer version. The upgrade instructions will tell you when and how to perform the schema updates. Intermediate upgrades need to be applied as well, but they are all timestamped and can be imported in the same way as described here for the initial schema. The upgrades are located in /usr/local/share/icinga2-ido-pgsql/schema/upgrade.
11 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Database Setup Configuring the IDO database The IDO (Icinga Data Output) will store information about our monitoring instance in the PostgreSQL database. We need to store our database connection credentials in the file /usr/local/etc/icinga2/features-enabled/ido-pgsql.conf with the following content:
object IdoPgsqlConnection "ido-pgsql" { user = "icinga" password = "
# icinga2 feature enable ido-pgsql api command # icinga2 feature list The feature list subcommand should list the ido-pgsql under the Enabled features. The only thing left for the database part is to restart the postgresql service and icinga2.
# service postgresql restart # service icinga2 start 12 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Nginx Configuration Overview
1 Introduction to Icinga
2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup
3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks
4 Restarting Services Automatically
13 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Nginx Configuration Nginx Configuration Icinga2 will work fine without the Icingaweb2 component and is in fact running along on many machines. However, a central management interface is good to get an overview and central point of data collection where administrators can see what kind of events are generated. Also, users can be given access to a subset of the monitored hosts and/or services without being able to see and manage other objects. For these users, it is often easier to use a web interface rather than a CLI. The first part of the webserver setup deals with getting PHP setup correctly as a fastCGI process that Nginx runs. First, we enable both nginx and php_fpm in our rc.conf, but do not start the services yet.
# sysrc php_fpm_enable=yes # sysrc nginx_enable=yes Then, we’re using sed to uncomment some options in www.conf in the /usr/local/etc/php-fpm.d/ directory.
WWWCONF=/usr/local/etc/php-fpm.d/www.conf # sed -i '' "s/listen\ =\ 127.0.0.1:9000/listen\ =\ \/var\/run\/php5-fpm.sock/" $WWWCONF # sed -i '' "s/;listen.owner/listen.owner/" $WWWCONF # sed -i '' "s/;listen.group/listen.group/" $WWWCONF # sed -i '' "s/;listen.mode/listen.mode/" $WWWCONF 14 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Nginx Configuration Changes to nginx.conf
We need to add the following lines before location / { ... } in /usr/local/etc/nginx/nginx.conf:
location ~ ^/icingaweb2/index\.php(.*)$ { fastcgi_pass unix:/var/run/php5-fpm.sock; fastcgi_index index.php; include fastcgi_params; fastcgi_param SCRIPT_FILENAME /usr/local/www/icingaweb2/public/index.php; fastcgi_param ICINGAWEB_CONFIGDIR /usr/local/etc/icingaweb2; fastcgi_param REMOTE_USER $remote_user; }
location ~ ^/icingaweb2(.+)? { alias /usr/local/www/icingaweb2/public; index index.php; try_files $1 $uri $uri/ /icingaweb2/index.php$is_args$args; }
15 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Nginx Configuration Setting up the Icinga CA
Icinga2 creates it’s own certificate authority for trusted API access. We activated the api feature already (check the output of icinga2 feature list to confirm). We need to initialize the CA to create keys and certificates using:
# icinga2 api setup
Next, we go into Icinga2’s main configuration directory (/usr/local/etc/icinga2/conf.d/) and edit api-users.conf. It defines all the users that can access the API and their permissions. We need to create a separate ApiUser for the Icingaweb2 interface. Add a section like this to the file:
object ApiUser "icingaweb2" { password = "
16 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Nginx Configuration Finishing Up the Nginx Configuration
A few more things remain: we copy the php.ini intended for production use to the proper location and change the timezone the server is in. Then, we can start both the php fastCGI process and the nginx webserver:
# service nginx start # service php-fpm start Once these services are running, we can start configuring the Icingaweb2 interface from a web browser of choice. Browse to http://local.IP.or.domain/icingaweb2/setup
17 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Overview
1 Introduction to Icinga
2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup
3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks
4 Restarting Services Automatically
18 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Setup Token Creation
First we need to enter the setup token. It protects the installation from other people who encounter this page while the setup is not completed yet. Without the proper token, they can not proceed. 19 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Setup Token Creation
To create the token, run the following: sudo /usr/local/www/icingaweb2/bin/icingacli setup token create \ --config=/usr/local/etc/icingaweb2 Example output: The newly generated setup token is: ba63907b6ce1ecac Make sure to set owner and group properly: sudo chown -R www:www /usr/local/etc/icingaweb2 Then, back in the browser window, enter the generated token into the form and press the "Next" button:
20 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Select Modules
In this screen, select at least the monitoring module. Documentation is recommended for newcomers. You can find all the information on icinga.com, too.
21 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Check Dependencies
The setup checks if all components for Icingaweb2 are installed. Make sure that no item is red. You can ignore the yellow message in the "Linux Platform" section. 22 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Authentication Setup
Here we define how Icingaweb2 should authenticate within the backend services. We select database here.
23 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Configure Database Backend
We enter the database information of your previously setup postgres database. Check it with "Validate Configuration". 24 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Setup Administrator Account for Icingaweb2
Next, we set up the administrator account. Chose a strong password. This account can create other accounts later.
25 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Configure the Icingaweb2 Application/Logging
Here we chose to store all logging and application information in our database. Select "Webserver Log" as the logging type. 26 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Summary
This is just a summary screen about what we configured so far.
27 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Configure Monitoring Module
Next, we set up the monitoring module we selected earlier.
28 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Configure IDO as Monitoring Backend
We chose to retrieve and store all monitoring data in the IDO (Icinga Data Output).
29 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Configure IDO Ressource
The IDO itself is stored in our postgresql database, so we enter the connection information here again. Click "Validate Configuration" to see if it can establish a connection before proceeding.
30 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Command Transport
Icinga2 can be controlled by a comprehensive API, for which we defined a separate ApiUser called icingaweb2 before. We need to enter it here and the password from /usr/local/etc/icinga2/conf.d/api-users.conf. 31 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Protected Custom Variables
Leave these default settings as they are.
32 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Summary
We’re almost done at this point. We just get another summary of the settings we’ve made.
33 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Icingaweb2 Setup Setup Steps in the Browser: Success!
34 / 66 Time for a break BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Introduction to Monitoring Objects Overview
1 Introduction to Icinga
2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup
3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks
4 Restarting Services Automatically
36 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Introduction to Monitoring Objects Creating and including our own configuration files Icinga2 provides a number of default configuration files in /usr/local/etc/icinga2/conf.d/ with basic settings. For a beginner, they can be a bit overwhelming, so we move them out of the way (for reference) and create our own config files one after the other. That way, we understand how these files work together and we’re not surprised by things we have not defined yet.
cd /usr/local/etc/icinga2/ /usr/local/etc/icinga2# mv conf.d conf.orig /usr/local/etc/icinga2# mkdir tutorial We need to tell Icinga to include that directory and all it’s subdirectories we’re going to create. This is done in /usr/local/etc/icinga2/icinga2.conf. We’ll comment the line include_recursive "conf.d" and add one for our newly created tutorial directory below it:
//include_recursive "conf.d" include_recursive "tutorial"
Make sure to copy the api-users.conf and other files we created to the tutorial directory.
37 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Introduction to Monitoring Objects Monitoring Objects in Icinga Icinga used various objects in it’s configuration files that have certain attributes. They are encoded in a human-readable way. The general syntax is as follows:
object
Object types could be Host, Service, User, Template, and many others. The following value types can be used: Type Example Number 42 Duration 5m String "This is a comment" Boolean true Array [ "value1", "value2" ] Dictionary "key1" = "value1", "key2" = 123
38 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Introduction to Monitoring Objects Defining a Host Creating a host object is one of the basic tasks that should be performed at the beginning when setting up a new monitoring system. It helps us structure our environment and list hosts that we want to monitor, defines common checks, names, etc. A basic host object is defined in /usr/local/etc/icinga2/tutorial/host.conf like this:
object Host "Hal9000" { address = "1.2.3.4" check_command = "hostalive" } We’ve defined a basic host object and assigned a basic service check that will alert us if the host is unreachable. To validate and make the new object known to Icinga, we run the following command, which will fail if there are any config errors:
# icinga2 daemon -C && service icinga2 reload If all went well, we should see the new host in Icingaweb2.
39 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Introduction to Monitoring Objects More Host Attributes
The Host object can take more optional parameters, but check_command must always be provided to be a valid host object. Other interesting attributes that we are going to use later are listed below: display_name (string): An alternative name to be displayed instead of the object name. address6 (string): The IPv6 address of the host. groups (string): A list of groups this host is a member of. vars (Dictionary): Custom attributes specific to this host than can be freely defined. check_interval (Duration): The interval in which the host should be checked in seconds. event_command (Object name): Name of an event command to be executed if the host changes state.
40 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Introduction to Monitoring Objects Displaying Host Attributes The Icinga CLI can display the current and default attributes for an object using the following syntax: icinga2 object list --type Host --name Hal9000 The information displayed is valuable in debugging (where are objects defined) and what kind of values are currently set, without having to look into individual config files.
Object 'Hal 9000 ' of type 'Host ': % declared in '/usr/local/etc/icinga2/tutorial/hal9000.conf ', lines 2:1-2:17 * __name = "Hal9000" * action_url = "" * address = "1.2.3.4" * check_command = "hostalive" % = modified in '/usr/local/etc/icinga2/tutorial/hal9000.conf ', lines 11:3-11:29 * check_interval = 300 * check_period = "" * check_timeout = null * command_endpoint = "" * display_name = "Hal 9000" ... Remove the filters --type and --name to display all objects that Icinga2 knows about. 41 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Introduction to Monitoring Objects Service Objects
We’re not only interested in monitoring server uptime, but also whether individual services running on these hosts are working fine (whatever that means). For that, we can define Service objects in a file called services.conf. Again, we use the general syntax, but use Service as the object type. Each service needs to be associated with a certain host object (attribute host_name), since not all hosts are running all the services.
object Service "ping4" { host_name = "Hal9000" check_command = "ping4" } Let Icinga check it’s configuration and reload the icinga service to see the newly created object.
42 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Introduction to Monitoring Objects Defining Groups Group objects help to logically group hosts or services together based on similar attributes. For example, hosts from a certain subnet mask could form a group or those with a certain number of CPUs. The most common way for defining groups is based on variables. We’ll add a variable called vars.os to our host object to filter on. The file looks like this now:
object Host "Hal9000" { address = "1.2.3.4" check_command = "hostalive" vars.os == "Linux" }
We can then define the group object in the file groups.conf in the following way:
object HostGroup "Linux Machines" { assign where host.vars.os == "Linux" }
object HostGroup "FreeBSD Machines" { assign where host.vars.os == "FreeBSD" }
43 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Introduction to Monitoring Objects Assigning Services to Hosts using Apply Rules
As our infrastructure and the number of objects (hosts and services) grows, it becomes tedious to add a service check for each new host and services running on that host. Instead, it would be great if Icinga would apply service checks to new hosts automatically, based on certain attributes. This can be achieved by defining variables like vars.os for example and then use so called apply rules to assign services based the variable values. That way, we only need to define host objects and add new services when they are introduced and let Icinga take care of the mapping for us. We also don’t clutter our config files with a lot of definitions for each and and every host or service, but keep them orderly and small enough to manage.
44 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Plugin Integration Overview
1 Introduction to Icinga
2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup
3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks
4 Restarting Services Automatically
45 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Plugin Integration Integrating External Plugins Icinga2 comes with a range of plugins for checking various standard system parameters (CPU and memory load, disk space, etc). In case this is not enough and other things should be monitored, we need to integrate these external plugins into our monitoring infrastructure. We find these plugins either on sites like https://www.monitoring-plugins.org or in the ports collection. Remember that Icinga can also run Nagios plugins without modification. For example, these are just a few plugins that are available on FreeBSD1:
# pkg install nagios-check_zpools nagios-check_cpu_usage nagios-check_ports Some plugins for the BSD community: • https://exchange.icinga.com/araujobsd/icinga2-network-plugin • https://exchange.icinga.com/bmccorkle/check_bsd_mem • https://exchange.icinga.com/anaumov/OpenBSD%20SNMPv3%20general%20check The plugins are installed in the /usr/local/libexec/nagios/ directory.
1Thank you porters! 46 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Plugin Integration Testing the External Plugins
First, we need to see whether the plugins work as expected for our monitored infrastructure. To that end, we execute them manually from /usr/local/libexec/nagios/ to check their output or if they require any parameters. For example, many plugins require the warning (-w) and critical (-c) thresholds to be set to send alerts when they are reached. /usr/local/libexec/nagios/check_zpools -p ALL -w 80 -c 90 In this case, we define that all ZFS pools should be checked and when they reach 80% full, a WARNING status is sent to Icinga. Similarly, when 90% are reached, the plugin emits CRITICAL.
47 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Plugin Integration Integrating the Plugin into our Infrastructure Each machine that should be checked needs to have the plugin installed/copied to /usr/local/libexec/nagios/. First, we add a new CheckCommand entry for the plugin in /usr/local/etc/icinga2/tutorial/nagios_plugins.conf:
object CheckCommand "zpools" { import "plugin-check-command" command = [ PluginDir + "/check_zpools" ] arguments = { "-p" = " ALL " "-w" = "80" "-c" = "90" } } In services.conf, we define which hosts should execute the plugin:
apply Service "zpools" { import "generic-service" check_command = "zpools" assign where host.vars.os == "FreeBSD" } Next, restart Icinga2: sudo icinga2 daemon -C && sudo service icinga2 restart 48 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Plugin Integration Example Output in Icingaweb2 for the check_ports Plugin
49 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Monitoring via OpenSSH Overview
1 Introduction to Icinga
2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup
3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks
4 Restarting Services Automatically
50 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Monitoring via OpenSSH Rationale for Monitoring Hosts via OpenSSH
There are a number of benefits for using OpenSSH to execute checks on remote hosts and report the results back to the Icinga server. Availability Most, if not all Unix hosts have SSH enabled for remote management, de facto standard. Little to no extra setup required. Secure Check results represent the state of remote system and can contain data that can be valuable to an attacker. SSH encrypts all the data to and from the remote machine so that listening on the wire will not provide more information to an attacker. One downside is that logs are increasing as every connection (depending on the check interval) will be logged auth.log). This may not be ideal on systems with not a lot of disk space (embedded, network equipment) available and without a proper log analysis solution, might hide other important authentication log entries in the flood. With proper log rotation and retention, this should be manageable. Logging to a central log host will also solve this problem.
51 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Monitoring via OpenSSH Overview
1 Introduction to Icinga
2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup
3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks
4 Restarting Services Automatically
52 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Monitoring via OpenSSH Generating the SSH Key Pair To generate a new key for the icinga user, use ssh-keygen:
icinga2@host$ ssh-keygen -t ed25519 -f icinga Generating public/private ed25519 key pair. Enter passphrase (empty for no passphrase): Your identification has been saved in /var/spool/icinga/.ssh/ansible. Your public key has been saved in /var/spool/icinga/.ssh/icinga.pub. The key fingerprint is: SHA256:Hcl0VflRPR12ebzG2F/EoM5z3NvFSf5mgI8N6Sl/5Tg icinga@host The key 's randomart image is: +--[ED25519 256]--+ | . o ..+ %| | o o o=B| | +.+ +. . =B| | . .= Bo =| | S .+ o=O+| | o-. o.ooO+| | +..+ o=| | o.C o| | .o | +----[SHA256]-----+
53 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Monitoring via OpenSSH Remote Machine User Setup
Create a user called icinga on each host that should be monitored via SSH. Define a shell, /bin/sh is usually enough. Set the home directory to /var/spool/icinga (optional). Use ssh-copy-id to copy the public key to the users home directory and register it to log in passwordless. (You can also create a Match rule in sshd for the icinga user in /etc/ssh/sshd_config). Some plugins must be run with root privileges (like check_hdd_health). We also want to make sure that no one can run arbitrary commands as the icinga user on the remote system. Therefore, we set up sudo to run only certain commands. Here is an example from /usr/local/etc/sudoers: icinga ALL=(ALL) NOPASSWD: /usr/local/libexec/nagios/* The icinga plugins are located in /usr/lib/nagios/plugins on Linux hosts. Test the passwordless login and the privilege execution of the plugins before writing checks for it on the Icinga master.
54 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Monitoring via OpenSSH Service Checks Using the by_ssh Plugin Here is a simple service check for the load plugin. It requires the check_command = "by_ssh", which will execute the plugin on the remote system. It needs to know where the plugin is located on the remote machine. The location and any other required parameters can be passed to the plugin using the vars.by_ssh_command line. Don’t forget to define to which hosts this plugin should be assigned to. The execution intervall of these checks is defined in the "generic-service" template in templates.conf.
1 apply Service "load_ssh" { 2 import "generic-service" 3 check_command = "by_ssh" 4 vars.by_ssh_command = PluginDir + "/check_load -w 5,5,5 -c 8,8,8"
6 assign where host.vars.os == "FreeBSD" 7 }
The PluginDir variable is defined in /usr/local/etc/icinga2/constants.conf as: const PluginDir = "/usr/local/libexec/nagios"
55 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Configuring Icinga2 Monitoring via OpenSSH Running checks depending on host OS Variables Since we defined the vars.os variable in our hosts.conf file for each host to monitor, we can make Icinga run checks based on that variable value.
1 apply Service "load_ssh" { 2 import "generic-service" 3 check_command = "by_ssh"
5 if (host.vars.os == "Linux") { 6 vars.by_ssh_command = LinuxPluginContribDir + "/check_load -w 5,5,5 -c 8,8,8" 7 }
9 if (host.vars.os == "FreeBSD") { 10 vars.by_ssh_command = PluginDir + "/check_load -w 5,5,5 -c 8,8,8" 11 }
13 assign where host.vars.os == "FreeBSD" || host.vars.os == "Linux" 14 }
We defined the variable LinuxPluginContribDir in our constants.conf like this: const LinuxPluginContribDir = "/usr/lib/nagios/plugins"
56 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Restarting Services Automatically
Overview
1 Introduction to Icinga
2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup
3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks
4 Restarting Services Automatically
57 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Restarting Services Automatically
Rationale
Since we’re already monitoring whether a service is in a certain state (DOWN), we can also trigger actions on certain events. For example, when the service state changes from UP to DOWN and there is no planned downtime, it can be assumed that this is an unscheduled outage. Icinga can issue service restart commands when the service stays down after a couple of retries. Once the number of retries has passed and the service is still in the DOWN state, Icinga will automatically call the configured restart routines without any administrator intervention. This way, a service downtime is limited. A word of caution: Automatically restarting services does not solve all problems. Some services take longer or have other dependencies (i.e. dependent databases) that can’t be handled by Icinga. A service downtime might need further investigation by a sysadmin to figure out the cause for it to prevent it from happening in the future. For simple services, Icinga’s functionality works and is quicker than any admin could act. Don’t use is for more complicated setups.
58 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Restarting Services Automatically
Setting up local_eventcommands.conf We create a file called local_eventcommands.conf for define commands to run as template that we can import in the actual events:
object EventCommand "event_by_ssh" { command = [ "/usr/local/libexec/nagios/check_by_ssh" ]
arguments = { "-H" = "$event_by_ssh_address$" "-p" = "$event_by_ssh_port$" "-C" = "$event_by_ssh_command$" "-l" = "$event_by_ssh_logname$" "-i" = "$event_by_ssh_identity$" "-q" = { set_if = "$event_by_ssh_quiet$" } "-w" = "$event_by_ssh_warn$" "-c" = "$event_by_ssh_crit$" "-t" = "$event_by_ssh_timeout$" }
vars.event_by_ssh_address = "$address$" vars.event_by_ssh_quiet = false }
59 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Restarting Services Automatically
Defining the commands to run
First, we import the event_by_ssh EventCommand. Then, we check the return code in $service.state_id$ using the test command. If we detect a non-zero return value, we let sudo execute the service command to restart the service defined by the $event_by_ssh_service$ variable.
object EventCommand "event_by_ssh_restart_service" { import "event_by_ssh"
//only restart the daemon if state > 0 (not-ok) //requires sudo permissions for the icinga user vars.event_by_ssh_command = "test $service.state_id$ -gt 0 && \ sudo service $event_by_ssh_service$ restart" }
This requires a line like this in /etc/sudoers to allow the icinga user to restart the service:
icinga ALL=(ALL) NOPASSWD: /usr/sbin/service nginx restart
60 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Restarting Services Automatically
Define the service to restart
For testing purposes, we define a new service in services.conf for the nginx webserver. We apply it to a specific host afterwards:
apply Service "http" { import "generic-service" check_command = "http" event_command = "event_by_ssh_restart_service" vars.event_by_ssh_service = "$host.vars.httpd_name$"
assign where host.name == "BSDHost" }
61 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Restarting Services Automatically
Setting the service name to restart
In hosts.conf we set the variable vars.httpd_name to the service name (nginx in this example):
object Host "BSDHost" { import "generic-host" address = "IP.address.of.host" vars.httpd_name = "nginx" }
62 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Restarting Services Automatically
Testing the Changes by Forcing an Automatic Service Restart
After testing the changes and restarting the icinga service, we want to test whether the service restart will trigger upon a service downtime. We manually stop the nginx service and look at the icinga.log file. We find entries like the following and the service should be up again:
[2018-11-02 15:09:29 +0100] information/ExternalCommandListener: Executing external command: [1541167769] SCHEDULE_FORCED_SVC_CHECK;BSDHost;http;1541167769 [2018-11-02 15:09:30 +0100] warning/PluginEventTask: Event command for object 'BSDHost!http ' (PID: 6328, arguments: '/usr/local/libexec/nagios/check_by_ssh ''-C' 'test 2 -gt 0 && sudo service nginx restart ''-H''IP.address.of.host ') terminated with exit code 3, output: Remote command execution failed: nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
[2018-11-02 15:09:38 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 1, rate: 12.3833/s (743/min 743/5min 743/15min); [2018-11-02 15:10:13 +0100] warning/PluginEventTask: Event command for object 'BSDHost!http ' (PID: 7134, arguments: '/usr/local/libexec/nagios/check_by_ssh ''-C' 'test 0 -gt 0 && sudo service nginx restart ''-H''IP.address.of.host ') terminated with exit code 1, output: WARNING - check_by_ssh: Remote command 'test 0 -gt 0 && sudo service nginx restart ' returned status 1
63 / 66 Questions? BSD-based Systems Monitoring with Icinga2 and OpenSSH Restarting Services Automatically
Thanks
• Lars Engels (lme@) for maintaining the Icinga2 port in FreeBSD, help and advice • Netways GmbH for their Icinga2 training that helped me fill in my Icinga2 knowledge gaps • Icinga2 community for their comprehensive documentation and willingness to help • Port maintainers and committers who maintain monitoring plugins in the ports collection • You for taking this tutorial. I hope it is useful to you. Happy monitoring!
65 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Restarting Services Automatically
Thanks
• Lars Engels (lme@) for maintaining the Icinga2 port in FreeBSD, help and advice • Netways GmbH for their Icinga2 training that helped me fill in my Icinga2 knowledge gaps • Icinga2 community for their comprehensive documentation and willingness to help • Port maintainers and committers who maintain monitoring plugins in the ports collection • You for taking this tutorial. I hope it is useful to you. Happy monitoring!
65 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH
Further Information
Icinga2 Documentation Icinga 2 Documentation https://icinga.com/docs/icinga2/latest/ Installing Icinga Web 2 with Apache 2.4, Icinga 2 and MySQL on FreeBSD Icingaweb2 Setup Tutorial by Lars Engels (Icinga port maintainer) https://lme.postach.io/post/ installing-icinga-web-2-with-apache-2-4-icinga-2-and-mysql-on-freebsd Monitoring Plugins Website https://www.monitoring-plugins.org
66 / 66