BSD-based Systems Monitoring with Icinga2 and OpenSSH BSD-based Systems Monitoring with Icinga2 and OpenSSH Benedict Reuschling [email protected] BSDCan 2019 Tutorial 1 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Monitoring When we talk about monitoring, we talk about collecting valuable information from target systems to help system administrators and others stay informed about their ever-increasing hardware and software landscape. Problems can be detected early and proactively, without little or no visible impact to the users of these systems. It is also a tool for decision-makers about how well service-levels are provided, outdated systems and services that could be replaced, as well as new systems to be introduced and monitored the same way as done on previous systems. Typically, monitoring deals with the following three aspects: Availability - Is the host or service available on the network? Metrics - Data collected from hosts and services Logs - Messages (errors, warnings) written to log files from hosts and services We’ll cover the first two aspects in these slides. The third one warrants its own due to size and complexity of the topic. Icinga is used to demonstrate one of many monitoring solutions due to its programmability, flexibility, and scalability with the number of hosts and services to monitor. 2 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Introduction to Icinga Overview 1 Introduction to Icinga 2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup 3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks 4 Restarting Services Automatically 3 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Introduction to Icinga Introduction to Icinga2 In this tutorial, we’ll take a closer look at using and configuring the unofficial Nagios replacement called Icinga2 to cover our monitoring needs. Icinga is a modern system that builds on many of the concepts that made Nagios great, but adds their own, including a modern web interface called Icingaweb2. Plugins from Nagios can be used in Icinga without modifications and there is a whole ecosystem of new plugins available for Icinga. Icinga was developed due to lack of response from the Nagios authors to react to bugs and feature requests. The Icinga project was created to integrate those and has since rewritten many of the backend components, while still maintaining compatibility to Nagios plugins as much as possible. Icinga offers its own ecosystem of plugins and there is an active open source community that develops the core Icinga system and creates new plugins (https://www.monitoring-plugins.org). There is also training, consulting, and support available for large and complex Icinga2 installations. 4 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Overview 1 Introduction to Icinga 2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup 3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks 4 Restarting Services Automatically 5 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Installing Required Software Packages Icinga2 and its Icingaweb2 graphical component is a classic LAMP stack, but can also use of some other components. In this tutorial, we’ll use Nginx as the webserver and PostgreSQL as database backend. The port is still using PostgreSQL 9.5 at the time of this writing. Newer versions of PHP should either be pulled in as dependencies automatically or can be selected based on your own experiences. There is no need to use the latest versions and teh monitoring system will run just fine with older versions. We’ll install the following packages (the rest is pulled in as dependencies): • pkg install icinga2 icingaweb2 postgresql95-server nginx ImageMagick-nox11 php72-pecl-imagick The ImageMagick ports are required for by Icingaweb2 during setup and are used for report generation. If that part is not used, omit this port. 6 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Adding Services to /etc/rc.conf We need to start three components of our monitoring system: 1 The backing PostgreSQL database 2 The Icinga2 monitoring core component 3 The Nginx webserver with fastCGI support for Icingaweb2 Using sysrc, this is not difficult: sysrc icinga2_enable=yes sysrc postgresql_enable=yes sysrc nginx_enable=yes Don’t start these services just yet, we have a bit of configuration ahead of us! But before we forget, we should rotate our log files: # cp /usr/local/share/examples/icinga2/newsyslog/icinga2 /etc/newsyslog.conf.d/ We’ll start our configuration with the database setup. 7 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Database Setup Overview 1 Introduction to Icinga 2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup 3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks 4 Restarting Services Automatically 8 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Database Setup Database Setup First of all, when running on OpenZFS, we’ll create a dataset for PostgreSQL and set a mountpoint. Lastly, we’ll change ownership to the pgsql user account that will run the service later. # zfs create -o mountpoint=/usr/local/pgsql/data sys/pgdata # zfs set recordsize=8k sys/pgdata # zfs set logbias=throughput sys/pgdata # zfs set redundant_metadata=most sys/pgdata # zfs set primarycache=metadata sys/pgdata # chown pgsql:pgsql /usr/local/pgsql/data The ZFS dataset settings are typical for any kind of PostgreSQL database. There are no heavy queries to the database, but Icinga2 still needs to store a lot of information about monitoring objects, it’s internal state, user accounts, etc. 9 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Database Setup Initializing the Database Cluster We’ll switch to the pgsql user account and initialize the database cluster. # su pgsql $ cd $ initdb -D ./data --no-locale --encoding=utf-8 --lc-collate=C $ pg_ctl start -D ./data After we started the database cluster, we’ll create a user account in the database for Icinga and set a password. We need this later when setting up the icinga tables and Icingaweb2. Then, we set up a database with the same name to hold the tables. $ createuser -dPrs icinga Password : Re-type password: $ createdb -O icinga -E UTF8 icinga Then, we create local access entries for the Icinga user in pg_hba.conf. Make sure not to change the trust parts into ident as this will not work. local icinga icinga md5 host icinga icinga 127.0.0.1/32 md5 host icinga icinga ::1/128 md5 10 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Database Setup Importing the Icinga2-Schema The database tables and indexes that Icinga2 needs to store various information are stored in /usr/local/share/icinga2-ido-pgsql/schema/pgsql.sql as a schema. $ psql -U icinga -d icinga < /usr/local/share/icinga2-ido-pgsql/schema/pgsql.sql Future versions of Icinga2 might require schema upgrades when updating to a newer version. The upgrade instructions will tell you when and how to perform the schema updates. Intermediate upgrades need to be applied as well, but they are all timestamped and can be imported in the same way as described here for the initial schema. The upgrades are located in /usr/local/share/icinga2-ido-pgsql/schema/upgrade. 11 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Database Setup Configuring the IDO database The IDO (Icinga Data Output) will store information about our monitoring instance in the PostgreSQL database. We need to store our database connection credentials in the file /usr/local/etc/icinga2/features-enabled/ido-pgsql.conf with the following content: object IdoPgsqlConnection "ido-pgsql" { user = "icinga" password = "<the_password_for_the_database_user_icinga_we_defined_earlier>" host = "localhost" database = "icinga" } Next, we log out of the psql user account and activate the ido-pgsql, api, and command features using the icinga2 CLI. # icinga2 feature enable ido-pgsql api command # icinga2 feature list The feature list subcommand should list the ido-pgsql under the Enabled features. The only thing left for the database part is to restart the postgresql service and icinga2. # service postgresql restart # service icinga2 start 12 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Nginx Configuration Overview 1 Introduction to Icinga 2 Installing the Monitoring System Database Setup Nginx Configuration Icingaweb2 Setup 3 Configuring Icinga2 Introduction to Monitoring Objects Plugin Integration Monitoring via OpenSSH SSH Setup Creating SSH Checks 4 Restarting Services Automatically 13 / 66 BSD-based Systems Monitoring with Icinga2 and OpenSSH Installing the Monitoring System Nginx Configuration Nginx Configuration Icinga2 will work fine without the Icingaweb2 component and is in fact running along on many machines. However, a central management interface is good to get an overview and central point of data collection where administrators can see what kind of events are generated. Also, users can be given access to a subset of the monitored