heck What the #%!@ is wrong ^ with my server?!?

Josh Malone Systems Administrator National Radio Astronomy Observatory Charlottesville, VA

1 Agenda • Intro to Monitoring • Internet protocols 101 • • SMTP • IMAP • Install/Config • HTTP • Usage • Custom plugins • Packet sniffing for dummies • Intro to Troubleshooting • Tools • , openssl • grep, sed • ps, lsof, netstat

2 MONITORING

3 Automated Monitoring Workflow

4 Monitoring Packages: Open Source • • Pandora FMS

• Opsview Core •

• •

• •

• • Captialware ServerStatus

• Core • Sensu

All Trademarks and Logos are property of their respective trademark or copyright holders and are used by permission or fair use for education. Neither the presenter nor the conference organizers are affiliated in any way with any companies mentioned here.

5 Monitoring Packages: Commercial • Nagios XI

• Groundwork

• PRTG network monitor

• CopperEgg

• WhatsUp Gold

• PRTG network monitor

• op5 (Naemon)

All Trademarks and Logos are property of their respective trademark or copyright holders and are used by permission or fair use for education. Neither the presenter nor the conference organizers are affiliated in any way with any companies mentioned here.

6 Why Automatic Service Monitoring? • Spot small problems before they become big ones • Learn about outages before your users do • Checklist when restoring from a power outage • Gives you better problem reports than users • Problems you might never spot otherwise • Failed HDDs in RAIDs • Full /var partitions • Logs not rotating • System temperature rising

7 Why Automatic Service Monitoring? • Capacity planning • Performance data can generate graphs of utilization • RAM, Disk, etc. • Availability reports - CAUTION • Easy to generate -- even easier to generate wrong • Make sure your configurations actually catch problems • Will also include problems with Nagios itself :( • If you’re going to quote your availability numbers (SLAs, etc.) make sure you understand what you’re actually monitoring.

8 Without Monitoring With Monitoring

! • dhcp out of leases “The Internet’s down - • dhcp server down fix it!!!” • dns server not responding • ethernet switch down • ISP link down / saturated

9 Without Monitoring With Monitoring

! • connectivity issues “ZOMG! Our web site • down is down! O Noes!!!” • apache not running • web server disk full • server load too high

10 Nagios • Open source host / service monitoring package • “Nagios Ain't Gonna Insist On Sainthood” • Originally released in 1999 as “NetSaint” • Available in 2 versions: Core and XI • Nagios Core: Open-source, freely available • Nagios XI: Commercial • Free license for up to 7 hosts • Available as source installer or VMware appliance

11 Terminology • Host - A logical (physical / virtual) machine running an OS • Service - A resource available on a host • Network service (www, dns, imap) • Local resource (free RAM, disk space, system load) • Plugin - An executable that checks something • Add-on - An extension to Nagios that adds functionality • Graphing, Trending, SNMP trap reporting, etc.

12 What’s a plugin? • Plugins actually run the service or host checks. • Each plugin monitors a different type of service • Data from plugin is communicated to Nagios using a (very) simple API • Plugins can also report “Performance Data” (perfdata) to be graphed or tracked • Requires a perfdata add-on (or Nagios XI) • Plugins can be written in any language • plugins can run using Nagios’s embedded perl interpreter for increased performance

13 INSTALLING NAGIOS

14 A word of caution... Installing Nagios on is much easier than Mac. While most Linux distros are still shipping Nagios 3.5.x, they usually come with all the pre-requisite packages available via their native package managers. This makes it much easier to build 4.x from source. Small to medium environments don’t require much of a server to run Nagios. Mac installations will require you to install numerous pre- requisites, especially for the plug-ins. Many of the pre-reqs can be installed using MacPorts, Homebrew and cpan. If you can run on Linux, do it.

15 About Nagios Replacements When Nagios went commercial, the “open-source community” decided that it needed not one, not two, but three replacements for Nagios: and Naemon (forks of Nagios) and (a drop-in replacement). Most linux distros are now shipping one or more of these compatible replacements rather than the official Nagios Core. Not a single distro I checked is shipping Nagios 4. Either Shinken, Naemon or Icinga should work fine using the material covered in the tutorial, but I have only briefly tested Icinga and have not tested Shinken or Naemon at all.

16 About Nagios Plugins source • On January 15, 2014, Nagios Enterprises “forked” the original code from the “Nagios plugins” project, effectively creating two sources for the Nagios plugins. • The original maintainers of the plugins package renamed it to “monitoring plugins” and now host their distribution at monitoring-plugins.org • Nagios Enterprises appears to be actively maintaining the nagios-plugins package, including security fixes • This tutorial will use the official Nagios plugins package

17 Requirements • compiler (Xcode + cmdline tools) • Perl 5.8+ (for many plugins) • Web server (for web interface) • PHP 5.3 • gd (for statusmap and trends CGIs) • , perl-RRD, -GD (for PNP4Nagios)

18 Nagios 4 - brew or build from source • Latest MacPorts version of nagios is 3.5.0 - boo :( • Uses old worker code and not as efficient as 4.x • Latest Homebrew version is 4.0.6 - yay! • brew install nagios! • We want Nagios 4.0.6 at least • fixes lots of bugs and is much faster

19 Create the user and group • Nagios needs a user / group to run as • Create user ‘nagios’ with group ‘nagios’ • Can use System Preferences -- no need to resort to command line

20 Create group ‘nagios’ • Go to Users and Groups, click the + • Change “New Account:” selector to “Group” • Set name to ‘nagios’

21 Create user nagios • Click + again • Create a “Standard” account • Account name: ‘nagios’ • Set a password for security

22 Create user nagios • Right-click the new ‘nagios’ account and select • Change group to ‘nagios’ • Set shell to /usr/bin/false

23 Download • Nagios Core • http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-4.0.7.tar.gz • Nagios plugins package • http://nagios-plugins.org/downloads/ • PNP4Nagios • http://docs.pnp4nagios.org/pnp-0.6/dwnld

24 Installing Nagios on OSX • Install Xcode and command line tools • Install gd library using MacPorts (or homebrew) • port install gd2!

• brew install gd!

• cpan! • install Net::SNMP! • Extract the tarball • Configure and install nagios ./configure --with-gd-inc=/opt/local/include \ --with-gd-lib=/opt/local/lib \ --with-httpd-conf=/etc/apache2/other!

25 Compile problems:

• ld: archive has no table of contents for architecture x86_64 • Probably a makefile bug • Solution: run ranlib lib/libnagios.a

26 Installation from source sudo -s! make install! make install-config! make install-webconf install-exfoliation! make install-commandmode! install -m 755 -o root -g wheel -init \! /usr/local/nagios/nagios.init! htpasswd -c /usr/local/nagios/etc/htpasswd.users nagios

27 Build and Install Plugins • The plugins have LOTS of prerequisites • LDAP libs • SNMP utils • samba client • openssl libs • Installed most prerequisites via MacPorts • Homebrew should work as well ! ./configure --prefix=/usr/local/nagios \! --enable-perl-modules \! --with-=/opt/local/lib/mysql5 \! --with-openssl=/opt/local/

28 Install pnp4nagios • Pre-reqs: rrdtool, perl RRD modules, php-GD

! ./configure --with-nagios-user=nagios \! —with-nagios-group=nagios! ! make! # Install without npcd (bulk mode)! make install-config install-processperfdata \! install-html install-plugins

29 Configure pnp4nagios cp /usr/local/pnp4nagios/etc/nagios.cfg-sample \! /usr/local/nagios/etc/pnp4nagios.cfg! cp /usr/local/pnp4nagios/etc/misccommands.cfg-sample \ ! /usr/local/nagios/etc/objects/misccommands.cfg

30 Create LaunchDaemon

! ! ! ! Label! org.nagios.nagios! ProgramArguments! ! /usr/local/nagios/bin/nagios! -d! /usr/local/nagios/etc/nagios.cfg! ! KeepAlive! ! !

31 Web server / PHP • Enable OSX web server • Set PHP timezone to avoid a warning echo "date.timezone = 'America/New_York'" >>/etc/php.ini

32 CONFIGURING NAGIOS

33 Configuration • Nagios has 2 parts to configure: the daemon and the cgi’s • Configure the CGIs to define access levels (authorization) • Configure the daemon • Commands • Check commands • Misc commands (how to notify, event handlers) • Services • Contacts • Monitoring and alerting policies

34 cgi.conf

• Enable authentication in the CGIs • use_authentication=1! • Expects your web server to authenticate users

• Only allow admin users to issue commands and view config details • authorized_for_system_information=...! • authorized_for_configuration_information=...! • authorized_for_system_commands=...! • authorized_for_all_service_commands=...! • authorized_for_all_host_commands=...!

• Allow any authenticated user to view Nagios status • authorized_for_all_services=*! • authorized_for_all_hosts=*

35 nagios.cfg • Main configuration file • Typically includes additional cfg files for commands, services, etc. (“objects”) • cfg_file=/path/to/file! • cfg_dir=/path/to/dir/of/configs!

• Interval length • “How many seconds are Nagios’s default unit of time” • If you need to monitor a service more than once per minute, you can lower this from 60 • interval_length=60

36 nagios.cfg • Timeouts (in seconds): • service_check_timeout=60! • host_check_timeout=30! • event_handler_timeout=30! • notification_timeout=30! • ocsp_timeout=5! • perfdata_timeout=5!

! • If you have checks, event handlers, etc. that take a long time to run, increase the appropriate timeout value

37 Included default config files • commands.cfg Define commands called in other files • contacts.cfg Define contacts to be notified • localhost.cfg Example services for the local system, • printer.cfg example printer, example network switch, • switch.cfg etc. • templates.cfg Templates for most objects; useful as a starting point for inheritance • timeperiods.cfg Def. periods of time (24x7, daytime, etc)

38 Inheritance • All objects in Nagios support inheritance via the “use