DOCSLIB.ORG

Distributed Monitoring with Opennms Distributed Environment, Distributed Monitoring, Central Visibility Executive Summary

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Distributed Monitoring with Opennms Distributed Environment, Distributed Monitoring, Central Visibility Executive Summary Distributed Monitoring with OpenNMS Distributed environment, distributed monitoring, central visibility Executive Summary As the edges of enterprise networks expand with more devices, processes, services, and locations, so do the challenges of distributed monitoring. Highly distributed networks present issues such as security, privacy, reachability, and latency that make the monitoring, collection, and processing of large volumes of data difficult. This white paper explores some of the challenges to effective monitoring in distributed network environments, and solutions to address them: • Distributed data collection to monitor systems and networks that are otherwise inaccessible • Digital experience monitoring (DEM) from different perspectives to provide a better understanding of local conditions • Dynamic scaling to adapt to changing network conditions and volumes of data collected for processing and storage • Data visualization and alarm correlation to better understand the data collected and improve response times • Customization for your unique monitoring, workflow, and personnel needs 2 Introduction As the edges of enterprise networks expand with more devices, processes, services, and locations, so do the challenges of distributed monitoring. Keeping up with this type of growth presents unique challenges: how to monitor everything you need to and effectively process and interpret the volume of data such monitoring produces, given the issues (security, privacy, reachability, and latency) that highly distributed networks present. To ensure maximum uptime and optimal performance of your network, you need to be able to do the following: • Monitor and collect data from anywhere, including remote and restricted locations (page 4) • Monitor digital experience (DEM) from different perspectives(page 6) • Scale to process large volumes of data (page 8) • View data from a central location, with one tool (page 10) • Correlate and categorize alarms (page 12) • Customize your monitoring environment for your unique monitoring needs (page 13) • Delegate issues to the right people at the right time • Store data for analysis to predict and adapt 3 Monitor and collect data from anywhere Infrastructure, services, and applications located in remote sites within large, distributed corporate networks, can be difficult, if not impossible, to reach and monitor from a central location such as a data center or the Cloud. Specific challenges include firewalls, network address translation (NAT) traversal, overlapping IP address ranges, and locked-down environments. A network monitoring platform needs to be deployable in a distributed configuration to provide reach into systems and networks that would otherwise be inaccessible, while keeping the monitoring logic centralized for easier operation and administration. The OpenNMS Minion provides access to the inaccessible, with the resilience and scalability to expand monitoring capabilities as your network expands. Comprehensive fault, performance, and traffic monitoring OpenNMS Horizon/Meridian offers comprehensive fault, performance, and traffic monitoring, as well as alarm generation for your entire network from one central place. Monitoring via a host of protocols from SNMP to Netflow to gRPC and more, OpenNMS collects data on the devices, interfaces, and services you define during provisioning. It triggers alarms when it detects a problem and stores the metrics it collects, so you can analyze trends for better capacity management and network optimization. Minion acts as the eyes and ears of OpenNMS, extending its reach so it can • Operate behind firewalls and NAT • Handle overlapping address spaces with a separate Minion in each space • Provide resilient deployments with multiple Minions per location • Scale horizontal ingestion for flow, trap, and syslog messages with multiple Minions per location • Scale flow processing with OpenNMS Sentinel 4 How it works Minion is a stateless service that runs in the lightweight Karaf application container, communicating with devices and services in remote locations, while OpenNMS Core maintains state and performs the coordination and task delegation. A location defines a network area associated with a Minion: an isolated network in a data center, a department, a branch office, or a customer’s network. Minions can operate behind a firewall and/or NAT as long as they can communicate with OpenNMS via an ActiveMQ or Apache Kafka message broker or through gRPC. Being stateless means a Minion is easy to sustain, horizontally scalable, and easy to orchestrate due to the simplicity of its design. REMOTE LOCATION A MONITORED MINION ELEMENTS APM REMOTE LOCATION B MONITORED MINION ELEMENTS Sample Minion Configuration The Minion connects to an OpenNMS REST endpoint to update its configuration and for initial provisioning. The REST endpoint can be secured with HTTPS and is authenticated with a username and password. The messaging system provides a second communication channel for the actual job of monitoring. When a device sends a message such as an SNMP trap to Minion, the trap is transformed into a Minion message and pushed to the message broker. OpenNMS listens on the location queues and transforms the message from the Minion to an OpenNMS event, which appears in the central OpenNMS UI. REST MINION SNMP TRAP OpenNMS - Minion messaging Minion also checks in for messages from OpenNMS, for monitoring tasks (remote procedure calls), and sends the responses back on the response queue in the message broker. 5 Monitor digital experience from different perspectives Understanding location-specific conditions makes it easier to pinpoint not only where an issue occurs, but its impact on a user’s (or machine’s) digital experience. When your central New York location can see the availability of a service hosted in Houston as accessed from Seattle, you can identify the perspective from which an outage occurs and troubleshoot the problem more easily. Application Perspective Monitoring (APM) APM uses the Minion infrastructure to monitor a service or application (central or external) from each Minion’s location, allowing you to view the reachability of a service from many different perspectives. When a service is not responsive, OpenNMS generates an outage record that includes the corresponding perspective that identified the outage. APM can combine these perspectives to provide a holistic view of the application or service. With APM you can easily monitor the digital experience (DEM) of corporate services and applications from the perspective of many different physical, geographical, or logical locations representative of a client’s perspective. Testing availability and latency measurements from different locations provides a better understanding of local conditions. The Minion’s ability to monitor from remote locations is what makes APM possible. How It works APM implementation requires one Minion set up on your network and a simple configuration through the OpenNMS web UI. Configure one or more Minions to monitor the services from specific locations. In the OpenNMS database model, an application combines several monitored services and contains references to locations. The application also references an optional polling package that users can customize. When a remote outage occurs, OpenNMS registers the outage and includes the location from which the outage was detected. This enables you to see the perspective from which an outage occurred, and filter for local-only or remote-only outages. 6 The following diagram illustrates a sample use scenario for APM. Three locations each have a Minion: Stuttgart, Raleigh, and Ottawa. Each Minion is configured to monitor an HTTP service. If the HTTP service goes down on one of the servers, the OpenNMS UI displays the outage, including the locations (perspectives) from which the outage was detected. MINION Location: Stuttgart Monitored MINION Services Location: Raleigh MINION Management Traffic Monitoring Traffic Location: Ottawa Sample APM Configuration APM provides granularity for detecting outages. Knowing which location detected the outage indicates that the issue could lie between the perspective location and the monitored location, rather than at the monitored location itself (since other locations still see the service operating). Topology view with APM OpenNMS extends the APM feature with effective visualization. The topology view in the UI displays the applications and services for each location, and includes service status from the perspective of the location monitoring them. A table below the topology map provides detailed status: 7 Scale to process large volumes of data A network monitoring system must be able to collect and process tens of thousands of data points per second. Of course, networks are not static: the volume of data you process increases as your network expands, and changes with fluctuations in network traffic, peak/ off-peak hours, and other factors. An NMS that can scale dynamically to collect and process large volumes of data helps administrators respond to the most current issues in a timely manner. OpenNMS Minion increases the total scale of your monitoring system by distributing the data collection load instead of handling it on one server with an OpenNMS instance. Minion is stateless software that users can containerize and deploy alone or in groups in various network locations to provide a secure and simple communications infrastructure. The ability to use more than one Minion per location provides resiliency
Load more

Recommended publications
  • A Survey of Network Performance Monitoring Tools
    A Survey of Network Performance Monitoring Tools Travis Keshav -- [email protected] Abstract In today's world of networks, it is not enough simply to have a network; assuring its optimal performance is key. This paper analyzes several facets of Network Performance Monitoring, evaluating several motivations as well as examining many commercial and public domain products. Keywords: network performance monitoring, application monitoring, flow monitoring, packet capture, sniffing, wireless networks, path analysis, bandwidth analysis, network monitoring platforms, Ethereal, Netflow, tcpdump, Wireshark, Ciscoworks Table Of Contents 1. Introduction 2. Application & Host-Based Monitoring 2.1 Basis of Application & Host-Based Monitoring 2.2 Public Domain Application & Host-Based Monitoring Tools 2.3 Commercial Application & Host-Based Monitoring Tools 3. Flow Monitoring 3.1 Basis of Flow Monitoring 3.2 Public Domain Flow Monitoring Tools 3.3 Commercial Flow Monitoring Protocols 4. Packet Capture/Sniffing 4.1 Basis of Packet Capture/Sniffing 4.2 Public Domain Packet Capture/Sniffing Tools 4.3 Commercial Packet Capture/Sniffing Tools 5. Path/Bandwidth Analysis 5.1 Basis of Path/Bandwidth Analysis 5.2 Public Domain Path/Bandwidth Analysis Tools 6. Wireless Network Monitoring 6.1 Basis of Wireless Network Monitoring 6.2 Public Domain Wireless Network Monitoring Tools 6.3 Commercial Wireless Network Monitoring Tools 7. Network Monitoring Platforms 7.1 Basis of Network Monitoring Platforms 7.2 Commercial Network Monitoring Platforms 8. Conclusion 9. References and Acronyms 1.0 Introduction http://www.cse.wustl.edu/~jain/cse567-06/ftp/net_perf_monitors/index.html 1 of 20 In today's world of networks, it is not enough simply to have a network; assuring its optimal performance is key.
    [Show full text]
  • Best Practices for Network Monitoring
    White Paper Best Practices for Network Monitoring How a Network Monitoring Switch Helps IT Teams Stay Proactive 26601 Agoura Road, Calabasas, CA 91302 | Tel: 818.871.1800 | Fax: 818.871.1805 | www.ixiacom.com | 915-6509-01 Rev. B, June 2013 2 Table of Contents Monitoring Challenges in Today’s Business Environment .......................................... 4 Giving Monitoring Tools Full Visibility to the Network ............................................... 5 Easing the Path to Upgrade ........................................................................................ 6 Making Monitoring Tools Work Better ........................................................................ 7 Filtering ...................................................................................................................... 7 Load Balancing ........................................................................................................... 8 Packet De-duplication ................................................................................................ 8 Packet Trimming ........................................................................................................ 8 MPLS Stripping .......................................................................................................... 9 Keeping Network Data Secure ................................................................................... 9 Improving Productivity in IT ....................................................................................... 9 Filter Libraries
    [Show full text]
  • Large Scale Monitoring of Home Routers
    IEEE International Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications 21-23 September 2009, Rende (Cosenza), Italy Large scale monitoring of home routers S. Costas-Rodr´ıguez+,R.Mart´ınez-Alvarez´ +,F.J.Gonz´alez-Casta˜no∗+, F. Gil-Casti˜neira∗,R.Duro× +Gradiant, ETSI Telecomunicaci´on, Campus, 36310 Vigo, Spain ∗Departamento de Ingenier´ıa Telem´atica, Universidad de Vigo, Spain ×Grupo Integrado de Ingenier´ıa, Universidad de La Coru˜na, Spain Tel: +34 986 813788, fax: +34 986 812116 E-mail: {scostas,rmartinez}@gradiant.org,{javier,xil}@det.uvigo.es,[email protected] Abstract – This paper describes our experience with reception power, memory usage or uptime are useful to concurrent asynchronous monitoring of large populations of detect and fix many potential problems. end-user broadband-access routers. In our real tests we focused on home/office ADSL Despite of the wealth of research in large-scale monitoring, which assumes that it is possible to inquiry individual nodes routers, although our results are valid for any other access efficiently, end-user access routers usually have manual legacy technology. We monitored the routers of the Spanish ISP interfaces, either HTTP- or telnet-oriented. They seldom and VoIP operator Comunitel (www.comunitel.es). With offer a direct interface to other programs. Moreover, the our approach, a full monitoring cycle of 22,300 such uptime of end-user routers is unpredictable. For all these routers took less than five minutes. reasons, commercial large-scale monitoring tools such as SNMP collectors are useless. The rest of this paper is organized as follows: In This research is motivated by the fact that some telecom- section II we review the background, comprising academic munications operators do not let end-users buy their routers research and existing industrial solutions.
    [Show full text]
  • Network Monitoring Using Nagios and Autoconfiguration for Cyber Defense Competitions
    NETWORK MONITORING USING NAGIOS AND AUTOCONFIGURATION FOR CYBER DEFENSE COMPETITIONS Jaipaul Vasireddy B.Tech, A.I.E.T, Jawaharlal Nehru Technological University, India, 2006 PROJECT Submitted in partial satisfaction of the requirements for the degree of MASTER OF SCIENCE in COMPUTER SCIENCE at CALIFORNIA STATE UNIVERSITY, SACRAMENTO FALL 2009 NETWORK MONITORING USING NAGIOS AND AUTOCONFIGURATION FOR CYBER DEFENSE COMPETITIONS A Project by Jaipaul Vasireddy Approved by: __________________________________, Committee Chair Dr. Isaac Ghansah __________________________________, Second Reader Prof. Richard Smith __________________________ Date ii Student: Jaipaul Vasireddy I certify that this student has met the requirements for format contained in the University format manual, and that this Project is suitable for shelving in the Library and credit is to be awarded for the Project. __________________________, Graduate Coordinator ________________ Dr. Cui Zhang Date Department of Computer Science iii Abstract of NETWORK MONITORING USING NAGIOS AND AUTOCONFIGURATION FOR CYBER DEFENSE COMPETITIONS by Jaipaul Vasireddy The goal of the project is to monitor the services running on the CCDC (College Cyber Defense Competition) network, using Nagios which uses plugins to monitor the services running on a network. Nagios is configured by building configuration files for each machine which is usually done to monitor small number of systems. The configuration of Nagios can also be automated by using shell scripting which is generally done in an industry, where the numbers of systems to be monitored are large. Both the above methods of configuration have been implemented in this project. The project has been successfully used to know the status of each service running on the defending team’s network.
    [Show full text]
  • Deploying F5 with Nagios Open Source Network Monitoring System Welcome to the F5 and Nagios Deployment Guide
    DEPLOYMENT GUIDE Version 1.0 IMPORTANT: This guide has been archived. While the content in this guide is still valid for the products and version listed in the document, it is no longer being updated and may refer to F5 or 3rd party products or versions that have reached end-of-life or end-of-support. See https://support.f5.com/csp/article/K11163 for more information. Deploying the BIG-IP LTM with the Nagios Open Source Network Monitoring System Archived Deploying F5 with Nagios Open Source Network Monitoring System Welcome to the F5 and Nagios deployment guide. This guide provides detailed procedures for configuring F5 devices with the Nagios Open Source Network Monitoring System. Network Management Systems (NMS) provide insight to device and application performance, system stability, network trouble spots and a host of other items that affect a company's ability to provide computer network services. Recent advances in automated event correlation, data storage and retrieval, as well as open source systems that allow greater integration, have greatly improved today's Network Management Systems. The Nagios system and network monitoring application is an open source project. Nagios maintains network status information by polling objects for specific service responses. Examples of monitored network services include SMTP, DNS, HTTP, and so on. For more information on the Nagios project, see http://www.nagios.org/ or http://sourceforge.net/projects/nagios/. For further Nagios resources and information, see http://community.nagios.org/. For more information on F5 devices described in this guide, see http://www.f5.com/products/big-ip/. To provide feedback on this deployment guide or other F5 solution documents, contact us at [email protected].
    [Show full text]
  • Network Management
    Chapter 12: Network Management Jian Ren and Tongtong Li, Michigan State University Introduction 2 Network Scanners . 25 OSI Network Management Model . .3 Packet Filters . 26 Network Management Layers . .4 Wireless Network Management 26 ISO Network Management Functions 6 Cellular Networks . 27 Configuration Management . .6 Location Management for Cellular Fault Management . .6 Networks . 28 Security Management . .7 Accounting Management . .7 Policy-based Network Management 29 What Is a Policy? . 30 Performance Management . .7 Benefits of PBNM . 31 Network Management Protocols 7 Architecture of a PBNM System . 31 SNMP/SNMPv1 . .8 Conclusion 32 SNMPv2 . 13 SNMPv3 . 15 Glossary 33 Remote Network Monitoring (RMON) 23 Acknowledgements 34 Network Management Tools 24 Network Monitors . 25 Acronyms 37 Abstract: The continuous growth in scale and diversity of computer networks and network components has made network management one of the most challenging issues facing network administrators. It has become impossible to carry out network management functions without the support of automated tools and applications. In this chapter, the major network management issues, including network management requirements, functions, techniques, security, some well- known network management protocols and tools, will be discussed. Location management for the wireless cellular networks will also be briefly described. Finally, policy-based network management, which is a promising direction for the next generation of network management, will be briefly described. Keywords:
    [Show full text]
  • Comparative Analysis of Two Open Source Network Monitoring Systems: Nagios & Opennms
    Master Thesis Computer Science Thesis no: MCS-2010-37 December 2010 Comparative Analysis of two Open Source Network Monitoring Systems: Nagios & OpenNMS Muhammad Qadir Muhammad Adnan SchoolSchool of Computing of Computing BlekingeBlekinge Institute Institute of Technology of Technology SE –Box 371 520 79 Karlskrona SwedenSE – 372 25 Ronneby Sweden This thesis is submitted to the School of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Computer Science. The thesis is equivalent to 20 weeks of full time studies. Contact Information: Author(s): Muhammad Qadir Folkparksvagen 18:03 Ronneby 372 40, Sweden [email protected] Muhammad Adnan Folkparksvagen 14:11 Ronneby 372 40, Sweden [email protected] University advisor(s): Jeff Winter School of Computing School of Computing Blekinge Institute of Technology Internet : www.bth.se/com SE – 371 79 Karlskrona Phone : +46 455 38 50 00 Sweden Fax : +46 455 38 50 57 1 ABSTRACT Context: Extensive and rapid continuous growth in Internet Protocol (IP) based networks and as the result of increasing dependencies on these networks makes them extremely challenging to manage and keep them running all the time. 24/7 non-stop monitoring is important to minimize the down time of the network. For this reason dependency on automated network monitoring has been increased. Objectives: There are many tools and systems available for network monitoring. This includes expensive commercial solutions to open source products. Nagios and OpenNMS are two of the most popular systems and they are considered to be close competitors. Comparison discussions about them are very common at different forums on internet.
    [Show full text]
  • Monitoring the I2P Network Juan Pablo Timpanaro, Chrisment Isabelle, Festor Olivier
    Monitoring the I2P network Juan Pablo Timpanaro, Chrisment Isabelle, Festor Olivier To cite this version: Juan Pablo Timpanaro, Chrisment Isabelle, Festor Olivier. Monitoring the I2P network. [Research Report] RR-7844, INRIA. 2011. hal-00653136 HAL Id: hal-00653136 https://hal.inria.fr/hal-00653136 Submitted on 18 Dec 2011 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET EN AUTOMATIQUE Monitoring the I2P network Juan Pablo Timpanaro, Isabelle Chrisment, Olivier Festor N° 7844 — version 1 initial version October 2011 — revised version Décembre 2011 Domaine 2 apport de recherche ISSN 0249-6399 ISRN INRIA/RR--7844--FR+ENG Monitoring the I2P network Juan Pablo Timpanaro, Isabelle Chrisment, Olivier Festor Domaine : Algorithmique, programmation, logiciels et architectures Equipe-Projet´ madynes Rapport de recherche n° 7844 | version 1 | initial version October 2011 | revised version D´ecembre 2011 | 15 pages Abstract: We present the first monitoring study aiming to characterize the usage of the I2P network, a low-latency anonymous network based on garlic routing. We design a distributed monitoring architecture for the I2P network and show through a one week long experiment the ability of the system identify a significant number of all running applications, among web servers and file- sharing clients.
    [Show full text]
  • DMP NETWORK MONITORING™ OPTIONS Application Note
    DMP NETWORK MONITORING™ OPTIONS Application Note INTRODUCTION DMP forged the way to a new era for the security and fire alarm industry with the first UL certifications for Network Monitoring™ for burglary in 2001 and for Commercial Fire in 2002. The ability to monitor alarm signals over the Internet offers many advantages for central stations and alarm subscribers. Data transmission is secure, extremely fast, and fully supervised. DMP offers all of this value at a significant cost savings over traditional dial-up or dedicated line communication. Sending alarm and system information to a central monitoring station through the Internet is not only superior to analog communication, but it is also a communication method approved and accepted by Underwriters Laboratories (UL) for Standard Line Security and for Commercial Fire using the communication protocol patented by DMP and protected under U.S. Patents 6,040,770 and 6,255,945. Before you configure your panel or central station for Network Monitoring, determine whether it must comply with Standard Line Security or ANSI/UL 864 Commercial Fire. All DMP XT Series and XR Series panels offer built-in network options. Each DMP SCS-104 Network Line Card is capable of monitoring 3,200 accounts that require check-in or up to 65,535 accounts that do not require check-in. This document demonstrates how to take advantage of the options for Network Monitoring using existing corporate Data Networks and the Internet. SERIES DMP XR Panel SCS-1R/SCS-VR Data Network Digital Central Station Receiver SERIES DMP XR Panel THE NETWORK MONITORING™ ADVANTAGE Network Monitoring offers a significant advantage for both central stations and alarm customers in terms of savings, speed, quality of service, and security.
    [Show full text]
  • Multi-Layer Network Monitoring and Analysis
    UCAM-CL-TR-571 Technical Report ISSN 1476-2986 Number 571 Computer Laboratory Multi-layer network monitoring and analysis James Hall July 2003 15 JJ Thomson Avenue Cambridge CB3 0FD United Kingdom phone +44 1223 763500 http://www.cl.cam.ac.uk/ c 2003 James Hall This technical report is based on a dissertation submitted April 2003 by the author for the degree of Doctor of Philosophy to the University of Cambridge, King's College. Some figures in this document are best viewed in colour. If you received a black-and-white copy, please consult the online version if necessary. Technical reports published by the University of Cambridge Computer Laboratory are freely available via the Internet: http://www.cl.cam.ac.uk/TechReports/ Series editor: Markus Kuhn ISSN 1476-2986 Abstract Passive network monitoring offers the possibility of gathering a wealth of data about the traffic traversing the network and the communicating processes generating that traffic. Significant advantages include the non-intrusive nature of data capture and the range and diversity of the traffic and driving applications which may be observed. Conversely there are also associated practical difficulties which have restricted the usefulness of the technique: increasing network bandwidths can challenge the capacity of monitors to keep pace with passing traffic without data loss, and the bulk of data recorded may become unmanageable. Much research based upon passive monitoring has in consequence been limited to that using a sub-set of the data potentially available, typically TCP/IP packet headers gathered using Tcpdump or similar monitoring tools. The bulk of data collected is thereby minimised, and with the possible exception of packet filtering, the monitor's available processing power is available for the task of collection and storage.
    [Show full text]
  • Comparison of Network Monitoring Systems from Wikipedia, the Free
    Comparison of network monitoring systems From Wikipedia, the free encyclopedia Jump to: navigation, search This is a comparison of the some common network monitoring systems. [edit] Definition of Categories Product Name The name of the software, linked to its Wikipedia article. Any software listed without being linked to its article, demonstrating its notability, will be removed. IP SLAs Reports Does the software feature reports on IP SLAs? Logical Grouping Does the software support arranging the hosts or devices it monitors into user-defined groups? Trending Does the software provide trending of network data over time? Trend Prediction Does the software feature algorithms designed to predict future network statistics? Auto Discovery Can the software automatically discover hosts or network devices it is connected to? Agent Does the product rely on a software agent that must run on hosts it is monitoring, so that data can be pushed back to a central server? "Supported" means that an agent may be used, but is not mandatory. An SNMP daemon does not count as an agent. SNMP Is the software able to retrieve and report on SNMP statistics? Syslog Is the software able to receive and report on Syslogs? Plugins Is the architecture of the software based on a number of 'plugins' that provide additional functionality? Triggers/Alerts Is the software capable of detecting threshold violations in network data, and alerting the administrator in some form? WebApp Does the product run as a web-based application? • No: There is no web-based frontend for this software. • Viewing: Network data can be viewed in a graphical web-based frontend • Acknowledging: Users can interact with the software through the web-based frontend to acknowledge alarms or manipulate other notifications.
    [Show full text]
  • Automated System Monitoring
    Automated System Monitoring Josh Malone Systems Administrator [email protected] National Radio Astronomy Observatory Charlottesville, VA https://blogs.nrao.edu/jmalone 2 One night, about 8 or 9 years ago, the chiller in our DC failed. Co-worker arrive in the morning to find room was 90F ambient. Quickly set up fans to vent the room. Checked servers - found that main web server had lost both disks in its OS RAID mirror. (15k disks, ran hot) Main page was being served from memory, but the OS was freaking out. We had minimal monitoring scripts. No environment monitoring. No disk health checks. Failure caught us completely by surprise. We decided that we weren’t going to let this happen ever again. Over the next year or so we implemented 2 independent monitoring systems - one for servers/ services and one for environmentals. Set up each system to also monitor the other. WHAT IS AUTOMATED MONITORING? 7 Some sort of dedicated, automatic instrumentation to check services and/or servers Detect and report service problems, server hardware issues Usually provides a central “dashboard” to track problems Can be distributed; but still under control of a central daemon * Diferentiates it from “a bunch of scripts” used to check on things; that doesn’t have the ability to determine cause or eliminate false alarms. Automated Monitoring Workflow 8 Most packages implement this type of workflow Not all packages provide event handlers ack’ing page is important - let’s other admins know that someone is working on the problem so they don’t step on each other’s toes Monitoring Packages: Open Source • • Pandora FMS • Opsview Core • Naemon • • • • • • Captialware ServerStatus • Core • Sensu All Trademarks and Logos are property of their respective trademark or copyright holders and are used by permission or fair use for education.
    [Show full text]