DOCSLIB.ORG

Grafana and Prometheus Just Married & PMM Was Born to Monitor Your

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Grafana and Prometheus Just Married & PMM Was Born to Monitor Your 23.11.2020 Grafana and Prometheus just married & PMM was born to monitor your DBs 1 Who we are The Company > Founded in 2010 > More than 70 specialists > Specialized in the Middleware Infrastructure > The invisible part of IT > Customers in Switzerland and all over Europe Our Offer > Consulting > Service Level Agreements (SLA) > Trainings > License Management Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 2 2 23.11.2020 About me Elisa Usai Delivery Manager & Consultant +41 78 638 09 78 elisa.usai[at]dbi-services.com Elisa Usai elisetta1984 Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 3 3 Agenda Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 4 4 23.11.2020 Agenda 1.Introduction 2.Percona Monitoring and Management (PMM) tool 3.Monitor your DBs with PMM 4.Conclusion Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 5 5 Introduction 1 > Monitor a DB 2 > Prometheus > Grafana > The benefits of a marriage 3 4 Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 6 6 23.11.2020 Introduction Monitor a DB Why is the monitoring so essential? > Services Health > System Optimization > Performance Problems > Business Process Improvement > Capacity Planning What could we monitor? > Databases, Instances > Hosts > CPU, Memory, I/O, Network > Storage, Filesystems > Application Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 7 7 Introduction Monitor a DB And finally what makes the difference nowadays? > Multiple databases > Comparisons between machines, instances configuration > Features > Price Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 8 8 23.11.2020 Introduction Once upon a time… Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 9 9 Introduction Prometheus What is Prometheus? > Open source tool Event monitoring Alerting > 100% community-driven > Apache 2 licence > Go > https://prometheus.io Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 10 10 23.11.2020 Introduction Prometheus push Alertmanager notify Components alerts HTTP pull metrics Prometheus Pushgateway Grafana ServiceJobs Exporter Exporter Short-lived jobs Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 11 11 Introduction Prometheus Some features… > Multi-dimensional data model > PromQL > No reliance on distributed storage > Pull model over HTTP > Service discovery or static configuration > Graphing and dashboarding support When to use it? When not to use it? > Recording numeric time series data (metrics) > If you need 100% accuracy (per-request billing) Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 12 12 23.11.2020 Introduction Grafana What is Grafana? > Open source tool Visualize data Monitoring Performance analysis Alerting > Multiple data sources > Dashboards and plugins > Configurable > https://grafana.com Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 13 13 Introduction Grafana push Alertmanager notify Components alerts HTTP pull metrics Prometheus PromQL data visualization Pushgateway Grafana ServiceJobs Exporter Exporter Short-lived jobs Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 14 14 23.11.2020 Introduction The benefits of a marriage Requirements > Good communication > Solid roots > Common interests Advantages > Living in the same space > Tasks sharing > Stability Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 15 15 Introduction PMM Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 16 16 23.11.2020 Percona Monitoring and Management (PMM) tool 1 > Percona 2 > PMM introduction > PMM server > PMM client 3 > PMM overview 4 Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 17 17 Percona Monitoring and Management (PMM) tool Percona The story of Percona What is Percona? > Peter Zaitsev > Provider of open source DB solutions > Vadim Tkachenko > Multi-vendor environments > Worked at MySQL AB MySQL > Co-founded Percona (2006) MariaDB MongoDB Postgres > No vendor lock-in > On-premises and Cloud environments Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 18 18 23.11.2020 Percona Monitoring and Management (PMM) tool PMM introduction What is PMM? > Open source tool for monitoring managing MySQL, PostgreSQL and MongoDB performances > Developed by Percona > https://www.percona.com/doc/percona- monitoring-and-management/index.html Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 19 19 Percona Monitoring and Management (PMM) tool PMM introduction Architecture Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 20 20 23.11.2020 Percona Monitoring and Management (PMM) tool PMM server > PMM server as appliance via: 1. Docker image 2. Open Virtual Appliance 3. Amazon Machine Image > Requirements (for option 1) Docker >= 1.12.6 Storage: > 1GB Memory: > 2GB Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 21 21 Percona Monitoring and Management (PMM) tool PMM client > PMM client Agents Exporters > https://www.percona.com/downloads/pmm/ DEB packages for Ubuntu RPM packages for CentOS Generic tarballs Source code > Requirements Different host name Storage: > 100MB Good connection to PMM server Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 22 22 23.11.2020 Percona Monitoring and Management (PMM) tool PMM overview Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 23 23 Percona Monitoring and Management (PMM) tool PMM overview Dashboards > Default Dashboards Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 24 24 23.11.2020 Percona Monitoring and Management (PMM) tool PMM overview Dashboards > Possibility to Create custom Dashboards Import Dashboards Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 25 25 Percona Monitoring and Management (PMM) tool PMM overview Metrics Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 26 26 23.11.2020 Percona Monitoring and Management (PMM) tool PMM overview PMM Settings > Metrics resolution, retention time, telemetry, check for updates, security tool Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 27 27 Percona Monitoring and Management (PMM) tool PMM overview PMM Settings > Diagnostic Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 28 28 23.11.2020 Percona Monitoring and Management (PMM) tool PMM overview Alerting Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 29 29 Percona Monitoring and Management (PMM) tool PMM overview Alerting … using Grafana Alerting feature … using External Prometheus Alertmanager > Integrated into PMM Server > More sophisticated alert rules > Simpler to set-up > Easier to manage many hosts > Configured in the Alert Tab of any dashboard > Maybe integrated into PMM in next releases! graph panel https://grafana.com/docs/grafana/latest/alerting/al https://prometheus.io/docs/alerting/alertmanager/ erts-overview/ https://www.percona.com/blog/2017/02/02/pmm- alerting-with-grafana-working-with-templated- dashboards/ Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 30 30 23.11.2020 Percona Monitoring and Management (PMM) tool PMM overview Alerting … using Grafana Alerting feature … using External Prometheus Alertmanager > Integrated into PMM Server > More sophisticated alert rules > Simpler to set-up > Easier to manage a large number of hosts > Configured in the Alert Tab of any dashboard > Maybe integrated into PMM in next releases! graph panel https://grafana.com/docs/grafana/latest/alerting/ru https://prometheus.io/docs/alerting/alertmanager/ les/ https://www.percona.com/blog/2017/02/02/pmm- alerting-with-grafana-working-with-templated- dashboards/ Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 31 31 Percona Monitoring and Management (PMM) tool PMM overview Prometheus Alertmanager Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 32 32 23.11.2020 Percona Monitoring and Management (PMM) tool PMM overview User Management > Users > Roles Admin Editor Viewer Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 33 33 Percona Monitoring and Management (PMM) tool PMM overview Query Analytics Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 34 34 23.11.2020 Monitor your DBs with PMM 1 > MySQL/MariaDB Monitoring 2 > Galera Cluster Monitoring > Postgres Monitoring > Oracle Monitoring 3 > and much more… 4 Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 35 35 Percona Monitoring and Management (PMM) tool MySQL/MariaDB Monitoring > PMM user in MySQL 8.0 default_authentication_plugin = caching_sha2_password SHA-256 authentication not supported by PMM PMM user: mysql_native_password! > Adding MySQL/MariaDB service monitoring # pmm-admin add mysql --query-source=slowlog --username=pmm --password=manager
Load more

Recommended publications
  • SQL Server 2017 on Linux Quick Start Guide | 4
    SQL Server 2017 on Linux Quick Start Guide Contents Who should read this guide? ........................................................................................................................ 4 Getting started with SQL Server on Linux ..................................................................................................... 5 Why SQL Server with Linux? ..................................................................................................................... 5 Supported platforms ................................................................................................................................. 5 Architectural changes ............................................................................................................................... 6 Comparing SQL on Windows vs. Linux ...................................................................................................... 6 SQL Server installation on Linux ................................................................................................................ 8 Installing SQL Server packages .................................................................................................................. 8 Configuration capabilities ....................................................................................................................... 11 Licensing .................................................................................................................................................. 12 Administering and
    [Show full text]
  • Latest Pgwatch2 Docker Image with Built-In Influxdb Metrics Storage DB
    pgwatch2 Sep 30, 2021 Contents: 1 Introduction 1 1.1 Quick start with Docker.........................................1 1.2 Typical “pull” architecture........................................2 1.3 Typical “push” architecture.......................................2 2 Project background 5 2.1 Project feedback.............................................5 3 List of main features 7 4 Advanced features 9 4.1 Patroni support..............................................9 4.2 Log parsing................................................ 10 4.3 PgBouncer support............................................ 10 4.4 Pgpool-II support............................................. 11 4.5 Prometheus scraping........................................... 12 4.6 AWS / Azure / GCE support....................................... 12 5 Components 13 5.1 The metrics gathering daemon...................................... 13 5.2 Configuration store............................................ 13 5.3 Metrics storage DB............................................ 14 5.4 The Web UI............................................... 14 5.5 Metrics representation.......................................... 14 5.6 Component diagram........................................... 15 5.7 Component reuse............................................. 15 6 Installation options 17 6.1 Config DB based operation....................................... 17 6.2 File based operation........................................... 17 6.3 Ad-hoc mode............................................... 17 6.4 Prometheus
    [Show full text]
  • Grafana and Mysql Benefits and Challenges 2 About Me
    Grafana and MySQL Benefits and Challenges 2 About me Philip Wernersbach Software Engineer Ingram Content Group https://github.com/philip-wernersbach https://www.linkedin.com/in/pwernersbach 3 • I work in Ingram Content Group’s Automated Print On Demand division • We have an automated process in which publishers (independent or corporate) request books via a website, and we automatically print, bind, and ship those books to them • This process involves lots of hardware devices and software components 4 The Problem 5 The Problem “How do we aggregate and track metrics from our hardware and software sources, and display those data points in a graph format to the end user?” à Grafana! 6 Which data store should we use with Grafana? ▸ Out of the box, Grafana supports Elasticsearch, Graphite, InfluxDB, KairosDB, OpenTSDB 7 Which data store should we use with Grafana? ▸ We compared the options and tried InfluxDB ▸ There were several sticking points with InfluxDB, both technical and organizational, that caused us to rule it out 8 Which data store should we use with Grafana? ▸ We already have a MySQL cluster deployed, System Administrators and Operations know how to manage it ▸ Decided to go with MySQL as a data store for Grafana 9 The Solution: Ingram Content’s Grafana-MySQL Integration 10 ▸ Written in Nim ▸ Emulates an InfluxDB server ▸ Connects to an existing The Integration MySQL server ▸ Protocol compatible with InfluxDB 0.9.3 ▸ Acts as a proxy that converts the InfluxDB protocol to the MySQL protocol and vice- versa 11 Grafana The Integration
    [Show full text]
  • Monitoring Container Environment with Prometheus and Grafana
    Matti Holopainen Monitoring Container Environment with Prometheus and Grafana Metropolia University of Applied Sciences Bachelor of Engineering Information and Communication Technology Bachelor’s Thesis 3.5.2021 Abstract Tekijä Matti Holopainen Otsikko Monitoring Container Environment with Prometheus and Grafana Sivumäärä Aika 50 sivua 3.5.2021 Tutkinto Insinööri (AMK) Tutkinto-ohjelma Tieto- ja viestintätekniikka Ammatillinen pääaine Ohjelmistotuotanto Ohjaajat Nina Simola, Projektipäällikkö Auvo Häkkinen, Yliopettaja Insinöörityön tavoitteena oli oppia pystyttämään monitorointijärjestelmä konttiympäristön re- surssien käytön seuraamista, monitorointia ja analysoimista varten. Tavoitteena oli helpot- taa monitorointijärjestelmän käyttöönottoa. Työ tehtiin käytettävien ohjelmistojen dokumen- taation ja käytännön tekemisellä opittujen asioiden pohjalta. Insinöörityön alussa käytiin läpi työssä käytettyjä teknologioita. Tämän jälkeen käytiin läpi monitorointi järjestelmän konfiguraatio ja käyttöönotto. Seuraavaksi tutustuttiin PromQL-ha- kukieleen, jonka jälkeen näytettiin kuinka pystyttää valvontamonitori ja hälytykset sähköpos- timuistutuksella. Työn lopussa käydään läpi kuinka monitorointijärjestelmässä saatua dataa analysoidaan ja mietitään miten monitorointijärjestelmää voisi parantaa. Keywords Monitorointi, Kontti, Prometheus, Grafana, Docker Abstract Author Matti Holopainen Title Monitoring Container Environment with Prometheus and Grafana Number of Pages Date 50 pages 3.5.2021 Degree Bachelor of Engineering Degree Programme Information
    [Show full text]
  • Eventindex Monitoring
    BigData tools for the monitoring of the ATLAS EventIndex Evgeny Alexandrov1, Andrei Kazymov1, Fedor Prokoshin2, on behalf of the ATLAS collaboration 1Joint Institute for Nuclear Research, Dubna, Russia. 2Centro Científico Tecnológico de Valparaíso-CCTVal, Universidad Técnica Federico Santa María. GRID Conference at JINR 12 September 2018 Introduction • The EventIndex is the complete catalogue of all ATLAS events, keeping the references to all files that contain a given event in any processing stage. • The ATLAS EventIndex collects event information from data both at CERN and Grid sites. • It uses the Hadoop system to store the results, and web services to access them. • Its successful operation depends on a number of different components. • Each component has completely different sets of parameters and states and requires a special approach. Monitoring Components Prodsys EIOracle ??? Event Picking Tests XML ORACLE? Old Monitoring System based on Kibana Disadvantages of Kibana Slow dashboard retrieving time: - for two days period: 15 seconds; - for 7 days period: 1 minute 30 seconds; - for a longer periods: it may take tens of minutes and eventually get stuck. Not very comfortable way of editing the dashboard’s page Grafana Grafana is one of the most popular packages for visualizing monitoring data. Uses modern technologies: - back-end is written using Go programming language; - front-end is written on typescript and uses angular approach. The following datasources are officially supported: Graphite InfluxDB MySQL Elasticsearch OpenTSDB Postgres CloudWatch Prometheus Microsoft SQL Server (MSSQL) InfluxDB InfluxDB is InfluxData's open source time series database designed to handle high write and query loads. Uses modern technologies: - it is written on GO; - It has the possibility of working in cluster mode; - availability of libraries for a large number of programming languages (Python, JavaScript, PHP, Haskell and others); - SQL-like query language, with which you can perform various operations with time series (merging, splitting).
    [Show full text]
  • Monitoring with Influxdb and Grafana
    Monitoring with InfluxDB and Grafana Andrew Lahiff STFC RAL ! HEPiX 2015 Fall Workshop, BNL Introduction Monitoring at RAL • Like many (most?) sites, we use Ganglia • have ~89000 individual metrics • What’s wrong with Ganglia? Problems with ganglia • Plots look very dated Problems with ganglia • Difficult & time-consuming to make custom plots • currently use long, complex, messy Perl scripts • e.g. HTCondor monitoring > 2000 lines Problems with ganglia • Difficult & time-consuming to make custom plots • Ganglia UI for making customised plots is restricted & doesn’t give good results Problems with ganglia • Ganglia server has demanding host requirements • e.g. we store all rrds in a RAM disk • have problems if trying to use a VM • Doesn’t handle dynamic resources well • Occasional problems with gmond using too much memory, affecting other processes on machines • Not really suitable for Ceph monitoring A possible alternative • InfluxDB + Grafana • InfluxDB is a time-series database • Grafana is a metrics dashboard • originally a fork of Kibana • can make plots of data from InfluxDB, Graphite, others… • Very easy to make (nice) plots • Easy to install InfluxDB • Time series database written in Go • No external dependencies • SQL-like query language • Distributed • can be run as a single node • can be run as a cluster for redundancy & performance (not suitable for production use yet) • Data can be written in using REST, or an API (e.g. Python) • or from collectd or graphite InfluxDB • Data organised by time series, grouped together into databases • Time series have zero to many points • Each point consists of: • time - the timestamp • a measurement (e.g.
    [Show full text]
  • Solution Deploying Elasticsearch on Kubernetes Using
    Solution Guide Deploying Elasticsearch on Kubernetes using OpenEBS Contents Part 1 - Before starting Part 2 - Preconfiguration Part 3 - Getting Started with OpenEBS Part 4 - Installing Kudo Operator Part 5 - Installing ElasticSearch using Kudo Part 6 - Installing Kibana Part 7 - Installing Fluentd-ES Part 8 - Monitoring Elastic search 01 www.mayadata.io Overview Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is built on Apache Lucene and was first released in 2010 by Elasticsearch N.V. (now known as Elastic). Known for its simple REST APIs, distributed nature, speed, and scalability, Elasticsearch is the central component of the Elastic Stack, a set of free and open tools for data ingestion, enrichment, storage, analysis, and visualization. Commonly referred to as the ELK Stack (after Elasticsearch, Logstash, and Kibana), the Elastic Stack now includes a rich collection of lightweight shipping agents known as Beats for sending data to Elasticsearch.[1]. This guide explains the basic installation for Elasticsearch operators on OpenEBS Local PV devices using KUDO. We will be installing Fluentd and Kibana to form the EFK stack. The guide will also provide a way to monitor the health of Elasticsearch using Prometheus and Grafana. Before starting You require an existing Kubernetes cluster. Kubernetes provides platform abstraction, cloud-native software runs, and behaves the same way on a managed Kubernetes service like AWS EKS, Google Cloud GKE, Microsoft AKS, DigitalOcean Kubernetes Service, or self-managed based on Red Hat OpenShift and Rancher. You can also use kubeadm, kubespray, minikube.
    [Show full text]
  • The Open-‐Source Monitoring Landscape
    The Open-Source Monitoring Landscape Michael Merideth Sr. Director of IT, VictorOps [email protected], @vo_mike My History and Background • Working in IT since the mid 90’s • Mostly at startups in the Colorado Front Range area • Along for the ride during the “dot com boom” • Build my career using open-source tools Since the 90’s now, there’s been a sharp division in tools and methodology between the enterprise space and the startup and small business communi;es. Obviously, smaller businesses, especially in the tech sector, were early and eager adopters of open- source technology, and much quicker to learn to rely on open-source tool chains in business-cri;cal applica;ons. Up at the enterprise level, at the public companies, they’re only now fully-embracing Linux as a business-cri;cal tool, and I think that’s largely because “the enterprise” is star;ng to be defined by companies that either came up in the dot com era, like Google, or built the dot com era, like Sun, or Dell, or let’s say RedHat. So, the “enterprise” had their toolchain, built on commercial solu;ons like HPUX and OpenView and so on, and the startup community, the “dot com” community had a completely different toolchain, based on Linux, based on open standards and protocols, built with open-source components like GNU, and Apache, and ISC Bind and so on. I’m lucky enough that I’ve been able to spend my career in the startup sphere, working with that open-source toolchain. I started working in IT in the mid 90’s in the Colorado front range, and I’ve spent my ;me since then working for and consul;ng at early-stage startups and other “non enterprise” shops.
    [Show full text]
  • Monitoring Open Source Databases with Icinga
    PGConf EU | Warsaw | 26.10.2017 Monitoring Open Source Databases with Icinga • Blerim Sheqa • Product Manager • Working @netways • @bobapple Introduction to Icinga2 Quick Poll Icinga is a scalable and extensible monitoring system which checks the availability of your resources, notifies users of outages and provides extensive metricsmetrics. • Multithreaded Perfdata Graphite Livestatus • Modular Features • Zone support Notify InfluxDB • Secure Agent Checker Gelf IDO • No Nagios® • Availability and scaling zones • Automatic redistribution of checks • Zones for multitenancymultitenancy environments Monitoring Databases • MySQLish • PostgreSQL • MongoDB • Firebird • SQLite • check_mongodb_py • connect • last_flush_time • connections • replset_state • • replicaton_lag index_miss_ratio • collections • memory • database_size • memory_mapped • database_indexes • lock • replica_primary • flushing https://github.com/mzupan/nagios-plugin-mongodb • check_postgres.pl • archive_ready • disabled_triggers • autovac_freeze • disk_space • fsm_pages • backends • prepared_txns • bloat • query_runtime • checkpoint • query_time • cluster_id • replicate_row • commitratio • same_schema • connection • sequence • custom_query • settings_checksum https://bucardo.org/check_postgres/check_postgres.pl.html • check_mysql_health • connection-time • bufferpool-wait-free • uptime • log-waits • threads-connected • tablecache-hitrate • threadcache-hitrate • table-lock-contention • • qcache-hitrate index-usage • tmp-disk-tables • qcache-lowmem-prunes • slow-queries • bufferpool-hitrate
    [Show full text]
  • GARR Cloud Monitoring
    GARR Cloud monitoring Claudio Pisa - GARR EAPConnect Workshop II 2019-11-22 Rome Monitoring - day 0 (one year ago) ● Different monitoring systems: ○ Nagios ■ OpenStack services ○ Zabbix ■ Ceph ■ Hardware sensors ● Some systems not monitored Claudio Pisa // EAPConnect Workshop II 2019 - Rome // 2019-11-22 2 Architecture - day 0 Cloud Nagios Nagios Web Infrastructural interface services Cloud Zabbix Web Infrastructure Zabbix interface + Ceph Claudio Pisa // EAPConnect Workshop II 2019 - Rome // 2019-11-22 3 Architecture - day 0 Cloud Nagios Nagios Web Infrastructural interface services Cloud Zabbix Web Infrastructure Zabbix interface + Ceph Dell Chassis Web Bare Metal Dell tools interface GINS GINS Web Network (SNMP) interface Claudio Pisa // EAPConnect Workshop II 2019 - Rome // 2019-11-22 4 Objective Objective: State of the art: build a unified Grafana comprehensive dashboard Claudio Pisa // OpenInfra Days 2019 - Rome // 2019-10-03 5 Considerations ● Existing monitoring systems seem to be doing their job well ○ the right tool for the right job ● What is missing is just a single viewpoint ● Grafana seems to have what we need Claudio Pisa // EAPConnect Workshop II 2019 - Rome // 2019-11-22 6 Grafana ● Grafana is a platform for data visualization, querying and alerting ● Several pluggable data sources: ○ Zabbix ○ PNP (Nagios) ○ Prometheus ○ Gnocchi ○ Monasca ○ JSON (general purpose) ○ MySQL / PostgreSQL (general purpose) ● Data from heterogeneous sources can be mixed in the same dashboard Claudio Pisa // EAPConnect Workshop II 2019 - Rome
    [Show full text]
  • Training Telegraf, Influxdb, Grafana
    Telegraf, InfluxDB, Grafana Training Still Using MRTG? n Simple all in one SNMP monitoring software n Send SNMP requests n Store replies into text-based database n Generate images and HTML pages n Measures two values (input / output) n Collects data every five minutes n Static pages n RRDTools, Cacti 2 2019/11/15 Presented by Warren Chang Why should not use MRTG anymore n Pull-based n Mainly SNMP, 2-D data n Not scalable n Static image, web page n Five minutes interval n Difficult to customize n No modern alert mechanism n No distributed databases 3 2019/11/15 Presented by Warren Chang What we need n Collect data n Store and process data n Visualize data n Monitoring and alert n Telemetry data more than SNMP n What is telemetry data? n Getting more important n Big Data to AI 4 2019/11/15 Presented by Warren Chang Modern Data Monitoring and Processing Model 5 2019/11/28 Presented by Warren Chang Modern Data Monitoring and Processing Model Telegraf Grafana InfluxDB Kapacitor InfluxDB Prometheus 6 2019/11/28 Presented by Warren Chang TICK Architecture 7 2019/11/28 Presented by Warren Chang Products Telegraf InfluxDB Chronograf Kapacitor Agents for collecting Streaming data and reporting metrics Time Series Database Data visualization processing enging and events Graphite Grafana Logstash Kafka Prometheus Kibana Prometheus Grafana OpenTSDB Datadog Fluentd Prometheus Elasticsearch Splunk 8 2019/11/28 Presented by Warren Chang Why InfluxDB? source: https://db-engines.com/en/ranking/time+series+dbms 9 2019/11/28 Presented by Warren Chang Why InfluxDB, Telegraf, Grafana InfluxDB Telgraf Grafana n High performance, written in Go n High performance, written in Go n Rich data sources support n Native HTTP API n Collect and send almost all kinds n InfluxDB, Prometheus, MySQL of data n Powerful SQL-like language n Templating n 200+ input, output plugins n Supports logs n Alerts n Down sampling n Plugin, App 10 2019/11/28 Presented by Warren Chang Time Series Data Tags Measurement Stock_Price Name=Apple Inc.
    [Show full text]
  • Cern Controls Open Source Monitoring System F
    17th Int. Conf. on Acc. and Large Exp. Physics Control Systems ICALEPCS2019, New York, NY, USA JACoW Publishing ISBN: 978-3-95450-209-7 ISSN: 2226-0358 doi:10.18429/JACoW-ICALEPCS2019-MOPHA085 CERN CONTROLS OPEN SOURCE MONITORING SYSTEM F. Locci*, F. Ehm, L. Gallerani, J. Lauener, J. Palluel, R. Voirin CERN, Geneva, Switzerland Abstract The CERN accelerator controls infrastructure spans sev- eral thousands of computers and devices used for Acceler- ator control and data acquisition. In 2009, a fully in-house, CERN-specific solution was developed (DIAMON) to monitor and diagnose the complete controls infrastructure. The adoption of the solution by a large community of users, followed by its rapid expansion, led to a final product that became increasingly difficult to operate and maintain. This was predominantly due to the multiplicity and redundancy of services, centralized management of data acquisition and visualization software, its complex configuration and its intrinsic scalability limits. At the end of 2017, a com- pletely new monitoring system for the beam controls infra- structure was launched. The new "COSMOS" system was developed with two main objectives in mind: firstly, de- tecting instabilities and preventing breakdowns of the con- trol system infrastructure. Secondly, providing users with Figure 1: Main types of control system hosts. a more coherent and efficient solution for development of their specific data monitoring agents and related dash- OBJECTIVES AND SCOPE boards. This paper describes the overall architecture of Reviewing the existing system and evaluating major COSMOS, focusing on the conceptual and technological products in the monitoring field (collectd, Icinga2, Zabbix, choices for the system.
    [Show full text]