DOCSLIB.ORG

Viewing MQ Configurations with Grafana

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Viewing MQ Configurations with Grafana Viewing MQ configurations with Grafana Mark E Taylor Published on 30/09/2019 / Updated on 08/10/2019 This post shows how you can use Grafana to selectively view information about your MQ configuration. Which may sound a little odd. Grafana’s strength is primarily to show statistics and metrics in pretty graphs. So why would we want to use it to look at queue definitions? The answer is that you usually would not! There are many more appropriate tools for displaying and updating the queue manager configuration – even the MQ Explorer or MQ Console are better. But there may be times when a limited set of information may be desirable, so you can link from a graph to a different view, within the same tool. But another important aspect that I hope this shows is the power of a common data format. The techniques I’ll show here could be used to combine a variety of different tools, and perhaps this will give you some ideas. Background Over the last few years I’ve written several articles about showing MQ statistics in Grafana. I was recently giving a demonstration of looking at z/OS data (you can see a video of that demonstration here) and was asked if it was possible to drill down from one of the graphs to see queues which had a particular configuration. The short answer to that question was “No” but it got me thinking about the feasibility. And I saw a completely independent question about using the dmpmqcfg program to obtain the MQ configuration in a format other than MQSC, to make it easier to work with programmatically. From those two prompts, I came up with some simple scripts to extract MQ object definitions and then use them to build tables in Grafana. The pieces of the idea Grafana includes a number of plugins which know how to query a variety of databases in order to extract information that can be formatted as graphs or tables. Most of the plugins are for time-series databases such as Prometheus and InfluxDB, specialising in storing metrics. But one of the plugins knows how to issue arbitrary SQL queries to a MySQL database. A MySQL database can work with data stored in JSON format. And MQ V9.1.3 included enhancements to the REST Admininistration API that outputs responses from MQSC commands as JSON. So putting these pieces together in the correct order, we can • Issue commands like “DISPLAY QLOCAL” via the REST Admin interface • Store the output in a MySQL database • Write Grafana dashboard queries that use SQL to extract data that will be displayed as a table Tools used Apart from normal shell script stuff, this demonstration uses the following tools: • MQ V9.1.3 – version needed for the JSON-formatted output • Grafana – of course • MySQL – the database • mysqlsh – a separately installed utility progam to simplify data import • curl – to drive the MQ web server • jq – to filter JSON data Extracting the queue manager configuration I wrote a script that looks a little like an MQSC command line, that builds the JSON version of the command and executes it via curl. This makes it simple to issue several similar commands and get the output in a state ready for import to MySQL. The script, which I called mqscJ, is invoked like this: mqscJ QM1 DISPLAY QLOCAL "*" The output from this script looks like { "commandResponse": [ { "completionCode": 0, "parameters": { "acctq": "QMGR", "altdate": "2019-08-21", "alttime": "15.32.28", "boqname": "", "bothresh": 0, "clchname": "", "clusnl": "", "cluster": "", "clwlprty": 0, "clwlrank": 0, "clwluseq": "QMGR", "crdate": "2019-08-21", "crtime": "15.32.28", ... For my environment, I didn’t need to make the webserver URL a variable – it is hardcoded in the script. A second script, dmpmqsql, calls this first script for all of the queue managers and object types we are interested in. It uses the jq command to do a little bit of reformatting of the output. One thing that is missing from the output of the REST API is the name of the queue manager. That’s perfectly understandable, as you know to which queue manager you’ve issued the command. But when the data is being stored for multiple queue managers, the name needs to be added back in. And jq makes that easy. I also strip return codes, and rename the parameters block to something much shorter as we may be typing it many times in the SQL queries. After jq has done its job, the output looks like { "queueManager": "QM2", "a": { "acctq": "QMGR", "altdate": "2019-01-26", "alttime": "10.41.27", "boqname": "", "bothresh": 0, "clchname": "", "clusnl": "", "cluster": "", "clwlprty": 0, "clwlrank": 0, "clwluseq": "QMGR", "crdate": "2019-01-26", ... Importing JSON into MySQL The mysqlsh utility makes it very simple to import JSON files into a MySQL database. If a table of the given name does not already exist, it creates one containing just two columns. The entire body of a JSON object goes into one of those columns called doc. More complex imports are possible, to allow for more flexible table definitions, but this works well enough for our purposes. For this process, a single database (MQCFG) will contain multiple tables, one for each of the MQ object types. In this case, I decided to separate the different subtypes of queue so we get individual tables for QLOCAL, QMODEL etc as well as one for the QMGR attributes and one for all the CHL definitions. After the import, we can refer to elements of the JSON object in SQL queries based on the real SQL column (“doc”) and the JSON field (eg “name”). The syntax used by MySQL to point at these looks like doc->>"$.name". The double arrow in this reference tells MySQL to remove quote characters surrounding strings. Creating a Grafana dashboard I chose to make a very simple dashboard, consisting of a single panel showing a table of local queues along with their alteration time. The table can be filtered by queue manager name, and by whether the queue is a transmission queue or not. Both the contents of the table, and the filtering, are managed by SQL queries. Grafana table showing queues The first step is to add a MySQL datasource to Grafana. Once you have tested that the connection works, you can start using the queries. Defining a MySQL datasource to Grafana Variables Variables are the way of choosing the subsets of data that are going to be shown. They will appear at the top of the dashboard with dropdowns to allow selection. Once you have created a new empty dashboard, then you use the Settings menus to add variables to it. Defining dashboard variables Here, you can see two variables defined, both of which have a similar pattern. The queue manager variable gets its possible values from a SELECT on one of the tables. There is a further ALL value possible. In the Grafana definition of the variable, the Custom all value is set to 'ALL'. The quotes around this string are important as this piece of text is later inserted into a SQL query. As no real queue manager can have quotes in its name, this should not cause any confusion. Similarly, the Usage variable gets defined – while I could have coded the only possible two values directly for this variable, I decided to follow the queue manager pattern and dynamically get the values via a query. Variables in Grafana have the ‘$’ prefix. Defining the QMGR variable Tables Now that we have the variables, we can use them as filters in the query that extracts the data. Add a panel to the dashboard, and say that it is going to be displayed as a table, using the MySQL datasource. The columns from the SQL query become the columns in the displayed table, including the column headers. The SQL uses the variables defined previously to construct the query. Defining the panel The full SQL in this query is copied here: SELECT doc->>"$.queueManager" as 'QMgr', doc->>"$.a.queue" as 'Queue', doc->>"$.a.usage" as 'Usage', concat(doc->>"$.a.altdate",":",doc->>"$.a.alttime") as "Alter Time" FROM MQCFG.QLOCAL WHERE doc->>"$.a.queue" NOT LIKE 'SYSTEM.%' AND ( CASE WHEN $USAGE = 'ALL' THEN doc->>"$.a.usage" LIKE '%' ELSE doc->>"$.a.usage" = $USAGE END ) AND ( CASE WHEN $QMGR = 'ALL' THEN doc->>"$.queueManager" LIKE '%' ELSE doc->>"$.queueManager" = $QMGR END ) ORDER by Queue; Getting the scripts The mqscJ and dmpmqsql scripts, along with an example Grafana dashboard are available from a GitHub gist here. It didn’t seem worth creating a full repository for these small pieces of code. Possible Enhancements The scripts are written only to the level needed to demonstrate the principles. They do not do much error checking, especially of the responses from the REST API. They also have hardcoded queue manager names and the URL for a single web server. The JSON processing in MySQL is not going to be as efficient as regular SQL columns and indexes. If the extracted configuration was expected to be used more heavily, I’d probably look at adding some of the JSON fields as standard typed columns. The MySQL configuration is also hardcoded to access a local database instance. The scripts have the userid and password fixed in them too. Constructing this dashboard was a very manual process. I might want to programmatically build the Grafana dashboard to more easily show a few more variables and tables. Summary Building general purpose data queries for Grafana to look at the MQ configuration is not a natural fit. I’d expect people using these tools to only define a few standard tables to fit common situations they might hit.
Load more

Recommended publications
  • SQL Server 2017 on Linux Quick Start Guide | 4
    SQL Server 2017 on Linux Quick Start Guide Contents Who should read this guide? ........................................................................................................................ 4 Getting started with SQL Server on Linux ..................................................................................................... 5 Why SQL Server with Linux? ..................................................................................................................... 5 Supported platforms ................................................................................................................................. 5 Architectural changes ............................................................................................................................... 6 Comparing SQL on Windows vs. Linux ...................................................................................................... 6 SQL Server installation on Linux ................................................................................................................ 8 Installing SQL Server packages .................................................................................................................. 8 Configuration capabilities ....................................................................................................................... 11 Licensing .................................................................................................................................................. 12 Administering and
    [Show full text]
  • Latest Pgwatch2 Docker Image with Built-In Influxdb Metrics Storage DB
    pgwatch2 Sep 30, 2021 Contents: 1 Introduction 1 1.1 Quick start with Docker.........................................1 1.2 Typical “pull” architecture........................................2 1.3 Typical “push” architecture.......................................2 2 Project background 5 2.1 Project feedback.............................................5 3 List of main features 7 4 Advanced features 9 4.1 Patroni support..............................................9 4.2 Log parsing................................................ 10 4.3 PgBouncer support............................................ 10 4.4 Pgpool-II support............................................. 11 4.5 Prometheus scraping........................................... 12 4.6 AWS / Azure / GCE support....................................... 12 5 Components 13 5.1 The metrics gathering daemon...................................... 13 5.2 Configuration store............................................ 13 5.3 Metrics storage DB............................................ 14 5.4 The Web UI............................................... 14 5.5 Metrics representation.......................................... 14 5.6 Component diagram........................................... 15 5.7 Component reuse............................................. 15 6 Installation options 17 6.1 Config DB based operation....................................... 17 6.2 File based operation........................................... 17 6.3 Ad-hoc mode............................................... 17 6.4 Prometheus
    [Show full text]
  • Grafana and Mysql Benefits and Challenges 2 About Me
    Grafana and MySQL Benefits and Challenges 2 About me Philip Wernersbach Software Engineer Ingram Content Group https://github.com/philip-wernersbach https://www.linkedin.com/in/pwernersbach 3 • I work in Ingram Content Group’s Automated Print On Demand division • We have an automated process in which publishers (independent or corporate) request books via a website, and we automatically print, bind, and ship those books to them • This process involves lots of hardware devices and software components 4 The Problem 5 The Problem “How do we aggregate and track metrics from our hardware and software sources, and display those data points in a graph format to the end user?” à Grafana! 6 Which data store should we use with Grafana? ▸ Out of the box, Grafana supports Elasticsearch, Graphite, InfluxDB, KairosDB, OpenTSDB 7 Which data store should we use with Grafana? ▸ We compared the options and tried InfluxDB ▸ There were several sticking points with InfluxDB, both technical and organizational, that caused us to rule it out 8 Which data store should we use with Grafana? ▸ We already have a MySQL cluster deployed, System Administrators and Operations know how to manage it ▸ Decided to go with MySQL as a data store for Grafana 9 The Solution: Ingram Content’s Grafana-MySQL Integration 10 ▸ Written in Nim ▸ Emulates an InfluxDB server ▸ Connects to an existing The Integration MySQL server ▸ Protocol compatible with InfluxDB 0.9.3 ▸ Acts as a proxy that converts the InfluxDB protocol to the MySQL protocol and vice- versa 11 Grafana The Integration
    [Show full text]
  • Monitoring Container Environment with Prometheus and Grafana
    Matti Holopainen Monitoring Container Environment with Prometheus and Grafana Metropolia University of Applied Sciences Bachelor of Engineering Information and Communication Technology Bachelor’s Thesis 3.5.2021 Abstract Tekijä Matti Holopainen Otsikko Monitoring Container Environment with Prometheus and Grafana Sivumäärä Aika 50 sivua 3.5.2021 Tutkinto Insinööri (AMK) Tutkinto-ohjelma Tieto- ja viestintätekniikka Ammatillinen pääaine Ohjelmistotuotanto Ohjaajat Nina Simola, Projektipäällikkö Auvo Häkkinen, Yliopettaja Insinöörityön tavoitteena oli oppia pystyttämään monitorointijärjestelmä konttiympäristön re- surssien käytön seuraamista, monitorointia ja analysoimista varten. Tavoitteena oli helpot- taa monitorointijärjestelmän käyttöönottoa. Työ tehtiin käytettävien ohjelmistojen dokumen- taation ja käytännön tekemisellä opittujen asioiden pohjalta. Insinöörityön alussa käytiin läpi työssä käytettyjä teknologioita. Tämän jälkeen käytiin läpi monitorointi järjestelmän konfiguraatio ja käyttöönotto. Seuraavaksi tutustuttiin PromQL-ha- kukieleen, jonka jälkeen näytettiin kuinka pystyttää valvontamonitori ja hälytykset sähköpos- timuistutuksella. Työn lopussa käydään läpi kuinka monitorointijärjestelmässä saatua dataa analysoidaan ja mietitään miten monitorointijärjestelmää voisi parantaa. Keywords Monitorointi, Kontti, Prometheus, Grafana, Docker Abstract Author Matti Holopainen Title Monitoring Container Environment with Prometheus and Grafana Number of Pages Date 50 pages 3.5.2021 Degree Bachelor of Engineering Degree Programme Information
    [Show full text]
  • Eventindex Monitoring
    BigData tools for the monitoring of the ATLAS EventIndex Evgeny Alexandrov1, Andrei Kazymov1, Fedor Prokoshin2, on behalf of the ATLAS collaboration 1Joint Institute for Nuclear Research, Dubna, Russia. 2Centro Científico Tecnológico de Valparaíso-CCTVal, Universidad Técnica Federico Santa María. GRID Conference at JINR 12 September 2018 Introduction • The EventIndex is the complete catalogue of all ATLAS events, keeping the references to all files that contain a given event in any processing stage. • The ATLAS EventIndex collects event information from data both at CERN and Grid sites. • It uses the Hadoop system to store the results, and web services to access them. • Its successful operation depends on a number of different components. • Each component has completely different sets of parameters and states and requires a special approach. Monitoring Components Prodsys EIOracle ??? Event Picking Tests XML ORACLE? Old Monitoring System based on Kibana Disadvantages of Kibana Slow dashboard retrieving time: - for two days period: 15 seconds; - for 7 days period: 1 minute 30 seconds; - for a longer periods: it may take tens of minutes and eventually get stuck. Not very comfortable way of editing the dashboard’s page Grafana Grafana is one of the most popular packages for visualizing monitoring data. Uses modern technologies: - back-end is written using Go programming language; - front-end is written on typescript and uses angular approach. The following datasources are officially supported: Graphite InfluxDB MySQL Elasticsearch OpenTSDB Postgres CloudWatch Prometheus Microsoft SQL Server (MSSQL) InfluxDB InfluxDB is InfluxData's open source time series database designed to handle high write and query loads. Uses modern technologies: - it is written on GO; - It has the possibility of working in cluster mode; - availability of libraries for a large number of programming languages (Python, JavaScript, PHP, Haskell and others); - SQL-like query language, with which you can perform various operations with time series (merging, splitting).
    [Show full text]
  • Monitoring with Influxdb and Grafana
    Monitoring with InfluxDB and Grafana Andrew Lahiff STFC RAL ! HEPiX 2015 Fall Workshop, BNL Introduction Monitoring at RAL • Like many (most?) sites, we use Ganglia • have ~89000 individual metrics • What’s wrong with Ganglia? Problems with ganglia • Plots look very dated Problems with ganglia • Difficult & time-consuming to make custom plots • currently use long, complex, messy Perl scripts • e.g. HTCondor monitoring > 2000 lines Problems with ganglia • Difficult & time-consuming to make custom plots • Ganglia UI for making customised plots is restricted & doesn’t give good results Problems with ganglia • Ganglia server has demanding host requirements • e.g. we store all rrds in a RAM disk • have problems if trying to use a VM • Doesn’t handle dynamic resources well • Occasional problems with gmond using too much memory, affecting other processes on machines • Not really suitable for Ceph monitoring A possible alternative • InfluxDB + Grafana • InfluxDB is a time-series database • Grafana is a metrics dashboard • originally a fork of Kibana • can make plots of data from InfluxDB, Graphite, others… • Very easy to make (nice) plots • Easy to install InfluxDB • Time series database written in Go • No external dependencies • SQL-like query language • Distributed • can be run as a single node • can be run as a cluster for redundancy & performance (not suitable for production use yet) • Data can be written in using REST, or an API (e.g. Python) • or from collectd or graphite InfluxDB • Data organised by time series, grouped together into databases • Time series have zero to many points • Each point consists of: • time - the timestamp • a measurement (e.g.
    [Show full text]
  • Solution Deploying Elasticsearch on Kubernetes Using
    Solution Guide Deploying Elasticsearch on Kubernetes using OpenEBS Contents Part 1 - Before starting Part 2 - Preconfiguration Part 3 - Getting Started with OpenEBS Part 4 - Installing Kudo Operator Part 5 - Installing ElasticSearch using Kudo Part 6 - Installing Kibana Part 7 - Installing Fluentd-ES Part 8 - Monitoring Elastic search 01 www.mayadata.io Overview Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is built on Apache Lucene and was first released in 2010 by Elasticsearch N.V. (now known as Elastic). Known for its simple REST APIs, distributed nature, speed, and scalability, Elasticsearch is the central component of the Elastic Stack, a set of free and open tools for data ingestion, enrichment, storage, analysis, and visualization. Commonly referred to as the ELK Stack (after Elasticsearch, Logstash, and Kibana), the Elastic Stack now includes a rich collection of lightweight shipping agents known as Beats for sending data to Elasticsearch.[1]. This guide explains the basic installation for Elasticsearch operators on OpenEBS Local PV devices using KUDO. We will be installing Fluentd and Kibana to form the EFK stack. The guide will also provide a way to monitor the health of Elasticsearch using Prometheus and Grafana. Before starting You require an existing Kubernetes cluster. Kubernetes provides platform abstraction, cloud-native software runs, and behaves the same way on a managed Kubernetes service like AWS EKS, Google Cloud GKE, Microsoft AKS, DigitalOcean Kubernetes Service, or self-managed based on Red Hat OpenShift and Rancher. You can also use kubeadm, kubespray, minikube.
    [Show full text]
  • The Open-‐Source Monitoring Landscape
    The Open-Source Monitoring Landscape Michael Merideth Sr. Director of IT, VictorOps [email protected], @vo_mike My History and Background • Working in IT since the mid 90’s • Mostly at startups in the Colorado Front Range area • Along for the ride during the “dot com boom” • Build my career using open-source tools Since the 90’s now, there’s been a sharp division in tools and methodology between the enterprise space and the startup and small business communi;es. Obviously, smaller businesses, especially in the tech sector, were early and eager adopters of open- source technology, and much quicker to learn to rely on open-source tool chains in business-cri;cal applica;ons. Up at the enterprise level, at the public companies, they’re only now fully-embracing Linux as a business-cri;cal tool, and I think that’s largely because “the enterprise” is star;ng to be defined by companies that either came up in the dot com era, like Google, or built the dot com era, like Sun, or Dell, or let’s say RedHat. So, the “enterprise” had their toolchain, built on commercial solu;ons like HPUX and OpenView and so on, and the startup community, the “dot com” community had a completely different toolchain, based on Linux, based on open standards and protocols, built with open-source components like GNU, and Apache, and ISC Bind and so on. I’m lucky enough that I’ve been able to spend my career in the startup sphere, working with that open-source toolchain. I started working in IT in the mid 90’s in the Colorado front range, and I’ve spent my ;me since then working for and consul;ng at early-stage startups and other “non enterprise” shops.
    [Show full text]
  • Grafana and Prometheus Just Married & PMM Was Born to Monitor Your
    23.11.2020 Grafana and Prometheus just married & PMM was born to monitor your DBs 1 Who we are The Company > Founded in 2010 > More than 70 specialists > Specialized in the Middleware Infrastructure > The invisible part of IT > Customers in Switzerland and all over Europe Our Offer > Consulting > Service Level Agreements (SLA) > Trainings > License Management Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 2 2 23.11.2020 About me Elisa Usai Delivery Manager & Consultant +41 78 638 09 78 elisa.usai[at]dbi-services.com Elisa Usai elisetta1984 Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 3 3 Agenda Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 4 4 23.11.2020 Agenda 1.Introduction 2.Percona Monitoring and Management (PMM) tool 3.Monitor your DBs with PMM 4.Conclusion Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 5 5 Introduction 1 > Monitor a DB 2 > Prometheus > Grafana > The benefits of a marriage 3 4 Grafana and Prometheus just married & PMM was born to monitor your DBs 19.11.2020 Page 6 6 23.11.2020 Introduction Monitor a DB Why is the monitoring so essential? > Services Health > System Optimization > Performance Problems > Business Process Improvement > Capacity Planning What could we monitor? > Databases, Instances > Hosts > CPU, Memory, I/O, Network > Storage, Filesystems > Application Grafana and Prometheus just married & PMM was born to monitor your DBs19.11.2020 Page 7
    [Show full text]
  • Monitoring Open Source Databases with Icinga
    PGConf EU | Warsaw | 26.10.2017 Monitoring Open Source Databases with Icinga • Blerim Sheqa • Product Manager • Working @netways • @bobapple Introduction to Icinga2 Quick Poll Icinga is a scalable and extensible monitoring system which checks the availability of your resources, notifies users of outages and provides extensive metricsmetrics. • Multithreaded Perfdata Graphite Livestatus • Modular Features • Zone support Notify InfluxDB • Secure Agent Checker Gelf IDO • No Nagios® • Availability and scaling zones • Automatic redistribution of checks • Zones for multitenancymultitenancy environments Monitoring Databases • MySQLish • PostgreSQL • MongoDB • Firebird • SQLite • check_mongodb_py • connect • last_flush_time • connections • replset_state • • replicaton_lag index_miss_ratio • collections • memory • database_size • memory_mapped • database_indexes • lock • replica_primary • flushing https://github.com/mzupan/nagios-plugin-mongodb • check_postgres.pl • archive_ready • disabled_triggers • autovac_freeze • disk_space • fsm_pages • backends • prepared_txns • bloat • query_runtime • checkpoint • query_time • cluster_id • replicate_row • commitratio • same_schema • connection • sequence • custom_query • settings_checksum https://bucardo.org/check_postgres/check_postgres.pl.html • check_mysql_health • connection-time • bufferpool-wait-free • uptime • log-waits • threads-connected • tablecache-hitrate • threadcache-hitrate • table-lock-contention • • qcache-hitrate index-usage • tmp-disk-tables • qcache-lowmem-prunes • slow-queries • bufferpool-hitrate
    [Show full text]
  • GARR Cloud Monitoring
    GARR Cloud monitoring Claudio Pisa - GARR EAPConnect Workshop II 2019-11-22 Rome Monitoring - day 0 (one year ago) ● Different monitoring systems: ○ Nagios ■ OpenStack services ○ Zabbix ■ Ceph ■ Hardware sensors ● Some systems not monitored Claudio Pisa // EAPConnect Workshop II 2019 - Rome // 2019-11-22 2 Architecture - day 0 Cloud Nagios Nagios Web Infrastructural interface services Cloud Zabbix Web Infrastructure Zabbix interface + Ceph Claudio Pisa // EAPConnect Workshop II 2019 - Rome // 2019-11-22 3 Architecture - day 0 Cloud Nagios Nagios Web Infrastructural interface services Cloud Zabbix Web Infrastructure Zabbix interface + Ceph Dell Chassis Web Bare Metal Dell tools interface GINS GINS Web Network (SNMP) interface Claudio Pisa // EAPConnect Workshop II 2019 - Rome // 2019-11-22 4 Objective Objective: State of the art: build a unified Grafana comprehensive dashboard Claudio Pisa // OpenInfra Days 2019 - Rome // 2019-10-03 5 Considerations ● Existing monitoring systems seem to be doing their job well ○ the right tool for the right job ● What is missing is just a single viewpoint ● Grafana seems to have what we need Claudio Pisa // EAPConnect Workshop II 2019 - Rome // 2019-11-22 6 Grafana ● Grafana is a platform for data visualization, querying and alerting ● Several pluggable data sources: ○ Zabbix ○ PNP (Nagios) ○ Prometheus ○ Gnocchi ○ Monasca ○ JSON (general purpose) ○ MySQL / PostgreSQL (general purpose) ● Data from heterogeneous sources can be mixed in the same dashboard Claudio Pisa // EAPConnect Workshop II 2019 - Rome
    [Show full text]
  • Training Telegraf, Influxdb, Grafana
    Telegraf, InfluxDB, Grafana Training Still Using MRTG? n Simple all in one SNMP monitoring software n Send SNMP requests n Store replies into text-based database n Generate images and HTML pages n Measures two values (input / output) n Collects data every five minutes n Static pages n RRDTools, Cacti 2 2019/11/15 Presented by Warren Chang Why should not use MRTG anymore n Pull-based n Mainly SNMP, 2-D data n Not scalable n Static image, web page n Five minutes interval n Difficult to customize n No modern alert mechanism n No distributed databases 3 2019/11/15 Presented by Warren Chang What we need n Collect data n Store and process data n Visualize data n Monitoring and alert n Telemetry data more than SNMP n What is telemetry data? n Getting more important n Big Data to AI 4 2019/11/15 Presented by Warren Chang Modern Data Monitoring and Processing Model 5 2019/11/28 Presented by Warren Chang Modern Data Monitoring and Processing Model Telegraf Grafana InfluxDB Kapacitor InfluxDB Prometheus 6 2019/11/28 Presented by Warren Chang TICK Architecture 7 2019/11/28 Presented by Warren Chang Products Telegraf InfluxDB Chronograf Kapacitor Agents for collecting Streaming data and reporting metrics Time Series Database Data visualization processing enging and events Graphite Grafana Logstash Kafka Prometheus Kibana Prometheus Grafana OpenTSDB Datadog Fluentd Prometheus Elasticsearch Splunk 8 2019/11/28 Presented by Warren Chang Why InfluxDB? source: https://db-engines.com/en/ranking/time+series+dbms 9 2019/11/28 Presented by Warren Chang Why InfluxDB, Telegraf, Grafana InfluxDB Telgraf Grafana n High performance, written in Go n High performance, written in Go n Rich data sources support n Native HTTP API n Collect and send almost all kinds n InfluxDB, Prometheus, MySQL of data n Powerful SQL-like language n Templating n 200+ input, output plugins n Supports logs n Alerts n Down sampling n Plugin, App 10 2019/11/28 Presented by Warren Chang Time Series Data Tags Measurement Stock_Price Name=Apple Inc.
    [Show full text]