These aren't the logs you're looking for

Alicja Kwasniewska, Intel Eric Lemoine, Mirantis Michal Jastrzebski, Intel Alicja Kwasniewska Eric Lemoine Michal Jastrzebski Agenda

Logging Central Hacking Elk or Kolla in logging Demo Rsyslog Ehk? dockers service Kolla Docker containers Ansible playbooks Kolla

Deployment Docker tools containers

Kollaglue Out of the box configuration Customization

● Production-ready Docker containers and deployment tools for operating OpenStack clouds ● Both out of the box configuration and possibility for complete customization ● Images built in the kollaglue namespace for every tagged release, but you can also build images on your own Logging in docker containers Docker logs

● What? Anything that is written out to the container’s standard output/error ● Logging capabilities available in the Docker are exposed as drivers

json syslog journald gelf fluentd

Default

● How to get path to this file? docker inspect -f "{{.LogPath}}" containerName ● The docker logs command retrieves logs present at the time of execution. NOTE: available only for containers with json-file and journald logging drivers Hacking rsyslog Previous solution based on rsyslog

● Hacky ● Shared /dev/log ● Not meant for containers ● New versions of docker breaks ● Syslog doesn’t work well with tracebacks ● Removed in both Liberty and Mitaka Central logging service advantages

“One Ring to rule them all, One Ring to find them...” Which node? Which container? Where should I look first? No need to search many places !

→ data stored in one location and accessed from single interface One more second and I will find the right place in one of the 10000+ lines...

“Developer looking at production logs after a regression with downtime” oil canvas, circa 1580; source: http://classicprogrammerpaintings.tumblr.com No need to search for one line in million ! → interface provides filtering options and visual representation of data using graphs and charts

Kolla Central logging service

New feature in Mitaka How it works? Heka Feed data /var/lib/k olla/...

Interact

Heka Feed data Elasticsearch View Kibana /var/lib/ Search kolla/...

Feed data Heka /var/lib/k olla/... Heka Unified data processing Introduction to Heka

Stream processing software

● Open source ● Developed by Mozilla ● Used in production (by Mozilla) ● Written in Go ● http://hekad.readthedocs.org Why Heka?

→“Unified Data Processing”

One tool for all the data:

Ops data/ Business Logging Error reports/ metrics data output tracebacks

● Ops data/metrics ● Business data ● Logging output ● Error reports/tracebacks Heka Pipeline

Plugins at each stage of the pipeline Heka Highlights

Lightweight Flexible

Configurable Easily and safely extendable

● Lightweight ● Flexible ● Configurable ● Easily and safely extendable ELK or EHK? Logstash vs Heka

Why did we choose Heka over Logstash?

● Avoid running a JVM on each node ● Heka is faster and more lightweight* ● Heka is very flexible (plugins defined w/ code)

* https://github.com/elemoine/heka-logstash-comparison → EHK Elasticsearch Open-source text search and analytics engine Introduction to Elasticsearch

● Open source product from Elastic group - https://github. com/elastic/elasticsearch ● Highly-scalable full-text search and analytics engine ● Used by applications with complex search requirements ● Written in Java Elasticsearch Highlights

Highly Available Highly Full-Text Scalable Search

RESTful Document- API oriented Kibana Open-source analytics and visualization platform Introduction to Kibana

● Open source product from Elastic group - https://github. com/elastic/kibana ● Written mainly in JavaScript, comes with Node.js server app ● Server component has a built-in proxy to Elasticsearch (designed to work with Elasticsearch) Kibana concepts

Interact

Search Visualize

View Analyse

● Allows to search, view, and interact with data stored in Elasticsearch indices ● Possibility to analyse and visualize data in a variety of charts, tables, and maps

Kibana - authentication

Shield From Elastic Nginx HAProxy Group

HTTP Basic authentication with ACL

Elastic Shield - provides security for entire stack, including Kibana plugin for session support and login screen, but requires a license…

Nginx - use it as proxy with simple auth mechanism for Kibana

HAProxy - add HTTP Basic authentication with ACL that uses the http_auth criterion and http-request auth rule Deploy

EHK with Kolla 3 easy steps

Adapt Build images Deploy Kolla configuration with one with one files and enable command command central logging

For details please watch webinar presented by Steven Dake - link provided at the last slide Demo → Thank you for your attention

Alicja Kwasniewska [email protected] Eric Lemoine [email protected] Michal Jastrzebski [email protected]

Irc: #openstack-kolla Webinar: https://vimeopro.com/midokura/345kl392