About Me
> Eduardo Silva ● Github & Twitter: @edsiper ● Personal Blog : http://edsiper.linuxchile.cl
> Treasure Data, Inc. ● Open Source Engineer Fluentd / FluentBit
> Projects ● Monkey Server / monkey-project.com ● Duda I/O / duda.io
Logging
Logging Matters Pros
● Applications status
● Debugging
● General information about anomalies: errors
● Troubleshooting / Support
● Local or Remote (network)
Logging Matters From a business point of view
● Input Data → Analytics
● User interactions / behaviors
● Improvements
Assumptions
Logging Matters Assumptions
● “I have enough disk space”
● “I/O operations will not block”
● “It's fast to write a simple message”
● “Log formats are human readable”
● “My logging mechanism scale”
Logging Matters Assumptions
Basically, yeah.. it should work.
Concerns
Logging Matters Concerns
● Logs increase = data increase
● Message format get more complex
● Did the Kernel flush the buffers ? (sync(2))
● Multi-thread application ?, Locking ?
● Multiple Applications = Multiple Logs
Logging Matters Concerns
If... Multiple Applications = Multiple logs Multiple Host x Multiple Applications = ???
OK...so: 1. Logging Matters 2. It's really beneficial 3. ???
but...
It needs to be done right.
Logging Common Sources / Inputs
● Applications Logs
• Apache • Nginx • Syslog (-ng)
● Custom applications / Languages
C, Ruby, Python, PHP, Perl, NodeJS, Java...
How to parse/store multiple data inputs ? note: performance matters!
Fluentd is an open source data collector
It let's you unify the data collection for a better use and understanding of data.
Fluentd Highlights
● High Performance
● Built-in Reliability
● Structured Information/Logs
● Pluggable Architecture
● More than 300 plugins (input/filtering/output)
Fluentd Before
Fluentd After
Configuration
Who uses Fluentd in Production ?
Logging and other use cases
We collect Logging and other 500.000use casesevents per second!
Logging and other use cases
Logging More use cases We talked mostly about Server applications, let's talk now about other contexts:
● Internet of Things (IoT)
● Embedded Applications
● OS Containers (Docker) The fun continues ;)
Internet of Things (IoT)
Internet of Things Facts
● IoT will grow to many billions of devices over the next decade.
● Now it's about device to device connectivity.
● Different frameworks and protocols are emerging.
● It needs Logging.
Internet of Things Frameworks In order to solve the device to device connectivity and proper messaging, multiple framework/stacks are emerging, all of them backed by different companies and vendors. The most remarkable at the moment are:
Brillo, by Google Just announced, no logo..sorry!
Internet of Things Logging
● How do we approach data collection for IoT ?
● How can we improve our IoT based product if we have data from thousand of devices with multiple sensors, meaning many inputs ?
Embedded
Embedded Importance of the Embedded Market
● Embedded devices are everywhere:
● Medical Devices
● Automotive
● Home Automation
● Many...
Fluentd for Embedded ?
Fluent Bit Objective Open Source data collection tool for Embedded Linux:
● Services
● Sensors / Signals / Radios
● Operating System Info
● Automotive / Telematics
Fluent Bit Requirements An Embedded environment requires special handling, specifically on performance and resources utilization, for hence it needs to be:
● Lightweight
● Written in C Language.
● Customizable, pluggable architecture.
● Integrate with Fluentd
Fluent Bit Requirements
● Support custom Inputs
● Support custom Outputs
● Use Binary Serialization (MessagePack)
● Configurable (though file system files)
Features
Fluent Bit Features
● Collection & Distribution.
● Built-in system metrics.
● C API for Developers (WIP).
● Integration with third party services.
● Open Source / Apache License v2.0
Fluentd Integration
Fluent Bit Integration
Direct Output
Fluent Bit Direct Output
Examples
Fluent Bit Built-in Metrics: CPU usage
Fluent Bit Kernel Log Messages
Fluent Bit Configuration Example
Roadmap
Fluent Bit Roadmap
● Library mode.
● Support a stock of sensors (inputs).
● HTTP input.
● Release first stable version, work in process!
● Documentation.
Thanks! http://fluentd.org http://fluentbit.io github.com/fluent/fluentd github.com/fluent/fluent-bit We are Hiring!
Eduardo Silva @edsiper / [email protected]