Elastic Stack for Security Monitoring 1 in a Nutshell
2019 Pass the SALT Workshop Overview 2
Introduction to Elastic Stack
Beats
Logstash
Elasticsearch
Kibana
Elastic Stack Alerting and Security Introductory Workshop! 3
• This is an introductory workshop • You probably won’t hear/see a lot of new things if you have: • Used Elastic Stack in the past; • Took the Elastic training…; • Followed SANS SEC455, SEC555, FOR572, etc.; • If you are stuck, please do not suffer in silence! Workshop VM 4
• ais_workshop_xubuntu-18.04.2-desktop-amd64 • VMware Workstation, Player, or Fusion • You can try VirtualBox too, but you are on your own with that… sorry! • 8 GB RAM • 30-50 GB disk space • Keyboard layout: EN-US !!! • Workshop VM (Ubuntu) user/pass: user / Workshop1234% • Normally, it should not require password for login and sudo About David 5
• Managing partner at Alzette Information Security (@AlzetteInfoSec) • Network penetration testing, security architectures, security monitoring, incident response • Instructor at SANS Institute: FOR572 • BSides Luxembourg organizer https://bsideslux.lu • Twitter: @DavidSzili • E-mail: [email protected] • Blog: http://jumpespjump.blogspot.com About Eva 6
• Managing partner at Alzette Information Security (@AlzetteInfoSec) • Web application penetration testing, source code review, security monitoring • CyberWayFinder • BSides Luxembourg organizer https://bsideslux.lu • Twitter: @EvaSzilagyiSec • E-mail: [email protected] • Blog: http://jumpespjump.blogspot.com Introduction to Elastic Stack 7
2019 Pass the SALT Workshop About Elastic Stack 8
What is Elastic Stack? Why Elastic Stack? • 4 main components: • (Free) Open Source Software • Elasticsearch • Distributed, real-time search and • Logstash analytics (very scalable) • Kibana • Parsing and data enrichment • Beats • Large Community • And several other smaller components • InfoSec Projects built around it: Security Onion • Elastic Stack Features (X-Pack) • • Moloch (Elasticsearch) • APM (Application Performance Monitoring) • SOF-ELK • SELKS • HELK • ROCK NSM Elastic Stack History 9
2019: Core security 2015: "Release 2017: Elastic features (TLS, Early 2000s: Bonanza“, Cloud RBAC) are Shay Banon’s Beats, Elastic Enterprise free, SIEM, Recipe App Cloud (AWS) (ECE) EndGame
2012: 2016: Elastic 2018: Open Elasticsearch Stack 5.0 source X-Pack, Inc. New York Stock Exchange
Source: https://www.elastic.co/about/history-of-elasticsearch Elastic Stack (Very) High-Level Overview 10
• Beats: single-purpose data shippers • Logstash: server-side data processing pipeline • Elasticsearch: distributed search and analytics engine • Kibana: visualization and dashboards
See also: https://www.elastic.co/assets/blt2614227bb99b9878/architecture-best-practices.pdf Beats 11
2019 Pass the SALT Workshop Beats: Lightweight Data Shippers 12
• Lightweight log agents • Written in Go • Can send to Logstash or directly to Elasticsearch • Beats Family: • Filebeat • Winlogbeat • Auditbeat • Packetbeat • Heartbeat • Metricbeat • Functionbeat • Etc.
See also: https://www.elastic.co/guide/en/beats/libbeat/current/index.html Beats Configuration Examples 13
Winlogbeat Filebeat Beats Hands-On 14
2019 Pass the SALT Workshop Logstash 15
2019 Pass the SALT Workshop Logstash Overview 16
• LOTS AND LOTS of plugins! • Input: tcp, udp, syslog, beats, jdbc, kafka, rabbitmq, file, exec, cloudwatch, etc. • Filter: csv, json, xml, kv, grok, date, mutate, split, useragent, ruby, drop, etc. • Output: elasticsearch, graphite, nagios, kafka, rabbitmq, radis, file, email, irc, etc. • Easy to learn and use
Input Filter Output
See also: https://www.elastic.co/guide/en/logstash/current/index.html Input Plugin Examples 17
Plugin Description input { stdin { beats Events from Elastic Beats } } cloudwatch Events from AWS CloudWatch file Streams events from files input { beats { jdbc Events from JDBC data port => 5044 kafka Reads events from Kafka } } rabbitmq Pulls events from RabbitMQ input { s3 Events from files in S3 syslog { snmp Polls devices using SNMP port => 5514 } syslog Reads syslog messages }
See also: https://www.elastic.co/guide/en/logstash/current/input-plugins.html Filter Plugin Examples 18
Plugin Description Plugin Description cidr Check IP against net blocks kv Parses key-value pairs csv Parses CSV data into fields mutate Performs mutations on fields date Parses dates from fields ruby Executes Ruby code dissect Extracts unstructured data split Splits multi-line messages drop Drops all events translate Replaces field contents elasticsearch Gets data from Elasticsearch truncate Truncates fields geoip Geo info about an IP urldecode Decodes URL-encoded fields grok Parses unstructured data useragent Parses user agent strings json Parses JSON data xml Parses XML data
See also: https://www.elastic.co/guide/en/logstash/current/filter-plugins.html Filters - The Easy Stuff 19
JSON CSV filter { filter { ...... json { csv { columns => ["ts", "uid", "id.orig_h", source => "message“ "id.orig_p", "id.resp_h", "id.resp_p", "proto", } "service", "duration", "orig_bytes", ... "resp_bytesconn_state", "local_orig", "local_resp", mutate { "missed_bytes", "history", "orig_pkts", "orig_ip_bytes", remove_field => [ "message" ] "resp_pkts", "resp_ip_bytes", "tunnel_parents"] } separator => " “ } } ... mutate { remove_field => [ "message" ] } } Filters - RegExp vs. Grok, Dissect (1) 20
RegExp Dissect • (?[0-9A-F]{10,11}) Grok Debuggers: • Heroku App: http://grokdebug.herokuapp.com • Source: https://github.com/nickethier/grokdebug • Docker: https://hub.docker.com/r/fdrouet/grokdebug • Kibana / Dev Tools / Grok Debugger Filters - RegExp vs. Grok, Dissect (2) 21
dissect grok filter { filter { ...... dissect { grok { mapping => { match => { "message" => "%{ts} %{+ts} "message" => %{+ts} %{src} %{prog}[%{pid}]: %{msg}“ "%{SYSLOGTIMESTAMP:syslog_timestamp} } %{SYSLOGHOST:syslog_hostname} } %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: ... %{GREEDYDATA:syslog_message}" } #"message" => "%{SYSLOGBASE2} %{GREEDYDATA:message}“ } } ... } Filters - Enrichment Examples 22
ruby geoip filter { filter { ...... if [program] == "bro_dns" { if [resp_h_routable] == "true" { ruby { geoip { code => "event.set('query_length', source => "id.resp_h“ event.get('query').length)“ target => "geoip“ } default_database_type => "City“ } } ... geoip { } source => "id.resp_h“ target => "geoip“ default_database_type => "ASN“ } } ... } Output Plugin Examples 23
Plugin Description csv Writes events to disk in CSV output { stdout { elasticsearch Stores logs in Elasticsearch codec => rubydebug } email Sends email to an address } exec Runs a command file Writes events to files output { graphite Writes metrics to Graphite elasticsearch { kafka Writes events to Kafka hosts => ["localhost:9200"] } rabbitmq Pushes events to RabbitMQ } redis Sends events to Redis
See also: https://www.elastic.co/guide/en/logstash/current/output-plugins.html Elastic Common Schema (ECS) 24
• Specification that provides a Level Description consistent and customizable way to structure your data in ECS Core Fields Fully defined set of field names Elasticsearch that exists under a defined set of • Searches can be created more ECS top-level objects narrowly ECS Extended Fields Partially defined set of field names • Field names are easier to remember that exists under the same set of • ECS Reference: ECS top-level objects https://www.elastic.co/guide/en Custom Fields Undefined and unnamed set of /ecs/current/index.html fields that exists under a user- • ECS GitHub: supplied set of non-ECS top-level https://github.com/elastic/ecs objects that must not conflict with ECS fields or objects Logstash Hands-On 25
2019 Pass the SALT Workshop Elasticsearch 26
2019 Pass the SALT Workshop Elasticsearch Overview 27
• Storage and Search More info: John Hubbard - The Elastic Stack as a SIEM: https://www.youtube.com/watch?v=v69kyU5XMFI • Built on Apache Lucene • “wrapper” written in Java • REST API • JSON over HTTP • Distributed • Real-time Elasticsearch Terms 28
• Cluster: All nodes • Node: Elasticsearch instance Node1 Node2 Node3 • Index: Set of documents (group of shards) • Shard: Index1 • Subset of documents in an index Documents • Apache Lucene instance Index2 Shard • Primary (like RAID 0) and IndexShard Index Replica (like RAID 1) Shard Shard • Document: JSON object in Elasticsearch Index3
Cluster Elasticsearch vs. Relational Database 29
• Mapping: Elasticsearch Relational Database • Defines field names and datatypes Index Database in documents • Can add new fields, but existing Mapping Schema fields cannot be changed! Document Row • Field: Field Column • Key-value pair in a document • Metadata like: _index, _id, etc. • WORM (Write Once Read Many) vs. ACID (Atomicity, Consistency, Isolation, Durability) Data Types (Few Examples) 30
Core Specialized • text • date • keyword • ip • long, integer, short, byte Complex • double, float, half_float, • array scaled_float • object • boolean • nested binary • Multi-fields Geo • Indexed as more one type • geo_point Etc. • geo_shape
See also: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html Text vs. Keyword 31
Text type Keyword type • “Full-text value” • “Exact value” • Payload, message, etc. • IP, port, protocol, user, etc. • Analyzed and tokenized • Exact match / not match • Cannot be used for • Can be used for • Sorting • Sorting • Aggregations • Aggregations Elasticsearch Hands-On 32
2019 Pass the SALT Workshop Kibana 33
2019 Pass the SALT Workshop Kibana Overview 34
• Web-based analytic interface • Searches • Apache Lucene syntax • Filters • Visualizations, Dashboards • Stored in JSON • Plugins • Reporting, Alerting, etc. Kibana Features 35
• Discover: Search • Logs: Filebeat monitoring • Visualize: Graphs, charts • APM: Application Performance • Vega, Vega-Lite Monitoring • Dashboard: Visualizations and saved • Uptime: Monitor the status of searches network endpoints • Timelion: Time series visualizations • SIEM: Interactive workspace for security investigations • Canvas: Presentation • Dev Tools: API access • Machine Learning (Paid) • Monitoring: Cluster health • Graph (Paid) • Management: Cluster management • Infrastructure: Metricbeats monitoring • etc. Index Patterns 36
• Must choose an index pattern Steps: • Discovery (Searches) 1. Create Elasticsearch index • Visualization 2. “Create index pattern” • Limits the indices searched 3. Select index/indices • Relates to index naming scheme 4. Define @timestamp field • Can use the * wildcard • “logstash-* ” Search – Apache Lucene Query Syntax (1) 37
Search Type Syntax Example Single Term
Search Type Syntax Example Field exists _exists_:
• Search: Using the Query bar and the Apache Lucene Query Syntax • Filter: Using the Filters Box and the Elasticsearch Query DSL (Domain Specific Language) Visualizations 40
Visualization Type Visualization Type Area Basic Charts Metric Data Heat Map Basic Charts Coordinate Map Maps Horizontal Bar Basic Charts Region Map Maps Line Basic Charts Timelion Time Series Pie Basic Charts Visual Builder (E) Time Series Vertical Bar Basic Charts Controls (E) Other Data Table Data Markdown Other Gauge Data Tag Cloud Other Goal Data Vega (E) Other Visualizations use Elasticsearch Aggregations 41
Metrics: value to calculate Bucket: aggregation (grouping) • Count • Standard • Date • Range • Average Deviation Histogram (by • Sampler time) • Sum • Top Hit • Significant • Date Range • Min • Percentiles Text • Filter • Max • etc. • Terms (by • Geo Distance field) • Unique Count • IP Range • etc. Vega and VegaLite 42
• Vega Graphs • Visualization grammar • Declarative language • JSON format • Supported from Elastic 6 • Vega vs VegaLite • VegaLite: simplified Vega • https://vega.github.io/vega/ Based on: https://www.elastic.co/blog/sankey- • https://vega.github.io/vega-lite/ visualization-with-vega-in-kibana Canvas 43 Elastic SIEM 44
Source: https://www.elastic.co/blog/introducing-elastic-siem Kibana Hands-On Scenario 45 Kibana Hands-On 46
2019 Pass the SALT Workshop Elastic Stack Alerting and Security 47
2019 Pass the SALT Workshop Watcher vs. ElastAlert 48
Watcher ElastAlert • Part of X-Pack • Developed by Yelp • https://www.elastic.co/guide/en https://github.com/Yelp/elast /x-pack/current/xpack- • alerting.html alert • Elasticsearch API • Simple framework for alerting • JSON format • YAML format • Watches: Triggers, Inputs, • Components: Rules and Alerts Conditions, Transforms, Actions ElastAlert Overview 49
1. Elasticsearch is periodically • Rule types: Any, Blacklist, queried Whitelist, Change, Frequency, 2. Data is passed to the rules Spike, Flatline, New Term, Cardinality, Metric Aggregation, 3. When a match occurs, one or Percentage Match more alerts are triggered • Alert types: Command, Email, 4. Alerts take action based on JIRA, ServiceNow, Slack, the match PagerDuty, GoogleChat, Mattermost, Telegram, etc. • https://elastalert.readthedocs.io ElastAlert Examples 50 ElastAlert Hands-On 51
2019 Pass the SALT Workshop Security 52
• Elastic Stack Security: https://www.elastic.co/products/stack/security • Part of Elastic Stack Features (formerly X-Pack) • “Starting in version 6.8 and 7.1, core security features like TLS, file and native realm authentication, and role-based access control are now free.” • ReadonlyREST: https://readonlyrest.com • 3rd party • Free community version • Search Guard: https://search-guard.com • 3rd party • Free community version • NGINX reverse proxy + Basic Auth: https://www.nginx.com • No RBAC at all Questions and Answers 53
2019 Pass the SALT Workshop References 54
• Elastic Website • https://www.elastic.co • Elastic Documentation • https://www.elastic.co/guide/index.html • John Hubbard - The Elastic Stack as a SIEM • https://www.youtube.com/watch?v=v69kyU5XMFI • ElastAlert • https://github.com/Yelp/elastalert