Harnessing the Power of Splunk and Google Cloud: Deploy, Ingest, And

Add your headshot to the circle below by clicking the icon in the center.

Alex Cain Roy Arsan Senior Product Manager | Splunk Partner Engineer | Google Cloud THIS SLIDE IS REQUIRED, DO NOT DELETE

During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC.

The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.

Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2019 Splunk Inc. All rights reserved. © 2019 SPLUNK INC. Agenda Here’s what is up

Deploying Splunk on GCP

▶ This White Paper is an excellent starting point for a Splunk deployment, regardless of the underlying infrastructure. • Use this document to map requirements to architectures and best practices.

▶ This Tech Brief contains GCP specific best practices and recommendations.

1. Networking

GCP Primer 2. Compute

3 Finland FASTER (US, JP, TW) 2016 Netherlands 3 London 3 3 3 Frankfurt Oregon 3 3 Montreal 3 Zurich Iowa 4 Belgium 3 N Virginia 3 3 3 3 Tokyo Los Angeles S Carolina Osaka Hong Kong 3 3 Taiwan 3 Mumbai

SJC (JP, HK, SG) 2013 Unity (US, JP) 2010 3 Singapore

PLCN Unity (HK, LA) 2018

3 Monet (US, BR) 2017 São Paulo Netw ork 3 Sydney Netw ork sea cable investments Junior (Rio, Santos) 2017

Edge points of presence >100 Tannat (BR, UY, AR) 2017 Indigo (SG, ID, AU) 2019 Google global cache edge nodes (>800) https://peering.google.com Region and number of zones https://cloud.google.com/about/locations https://cloud.google.com/compute/docs/regions-zones/regions-zones © 2019 SPLUNK INC. GCP Networking

Project VPC

Region Region ▶ VPC network is global

VM VM

Zone A Zone A

VM VM

Zone B Zone B

Project VPC

Region Region ▶ VPC network is global

VM VM ▶ Subnet spans entire region Zone A Zone A

VM VM

Zone B Zone B

Subnet X1: 10.0.0.0/24 Subnet Y1: 172.16.0.0/24

Project VPC

Region Region ▶ VPC network is global

Private: 172... Private: 10... VM VM Public: 203... Public: 203... ▶ Subnet spans entire region Zone A Zone A

Private: 172... Private: 10... VM VM Public: 203... Public: 203... ▶ VM private IP address is regional

Zone B Zone B

Subnet X1: 10.0.0.0/24 Subnet Y1: 172.16.0.0/24

Project VPC

Region Region ▶ VPC network is global

Private: 172... Private: 10... VM VM Public: 203... Public: 203... ▶ Subnet spans entire region Zone A Zone A

Private: 172... Private: 10... VM VM Public: 203... Public: 203... ▶ VM private IP address is regional

Zone B Zone B

Subnet X1: 10.0.0.0/24 Subnet Y1: 172.16.0.0/24 ▶ Routing table is global Region X Region Y

Destination Next hop Network

172.16.0.0/24 Virtual network default

Project VPC

Region Region ▶ VPC network is global

Private: 172... Private: 10... VM VM Public: 203... Public: 203... ▶ Subnet spans entire region Zone A Zone A

Private: 172... Private: 10... VM VM Public: 203... Public: 203... ▶ VM private IP address is regional

Zone B Zone B

Subnet X1: 10.0.0.0/24 Subnet Y1: 172.16.0.0/24 ▶ Routing table is global Region X Region Y

Destination Next hop Network

172.16.0.0/24 Virtual network default

Google Compute Engine (GCE) Virtual machines, networks (IaaS)

Google Kubernetes Engine (GKE) Managed Docker containers (CaaS)

Google App Engine (GAE) Serverless app platform (PaaS)

▶ VMs • Predefined and custom machine types • Live migration • Managed instance group (zonal or regional)

▶ Billing • Per-second billing • Sustained use discounts (up to 30%) • Committed use discounts (up to 57% or 70%) © 2019 SPLUNK INC. Google Compute Engine - Machine Types for Splunk Enterprise workload

Indexers: Search Heads:

Instance Type Daily Volume (GB) Instance Type Concurrent Users Performance n1-standard-16 Up to 100 n1-standard-16 8 Good n1-standard-32 100-250 n1-standard-32 16 Better

Deployment Server, License or Cluster Master:

Cloud Cloud Cloud Cloud Storage Bigtable Datastore Memorystore

▶ Block Storage • Local SSD vs Persistent Disk • Standard vs SSD Persistent Disks Persistent • Zonal vs Regional Persistent Disks Disk

▶ Persistent Disk (PD)

• High performance, low latency (single-digit ms for SSD) Storage Type Cost per GB/Month • Durable. Persists if instance dies - can be re-attached PD Standard $0.04 • Up to 64 TB per disk – no RAID required $0.08 • Online or live resizing with no downtime Regional PD • Regional PD for added redundancy and HA PD SSD $0.17

▶ Local SSD Regional PD SSD $0.34 • Very high throughput, lowest latency Local SSD $0.08 • Data is lost when instance terminates • 375 GB - can attach up to 8 for total of 3TB GCS Standard $0.02

▶ Cloud Storage (GCS) Listed pricing (us-central1) does not include any discounts, and is subject to change. See latest • Snapshots (backups) are global pricing at https://cloud.google.com/pricing/list © 2019 SPLUNK INC.

▶ Consider local SSD only with clustering • Limited to 3 TB / indexer • Must manage striping of local SSDs

▶ Use PD (SSD or Standard) for all other cases • Peak* IOPS & throughput at only 4TB PD SSD • Can dynamically resize indexer(s) storage • Can use SSD PD and Standard PD for hot/warm vs cold storage

▶ Use PD SSD for boot device • At least 50 GB in size for performance

Regional High Availability ▶ Regional Managed Instance Group Health Checking • For clustered-nodes Consistency • Search Head cluster, Indexer cluster

▶ Failover using Regional PD Regional High Availability • For single-node roles Fast RPO & RTO – Better than snapshot • Cluster Master, Deployer, etc. Automatic Failover

Goal: Why migrate Reliable global forensics analytics, real-time event Existing infrastructure expensive and unreliable threat detection, resilient and easy to scale with demand Limited on-prem capacity, lack of agility Did not fit HW spec ▶ 20 TB/day Leverage ML capabilities of Google Cloud ▶ 90-day retention

▶ Splunk Enterprise + ES + UBA

Deployment size: Results:

▶ Multi-site HA Lower TCO (40% lower cost)

▶ 240 indexers Deployment in days/hours vs months

▶ 15 search heads Ability to easily scale – now 25 TB/day

On- Real-world deployment prem

Cloud Load Balancing

Site A - us-west2-a Site B - us-west2-b Site C - us-west2-c

* Deployer Search Heads Search Heads Search Heads Search Head Compute Engine Compute Engine Compute Engine Compute Engine Cluster

License Compute Engine

Indexer Indexer Master Indexer Nodes * Nodes Nodes Indexer Compute Engine Compute Engine Compute Engine Compute Engine Cluster * Deployment Compute Engine Subnet 10.0.0.0/24

Splunk Enterprise Terraform scripts Now open-sourced on GitHub How do I get https://github.com/GoogleCloudPlatform/ started? terraform-google-splunk-enterprise or bit.ly/splunk-on-gcp © 2019 SPLUNK INC. Splunk Enterprise on GCP © 2019 SPLUNK INC.

Getting GCP data into Splunk

Security IT Ops Business

• Are buckets secure? Do they contain • How many servers do I have? • Where is most of my cloud spend? sensitive data? • Are services meeting SLAs? • What are areas to optimize cost? • What assets are being modified? • Are are there any perf bottlenecks? • Is infrastructure properly sized? • Are we following our access policies? • What’s the usage over time? • Are we using what we’ve paid for? • Is there any unusual activity or • How many events/requests threat? processed per second? © 2019 SPLUNK INC. Data Coverage Google Cloud offers mountains of data, so get to know it

Security IT Ops Business

• Cloud Security Command Center • Stackdriver Logs • Billing Reports • Cloud Asset Inventory • Stackdriver Metrics • Cloud Audit Logs • GKE & GKE On-Prem Metrics, Logs, Metadata • G Suite Admin Audit Logs © 2019 SPLUNK INC. GDI Service Map

All GCP-Monitored Services & Resources Compute, Storage, DB, Networking Services

Cloud Security Cloud Asset Stackdriver Stackdriver Command Center Inventory Logging Monitoring

GKE + GKE On-Prem

Cloud Pub/Sub

BigQuery Cloud Storage Cloud Dataflow

Unified batch and streaming processing

Fully Managed, No-Ops data processing

Open source programming model

Data Sinks Sources

BigQuery BigQuery Cloud Dataflow

Cloud Cloud Storage Storage

Cloud Cloud Pub/Sub Pub/Sub

Cloud Cloud Bigtable Bigtable

Cloud Third- Datastore Party DB See Google-provided Dataflow templates for common use cases https://github.com/GoogleCloudPlatform/DataflowTemplates © 2019 SPLUNK INC. Pub/Sub to Splunk Dataflow template Streaming data to Splunk HEC

● In the Splunk-GCP world, Dataflow can be used to stream data from Pub/Sub to Splunk

● Use “Pub/Sub to Splunk” template pipeline from Google-provided templates: ○ https://github.com/GoogleCloudPlatform/DataflowTemplates ○ Supports dead letter queue into Pub/Sub topic (fallback), secondary sink to GCS (archive) ○ Supports JavaScript User-defined functions (UDF) to transform event before sending to Splunk

Dataflow template transforms/enriches data before Splunk HEC pushing to Splunk HEC

Cloud Pub/Sub

Cloud Dataflow

Dataflow vs Addon How do these ingestion methods compare?

• Send data to Splunk via HEC • Collect data via Pub/Sub in matches − Normal HEC limitations • Some predefined source-types • Cloud-native streaming (simplicity, security, scale) • Also collects: • Wide coverage: Pub/Sub, GCS, BigQuery, etc. − Stackdriver Metrics • Simplifies collecting: − Billing − Asset inventory − G Suite © 2019 SPLUNK INC. GCP Stackdriver Logs GCP GDI Pattern

● GCP logs (audit, etc.) end up in Stackdriver Logs ○ Also referred to as GCP Logging ● Stackdriver logs can be configured to have Pub/Sub as a sink destination ○ Remember - Splunk can scalably pull from Pub/Sub ○ Alternatively, can use GCP Dataflow to stream directly from Pub/Sub to Splunk HEC Stackdriver Logging export GCP Services export logs to Splunk pulls from Pub/Sub sets Pub/Sub as a sink for Stackdriver logging OR stream to Splunk HEC incoming logs

Logging

Cloud Pub/Sub

Stackdriver Alternate path

● ALL Stackdriver Metrics are supported by the Splunk Add-on for GCP ○ For detailed VM instance metrics - Stackdriver agent must be installed ○ List of GCP service metrics here: • https://cloud.google.com/monitoring/api/metrics_gcp

GCP Services export metrics Splunk pulls in specific metrics to Stackdriver monitoring with scheduled API calls

Monitoring

● GCP Cloud billing reports can be configured to be pushed daily to a GCS bucket. (File Export) ● The Splunk Add-on for Google Cloud Platform comes with an input for pulling reports from a GCS bucket

GCP exports billing Splunk pulls in billing reports reports to a GCS bucket with scheduled API calls

Cloud Cloud Billing API Storage

● Alternative: Billing data exported to BigQuery ○ Export to GCS, then use existing Billing input - need to automate this process ○ BigQuery billing data is actually in a different format and more verbose than supported file (GCS) export approach © 2019 SPLUNK INC. Google Cloud Storage Other than billing data

● Other GCS data? ○ Option 1: Use Dataflow templates to stream or batch to Pub/Sub, then pull via Add-on ○ Option 2: Use Pub/Sub to Splunk Dataflow template, and set source connector to GCS ○ For low bandwidth data use scheduled batch jobs ($), otherwise streaming jobs ($$$) DataFlow template transforms/enriches Splunk pulls from Pub/Sub data before pushing to Pub/Sub

Pulling via Add-on

Cloud Cloud Cloud Storage Dataflow Pub/Sub

Streaming to HEC

▶ Can we stream to Pub/Sub and use the GCP Add-on? • G Suite audit logs can be exported to BigQuery • BigQuery -> DataFlow -> Splunk OR • BigQuery -> Export to GCS -> DataFlow -> Splunk • Latency becomes a consideration

DataFlow template transforms/enriches data before pushing to Splunk HEC

BigQuery Cloud Dataflow

Optional alternate path

▶ This Add-on supports data collection for a number of GCP sourcetypes out of the box • https://splunkbase.splunk.com /app/3088/ © 2019 SPLUNK INC. Getting Data In – GCP side GCP Organization structure. Simplified logging across projects

▶ Multi-tiered organization structure allows for separation of projects, products, departments, etc. within GCP

▶ GCP Stackdriver can export aggregated logs from all or a subset of projects, folders, etc.

Data Source Mechanism

Stackdriver Logs Splunk GCP Add-on Mod Input, OR (includes Cloud Audit) Streaming via Pub/Sub to Splunk Dataflow pipeline

Stackdriver Metrics Splunk GCP Add-on Mod Input

Cloud Storage – Billing Reports Splunk GCP Add-on Mod Input

Cloud Asset Inventory GCP streaming pipeline + Splunk GCP Add-on Mod Input

Cloud Security Command Center GCP streaming pipeline + Splunk GCP Add-on Mod Input

GKE & GKE On-Prem (Anthos) Splunk Connect for K8s (and Splunk App for Infrastructure)

Check out Session FN2132 for a deeper dive on GCP.”

GCP Side Best Practices What levers to pull on the GCP side

▶ Enable Data Access logs for select or all services • Best practice: configure at the organization level • Admin Activity logs enabled by default

▶ Configure logging export to Pub/Sub topic • Best practice: Set up aggregated export for the organization • Filtering: Can include/exclude logs for specific resources and types

▶ Set IAM policy permission for Pub/Sub topic • Grant SA permission to publish to topic

Splunk Side Best Practices What levers to pull on the Splunk side

▶ Scaling data collection • Setup more inputs for the same Pub/Sub topic (no this wont cause duplication) • Create more Pub/Sub topics split by use-case (security centric logs in their own topic) • Add more Instances to collect from Pub/Sub or increase number of HEC listeners • Pub/Sub to HEC: bump timeout from 10s to 50s

Thank You!

Go to the .conf19 mobile app to RATE THIS SESSION