Google Cloud Platform Solutions for Devops Engineers
Total Page:16
File Type:pdf, Size:1020Kb
Google Cloud Platform Solutions for DevOps Engineers Márton Kodok / @martonkodok Google Developer Expert at REEA.net - Targu Mures March 2019 - Vilnius, Lithuania About me ● Geek. Hiker. Do-er. ● Among the Top3 romanians on Stackoverflow 130k reputation ● Google Developer Expert on Cloud technologies ● Crafting Web/Mobile backends at REEA.net ● BigQuery/Redis and database engine expert ● Active in mentoring and IT community StackOverflow: pentium10 GitHub: pentium10 Slideshare: martonkodok Twitter: @martonkodok GCP Solutions for DevOps Engineers @martonkodok Agenda 1. Application development in the Cloud 2. App Engine: Scale your apps seamlessly from zero to planet scale 3. Cloud Functions: your gateway to GCP Services 4. Google Stackdriver: Metrics, logging, alerting are a universal right! 5. BigQuery: federated data access warehouse 6. Doing hybrid cloud mixing on premise with cloud 7. Practical use cases 8. Qwiklabs GCP Solutions for DevOps Engineers @martonkodok What’s so hard about traditional app development? Select Containerization Monitoring Testing IaaS Networking Storage Dev OS User Setup Logging Container Orchestration GCP Solutions for DevOps Engineers @martonkodok REEA.net uses GCP Build on the same infrastructure that powers Google Google Cloud Platform (GCP) Compute Big Data Identity & Security Compute App Kubernetes Cloud Cloud Cloud Cloud Resource Cloud Security Key BigQuery Cloud IAM Engine Engine Engine Dataflow Dataproc Dataprep Manager Scanner Management Service Cloud Container- Cloud Cloud Data Data Loss Identity-Aware Security Key GPU Genomics BeyondCorp Functions Optimized OS Datalab Pub/Sub Studio Prevention API Proxy Enforcement Internet of Things Machine Learning Storage & Databases Cloud IoT Cloud Machine Cloud Cloud Cloud Video Cloud Cloud Cloud Transfer Core Learning Vision API Speech API Intelligence Storage Bigtable Datastore Appliance API Cloud Persistent Cloud Natural Cloud Cloud Advanced Cloud SQL Language API Translation Jobs API Solutions Lab Spanner Disk API Google Cloud Platform (GCP) Management Tools Networking Error Virtual Cloud Load Cloud Cloud Cloud Cloud Stackdriver Monitoring Logging Trace Reporting Private Cloud Balancing CDN External IP Firewall Rules Router Addresses Cloud Cloud Cloud Cloud Cloud Cloud Cloud Dedicated Debugger Cloud DNS Cloud VPN Deployment APIs Console Shell Interconnect Network Routes Interconnect Manager Developer Tools Cloud Mobile Cloud Profiler App Billing API Cloud Cloud Source Cloud Cloud Tools Container Cloud SDK Deployment Repositories Tools for for IntelliJ Builder Manager Android Studio Cloud Cloud Container Google Plug-in Cloud Test Tools for Tools for Registry for Eclipse Lab PowerShell Visual Studio Google sees serverless as Programming model Operational model Billing model Focus on code Zero ops Pay for usage Event-driven Automatic scaling Stateless Managed security Dev Ops $ GCP Solutions for DevOps Engineers @martonkodok Serverless is more than a set of functions BigQuery Cloud Storage Cloud Dataflow Cloud Tasks Cloud Functions App Engine Stackdriver Cloud PubSub GCP Solutions for DevOps Engineers @martonkodok Meet Serverless serverless data center depicted GCP Solutions for DevOps Engineers @martonkodok Serverless is about maximizing elasticity, cost savings, and agility of cloud computing. GCP Solutions for DevOps Engineers @martonkodok Serverless types Platforms Triggered Code GCP Solutions for DevOps Engineers @martonkodok App Engine - managed application platform ● Scale your applications seamlessly from API 1 Multiple App Engine Clients zero to planet scale API Requests Split App Versions ● Automatically scales depending on your application traffic API 2 Cloud Load Compute Engine ● Traffic Splitting (app versions, A/B tests, Balancing Virtual Machines incremental rollouts) API 3 Container Engine Kubernetes Services Best used: ● For HTTP services ● For existing applications GCP Solutions for DevOps Engineers @martonkodok App Engine: Services and versions Application Service Service Version Version Version Version Instance Instance Instance Instance GCP Solutions for DevOps Engineers @martonkodok App Engine: Traffic splitting A/B testing and Canary releasing with a few clicks or a single gcloud command GCP Solutions for DevOps Engineers @martonkodok App Engine: Runtimes Java 8 1.11 GCP Solutions for DevOps Engineers @martonkodok Async message processing GCP Solutions for DevOps Engineers @martonkodok Asynchronous task execution Cloud Tasks Cloud PubSub Message queue system Deliver event data based on topics Guaranteed at-least-once delivery Exactly-once processing Future Scheduling Build multi-cloud on premise, hybrid apps Will only be dispatched once on dups Cross zone message replication Best used: Best used: ● For HTTP services ● For large-scale ingestion of events, streams ● For App Engine handlers ● Topics, publish/subscribe patterns, IoT GCP Solutions for DevOps Engineers @martonkodok Cloud Tasks - Message Queue systems Static Content Cloud Storage Dynamic Content Front End App Cloud SQL Batch App Cloud Load App Engine App Engine Balancing Autoscaling Dynamic Content Autoscaling Cloud Datastore Memcache Cloud DNS Workers Workers Compute Engine Cloud Compute Engine Tasks Autoscaling GCP Solutions for DevOps Engineers @martonkodok Task Queues Push queues ● Put with delays Pull queues ● Ability to “tag” ● Lease multiple eg: gameboard updated, game id as tag. Slides: https://www.slideshare.net/martonkodok Title: Architectural Patterns - Message Queues GCP Solutions for DevOps Engineers @martonkodok Reduce request latency GCP Solutions for DevOps Engineers @martonkodok Cloud Functions GCP Solutions for DevOps Engineers @martonkodok Cloud Functions Triggered Code Frontend Platform Services Event Cloud Event Sourcing Application Result Triggered Functions Metrics / Logs/ Streaming GCP Solutions for DevOps Engineers @martonkodok Cloud Functions Unit: Function Trigger: Events and HTTP Best used: ● For Events & Async workloads ● For single-purpose microservices GCP Solutions for DevOps Engineers @martonkodok Cloud Functions - Event-driven - FaaS ● Simplest way to run your code in the cloud - just deploy ● Automatically scales, highly available and fault tolerant ● No servers to provision, manage, patch or update ● Pay only while your code runs ● Connects and extends cloud services (In alpha: Java) ● Node 8.11 ● Python 3.7.1 ● Go 1.11 ● Support for async/await ● Flask microframework ● Familiar building blocks GCP Solutions for DevOps Engineers @martonkodok Functions: your gateway to GCP Services Access 20+ Google services from GCF Services Cloud Cloud Cloud Cloud BigQuery Pub/Sub Storage Bigtable Datastore Cloud Vision Cloud Machine Cloud Speech Cloud Applications API Learning API Spanner Cloud Stackdriver Cloud Natural Firebase Assistant Translation API APIs Language API GCP Solutions for DevOps Engineers @martonkodok Cloud Functions Trigger Cloud Storage Compute Engine EAP Cloud Pub/Sub Finalize/Create Instance Topic Delete Instance Group Archive Autoscaler Metadata Update Firewall Set Labels GCP Solutions for DevOps Engineers @martonkodok Cloud Functions Trigger Firebase BigQuery Cloud Firestore Realtime DB change Job create Create Authentication Job complete Update Remote config Write Google Analytics log Delete GCP Solutions for DevOps Engineers @martonkodok Summary: When to use what Cloud Functions App Engine Serverless add-on Smallest unit of Unit of computing Run functions, apps computing is apps & containers on GKE Event driven HTTP architecture request/response Full portability of your artifacts Connect & extend Large scalable services backends Run on your own cluster GCP Solutions for DevOps Engineers @martonkodok The unit of computing is yours to chose! Slice and dice your application any way you think it makes sense to be more agile, to build better apps that deliver value. GCP Solutions for DevOps Engineers @martonkodok How I Learned How to Stop Worrying and Dig Hosted Monitoring GCP Solutions for DevOps Engineers @martonkodok What to do for monitoring? ● In-house or hosted? ● Modern vs proven? ● Specialized or general? ● Paid vs free? Let’s be honest: ● Not enough time ● Small environment ● More important work to do ● Everyone else is doing it GCP Solutions for DevOps Engineers @martonkodok Google Stackdriver Intelligent monitoring and management for services, containers, applications, and infrastructure. ● Works with GCP, AWS, on prem ● Logging, tracing, alerts ● Collectd agent ● Cost per resource ● Easy point and click alerting policy ● Debugger, Error reporting, profiler *free ● Uptime checks *free GCP Solutions for DevOps Engineers @martonkodok Google Stackdriver: Uptime check (offered free) ● Check Type: HTTP, TCP ● Resource Type: URL, Instance, App Engine, Load Balancer ● Global Locations ● Custom Headers (encrypted) ● Authentication ● Whitelisted source IPs ● Alerting policy: Email, SMS, Slack, PagerDuty, Hipchat, Campfire, Webhooks GCP Solutions for DevOps Engineers @martonkodok Google Stackdriver: Uptime check GCP Solutions for DevOps Engineers @martonkodok Life of a log GCP Solutions for DevOps Engineers @martonkodok Google Stackdriver Metrics, logging, alerting are a universal right! GCP Solutions for DevOps Engineers @martonkodok GCP Solutions for DevOps Engineers @martonkodok What is BigQuery? Analytics-as-a-Service - Data Warehouse in the Cloud Scales into Petabytes on Managed Infrastructure - load up to 5TB large files SQL 2011 + Javascript UDF (User Defined Functions) Familiar DB Structure (table, columns,