Cloud Onboard

Cloud OnBoard

Welcome to Cloud OnBoard

#GCPOnBoard

©Google Inc. or its affiliates. All rights reserved. Do not distribute. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. May only be taught by Google Cloud Platform Authorized Trainers. Cloud OnBoard

2 Cloud OnBoard

3 { ( ) Registrierung 08:30 AM 5 Welcome und Building What’s Next (‘Google_Cloud’) 09:30 AM 6 (‘Module 1’) Vorstellung Google Cloud Platform 10:00 AM 7 (‘Module 2’) Starten Sie mir der Google Cloud Platform durch 10:40 AM 8 ( ) Pause 11:20 PM 9 (‘Module 3’) Virtuelle Maschinen in der Cloud 11:40 PM

10 (‘Module 4’) Storage in der Cloud 12:20 PM

11 ( ) Mittagessen 12:50 PM 12 (‘Module 5’) Container in der Cloud 01:40 PM

13 (‘Module 6’) Applikationen in der Cloud 02:50 PM

14 ( ) Pause 03:30 PM (‘Module 7’) Entwickeln, Umsetzen und Nachverfolgen in der Cloud 03:50 PM 15 (‘Module 8’) Big Data und Machine Learning in der Cloud 04:30 PM 16 } Karrierechancen mit Google Cloud| Q & A 05:10 PM 17

32 CloudCloud OnBoard

7 Introducing

9 Google Cloud Platform

10 Google Cloud Platform Fundamentals: Core Infrastructure 11 V4.0

32 Cloud OnBoard

2 Computing trends toward pay-as-you-go, fully automated services

Now6 Next

12 Storage Processing Memory Network Storage Processing Memory Network

14 Physical/Colo Virtualized Serverless

16 User-configured, managed, and maintained Fully automated 17

32 Cloud OnBoard

8 Every company is

10 a data company

32 Cloud OnBoard

1 GCP offers a range of computing architectures 2

9 Kubernetes Cloud Compute Engine App Engine Managed 10 Engine Functions services 11

12 IaaS Hybrid PaaS Serverless Automated elastic logic resources 13

16 Toward managed Toward dynamic infrastructure infrastructure 17

32 Google Cloud Platform Regions GCP is adding 10 new regions in 2017/2018

Netherlands 3 Finland 2 London 3 3 3 Frankfurt Oregon 2 3 Montreal Iowa 4 Belgium

California 3 3 N Virginia 3 Tokyo 3 S Carolina 3 Taiwan 3 Mumbai

2 Singapore

3 São Paulo

3 Sydney

# Future region and number of zones

# Current region and number of zones $29B / Trailing 3 Year CAPEX Investment Cloud OnBoard

32 Google is committed to environmental responsibility

100% carbon neutral One of the world’s First data centers to since 2007 largest corporate achieve ISO 14001 purchasers of certification renewable energy

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other company and product names may be trademarks of the respective companies with which they are associated. Google Cloud Network Google Cloud’s well-provisioned global network is comprised of hundreds of thousands of miles of fiber optic cable and seven submarine cable investments

FASTER (US, JP, TW) 2016

SJC (JP, HK, SG) 2013 Unity (US, JP) 2010

PLCN Unity (HK, LA) in construction for 2018

Monet (US, BR) in construction for 2017

Junior (Rio, Santos) Network in construction

Tannat (BR, UY, AR) Network sea cable investments in construction Indigo (SG, ID, AU) in construction for 2019 Edge points of presence >100 $29B / Trailing 3 Year CAPEX Investment Cloud OnBoard

1 Google offers customer-friendly pricing innovations

5 Billing in Discounts for Custom VM

6 sub-hour sustained use instance types 7 increments

9 For virtual machines Automatically Pay only for the 10 and containers in applied to virtual resources you need

11 the cloud; data machine use over for your application

12 processing and other 25% of a month services too 13

32 Cloud OnBoard

3 Google Technology

6 2002 2004 2005 2006 2010 2012 2014 2015

12 GFS Map Reduce BigTable Dremel Flume Java Millwheel Dataflow Tensorflow

32 Open Sourced to the world..

2004 2005 2006 ...and creating a revolution

2005 2008 2012 Cloud OnBoard

1 Open APIs and open source mean flexibility

3 Open APIs; Open source for a rich Multi-vendor-friendly 5 compatibility with ecosystem technologies 6 open-source services

11 Google Stackdriver Cloud Bigtable Kubernetes 12

15 Forseti Security 16 Cloud Dataproc Kubernetes Engine

32 Cloud OnBoard

1 Security is designed into Google’s technical infrastructure

5 Layer Notable security measures (among others)

6 Intrusion detection systems; techniques to reduce insider risk; employee Operational security 7 U2F use; software development practices

8 Internet communication Google Front End; designed-in Denial of Service protection 9

10 Storage services Encryption at rest

11 User identity Central identity service with support for U2F 12

13 Service deployment Encryption of inter-service communication

14 Hardware infrastructure Hardware design and provenance; secure boot stack; premises security 15

32 Titan Google’s purpose-built chip to establish hardware root of trust for both machines and peripherals on cloud infrastructure Encryption by Default

Data is chunked and Data encryption Encrypted chunks and All connections to each chunk is keys (DEKs) are wrapped encryption keys are Google Cloud encrypted with its own wrapped using a distributed across Google’s require TLS data encryption key key encryption key storage infrastructure. (KEK) Why choose Google Cloud Platform? Cloud OnBoard

1 2 Google Cloud Platform 3

5 enables developers to 6 build, test, and 7

8 deploy applications on 9 Google’s highly 10 secure, reliable, and 11

12scalable 13infrastructure. 14

32 Cloud OnBoard

2 Google Cloud Platform offers services for getting value from data

3 Compute Storage 5

8 Cloud Cloud 9 Compute Kubernetes App Engine Cloud Bigtable Cloud Cloud SQL Engine Functions Storage Spanner Datastore 10 Engine

11 Big Data Machine Learning

15 Natural Vision API Machine Speech Translate 16 BigQuery Pub/Sub Dataflow Dataproc Datalab Language Learning API API

17 API

32 Cloud OnBoard

1 Review 2

3 Name some of Google Cloud ● Sub-hour billing 5 Platform’s pricing ● Sustained-use discounts 6 innovations. ● Compute Engine custom machine types

11 Name some benefits of using ● Commitment to environmental responsibility

12 Google Cloud Platform other ● Commitment to open-source technologies

13 than its pricing. ● Robust infrastructure

16 Cloud OnBoard

6 Getting Started with

7 8 Google Cloud Platform 9

10 GCP Fundamentals: Core Infrastructure 11 V4.0

14 15

16 Cloud OnBoard

Cloud OnBoard

On- Infrastructure Platform as a Managed 1 Responsibility premises as a Service Service services Cloud security 2 Content requires collaboration

3 Access policies

5 Usage ● Google is responsible

6 for Managing its Deployment infrastructure Security. 7 Web application security

8 ● You are responsible for Identity Securing your data. 9 Operations

10 ● Google helps you with Access and authentication best practices, 11 Network security templates, products, 12 and solutions. OS, data, and content 13 Audit logging 14 Network 15 Customer-managed Google-managed Storage and encryption 16 © 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other company and product names may be trademarksHardware of the respective companies with which they are associated. Cloud OnBoard

1 Agenda 2

5 Google Cloud Platform resource hierarchy

6 Identity and Access Management (IAM) 7

8 Interacting with Google Cloud Platform 9

10 Cloud Launcher

12 Review

16 Cloud OnBoard

1 Resource hierarchy levels 2 define trust boundaries 3

5 ● Group your resources 6 according to your 7 organization structure

8 ● Levels of the hierarchy 9 provide trust boundaries 10 and resource isolation.

16 Cloud OnBoard

1 Projects organize resources 2

3 ● 5 All Google Cloud Platform services you use are associated 6 with a project

7 ● Use the project to: 8

9 Track resource and quota usage Enable billing. 10 Manage permissions and

11 credentials. Enable services and APIs. 12

16 Cloud OnBoard

1 Folders offer flexible management 2

3 example.com ● Folders group projects under an 5 organization.

6 ● Folders can contain projects, 7 other

8 Prod folders, or both.

9 ● Use folders to assign policies. 10 ExMail

14 Ex Drive Ex Cal Dev Prod Corp 15

16 Cloud OnBoard

1 The organization node organizes 2 projects 3

5 ● 6 The organization node is the root [email protected] example.com node for Google Cloud resources. 7 Organization Admin ● 8 Notable organization roles:

9 ◦ Organization Policy Administrator: Broad control over all cloud resources 10 Create ◦ Project Creator: Fine-grained control of 11 project creation

12 Ex Drive Ex Mail 13 [email protected] 14 Project Creator 15

16 Cloud OnBoard

1 An example IAM 2

3 resource hierarchy example.com Organization 5 ● A policy is set on 6 a resource.

◦ Each policy contains, 7

Project a set of roles and bookshelf static-assets stream-ingest and role members. 8 ● Resources inherit policies

9 Policy Inheritance from parent.

10 ◦ Resource policies are a union of parent and resource. 11 Compute App Cloud Cloud Cloud BigQuery Engine Engine Storage Storage Pub/Sub ● A less restrictive parent 12 policy overrides a more

Resources restrictive resource policy. 13

15 instance_a queue_a bucket_a bucket_b topic_a dataset_a

16 29 Cloud OnBoard

1 Agenda 2

5 Google Cloud Platform resource hierarchy

6 Identity and Access Management (IAM) 7

8 Interacting with Google Cloud Platform 9

10 Cloud Launcher

12 Quiz

16 Cloud OnBoard

1 Google Cloud Identity and Access Management defines.. 2

12 Who can do what on which resource

16 Cloud OnBoard

1 IAM policies can apply to any of four types of principals 2

5 Google account test@gmail.com 6

8 Service account test@project_id.iam.gserviceaccount.com

11 Google group [email protected] 12 Who

13 G Suite or Cloud 14 Identity domain 15 [email protected]

16 32 Cloud OnBoard

1 There are three types of IAM roles 2

12 Beta 13 Primitive Predefined Custom

16 Cloud OnBoard

2 IAM primitive roles apply across all GCP services in a

3 project

12 can do what on all resources 13

16 Cloud OnBoard

1 IAM primitive roles offer fixed, coarse-grained levels of access 2

9 Owner Editor Viewer Billing administrator

10 ● Invite members ● Deploy ● Read-only ● Manage billing 11 ● Remove members applications access ● Add and remove ● Delete projects ● Modify code administrators 12 ● And... ● Configure services 13 ● And...

15 A project can have multiple owners, editors, viewers, and billing administrators. 16 Cloud OnBoard

2 IAM predefined roles apply to a particular GCP service in a

3 project

12 on Compute Engine resources in 13 can do what this project, or folder, or org

16 Cloud OnBoard

1 IAM predefined roles offer more fine-grained permissions on particular 2 services 3

5 .. 6 Google

7 Group ✔ compute.instances.delete 8 ✔ compute.instances.get 9 InstanceAdmin ✔ compute.instances.list 10 Role ✔ compute.instances.setMachineType 11 ✔ compute.instances.start 12 ✔ compute.instances.stop 13

14 project_a 15

16 Cloud OnBoard

1 IAM custom rolesBeta let you define a precise set of permissions 2

6 Google ✔ compute.instances.get 7 Group ✔ 8 compute.instances.list ✔ 9 compute.instances.start InstanceOperator ✔ compute.instances.stop 10 Role 11

14 project_a 15

16 Cloud OnBoard

1 Service Accounts control server-to-server interactions 2

5 ● Provide an identity for carrying out server-to-server interactions in a project

6 ● Used to authenticate from one service to another 7

8 ● Used to control privileges used by resources ○ So that applications can perform actions on behalf of authenticated end users 9

10 ● Identified with an email address:

12 [email protected]

13 [email protected]

16 © 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other 40 company and product names may be trademarks of the respective companies with which they are associated. Cloud OnBoard

2 Identity IAM Role Resource Service Accounts and IAM

5 ● Service accounts authenticate

6 using keys.

7 ◦ Google manages keys for Service Account InstanceAdmin Role Compute Instances Compute Engine and App 8 Engine. 9 ● You can assign a curated or 10 custom IAM role to the 11 service account.

12 ● You can also assign the 13 User/Group ServiceAccountActor Service Account ServiceAccountActor role to

14 Role users and groups.

● Service accounts authenticate using keys.

○ Google manages keys for Compute Engine and App Engine. ● You can assign a curated or custom IAM role to the service account. ● You can also assign the ServiceAccountActor role to users and groups. Cloud OnBoard

1 Ex Mail Ex Drive Example: Service Accounts 2 and IAM

5 ● VMs running FrontEnd are 6 granted Editor access to FrontEnd VM Service 7 Account 1 project_b using Service Editor Account 1. 8

9 ● VMs running BackEnd are granted objectViewer access 10 to bucket_1 using Service 11 Account 2. Service 12 BackEnd VM Account 2 ● Service account permissions 13 Storage. can be changed without objectViewer 14 recreating VMs.

15 bucket_1 16

42 Cloud OnBoard

1 Agenda 2

5 Google Cloud Platform resource hierarchy

6 Identity and Access Management (IAM) 7

8 Interacting with Google Cloud Platform 9

10 Cloud Launcher

12 Review

16 Cloud OnBoard

1 There are four ways to interact with GCP 2

5 Cloud Platform Cloud Shell Cloud Console REST-based API 6 Console and Cloud SDK Mobile App 7 For custom

8 Web user Command-line For iOS and applications

9 interface interface Android

12 >_

16 Cloud OnBoard

1 Google Cloud Platform Console 2

5 ● Centralized console for all project data

6 ● Developer tools

7 ○ Cloud Source Repositories

8 ○ Cloud Shell ○ Test Lab (mobile app testing) 9 ● Access to product APIs 10

11 ● Manage and create projects

16 Cloud OnBoard

1 Google Cloud SDK 2

3 ● SDK includes CLI tools for Cloud Platform 5 products and services 6 ○ gcloud, gsutil (Cloud Storage), 7 bq (BigQuery)

8 ● Available as Docker image 9 ● Available via Cloud Shell 10

11 ○ Containerized version of Cloud SDK running on Compute Engine instance 12

16 Cloud OnBoard

1 RESTful APIs 2

3 ● Programmatic access to products and services 5

6 ○ Typically use JSON as an interchange format

7 ○ Use OAuth 2.0 for authentication and authorization

8 ● Enabled through the Google Cloud Platform Console 9

10 ● Most APIs include daily quotas and rates (limits)

11 that can be raised by request

12 ○ Important to plan ahead to manage your required capacity 13

16 Cloud OnBoard

2 APIs Explorer

5 ● The APIs Explorer is an interactive tool that lets you easily try Google 6 APIs using a browser.

7 ● With the APIs Explorer, you can: 8

9 ○ Browse quickly through available APIs and versions.

10 ○ See methods available for each API and what parameters they support 11 along with inline documentation. 12

13 ○ Execute requests for any method and see responses in real time.

15 ○ Easily make authenticated and authorized API calls.

16 Cloud OnBoard

1 Cloud Console Mobile App 2

5 ● Manage virtual machines and

6 database instances

7 ● Manage apps in Google App 8 Engine 9 ● Manage your billing 10 ● Visualize your projects with a 11

12 customizable dashboard

16 Cloud OnBoard

2 Client Libraries

5 ● Cloud Client Libraries 6

7 ○ Community-owned, handcrafted client libraries

8 ● Google API Client Libraries 9

10 ○ Open source, generated

11 ○ Support various languages 12

13 ● Java, Python, JavaScript, PHP, .NET, Go, Node.js, Ruby, 14 Objective-C, Dart

16 Cloud OnBoard

1 Agenda 2

5 Google Cloud Platform resource hierarchy

6 Identity and Access Management (IAM) 7

8 Interacting with Google Cloud Platform 9

10 Cloud Launcher

12 Review

16 Cloud OnBoard

2 Cloud Launcher gives quick 3 access to solutions 5

6 ● A solution marketplace containing 7 pre-packaged, ready-to-deploy 8 solutions 9 ○ Some offered by Google 10 ○ Others by third-party vendors 11

12 ● You pay for the underlying GCP 13 resource usage. 14 ○ Some solutions also assess 15 third-party license fees. 16 Cloud OnBoard

5 Google Cloud Platform resource hierarchy

6 Identity and Access Management (IAM) 7

8 Interacting with Google Cloud Platform 9

10 Cloud Launcher

12 Review

16 Cloud OnBoard

1 Quiz: Service Accounts 2

3 Service accounts are used to provide which of the following? 5

6 ❏ Authentication between Google Cloud Platform services 7 ❏ Key generation and rotation when used with App Engine and Compute Engine 8 ❏ A way to restrict the actions a resource (such as a VM) can perform 9 ❏ A way to allow users to act with service account permissions

10 ✓ All of the above

16 Cloud OnBoard

6 Virtual Machines

7 8 in the Cloud 9

10 GCP Fundamentals: Core Infrastructure

11 V4.0 12 13

14 15

16 Cloud OnBoard

10 Purpose-built Purpose-built Purpose-built Purpose-built Purpose-built data 11 chips servers storage network centers 12

16 Cloud OnBoard

Agenda 1 Agenda 2

6 Virtual Private Cloud (VPC) Network

8 Compute Engine

10 Operations and tools

11 Review 12

16 Cloud OnBoard

2 Virtual Private Cloud Network

5 ● Managed networking

6 functionality for Google

7 Cloud Platform resources

9 10 ● Provision Cloud Platform 11 resources, connect them to 12 each other, and isolate them 13

14 from one another in a

15 Virtual Private Cloud (VPC).

16 Cloud OnBoard

1 Google Cloud VPCs are global; subnets are regional 2

5 My VPC

6 us-east1

7 my-subnet us-east1-b us-east1-c 8 1

13 10.0.0.0/24 14

15 10.0.0.2 10.0.0.3

16 Cloud OnBoard

Agenda 1 Agenda 2

6 Virtual Private Cloud (VPC) Network

8 Compute Engine

10 Operations and tools

11 Quiz 12

16 Cloud OnBoard

1 Compute Engine offers managed 2

3 virtual machines

6 ● High CPU, high memory, standard

7 and shared-core machine types

8 ● Persistent disks 9

10 ○ Standard, SSD, local SSD 11 ○ Snapshots

12 ● 13 Resize disks with no downtime

14 ● Instance metadata and startup 15 scripts

16 Cloud OnBoard

1 Compute Engine offers 2

3 innovative pricing

6 ● Per-second billing, sustained use

7 discounts 8 ● Preemptible instances 9

10 ● High throughput to storage at no

11 extra cost 12 ● Custom machine types: Only pay for 13 the hardware you need 14

16 Cloud OnBoard

Agenda 1 Agenda 2

6 Virtual Private Cloud (VPC) Network

8 Compute Engine

10 Operations and tools

11 Review 12

16 Cloud OnBoard

2 VPC Network offers many

3 internetworking features

5 ● 6 Fine-grained networking

7 policies

8 ● Fine-grained IP address

9 range selection 10 ● Routes 11 ● Firewalls 12 ● Virtual Private Network 13 (VPN) 14 ● Cloud Router 15

16 Cloud OnBoard

1 Google Cloud Platform offers many interconnect options 2

8 Carrier Interconnect Direct Peering CDN Interconnect

9 Enterprise-grade Connect your business Allows select CDN providers to establish direct

10 connections provided by directly to Google interconnect links with Google’s edge network at carrier service providers various locations 11

16 Google Cloud Interconnect Cloud OnBoard

2 Cloud DNS is highly available

3 and scalable

6 ● Create managed zones, then add,

7 edit, delete DNS records

8 ○ Programmatically manage zones 9 and records using RESTful API 10 or command-line interface 11

16 1

16 Cloud OnBoard

1 Cloud Load Balancing: HTTP(S) 2

5 ● Balance HTTP-based traffic across

6 multiple Compute Engine regions

7 ● Global, external IP address routes 8 traffic = 9 ● Traffic is directed only to 10 instances that pass health checks 11

12 ● Scalable, requires no pre-warming

13 and provides resilience, fault

14 tolerance

16 Cloud OnBoard

1 Cloud Load Balancing: 2

3 TCP/SSL, UDP

6 ● Spread TCP/SSL and UDP traffic 7 over pool of instances within a 8 Compute Engine region

9 ● 10 Traffic is directed only to

11 instances that pass health checks

12 ● Scalable, requires no pre-warming

16 Cloud OnBoard

1 Cloud CDN (Content Delivery 2

3 Network)

5 ● Use Google's globally distributed 6 edge caches to cache HTTP(S) 7 load-balanced content far closer to 8 your users than your instances 9

10 ○ Faster delivery of content to users while reducing costs 11

12 ● Cloud CDN uses caches at network

13 locations to store responses

14 generated by instances

16 Cloud OnBoard

Agenda 1 Agenda 2

6 Virtual Private Cloud (VPC) Network

8 Compute Engine

10 Operations and tools

11 Review 12

16 Cloud OnBoard

Quiz 1

2 Name 3 robust networking Cloud Virtual Network, Cloud 3 services available to your Interconnect, Cloud DNS, Cloud

5 applications on Google Cloud Load Balancing, and Cloud CDN.

6 Platform.

8 Name 3 Compute Engine pricing Per-second billing, custom machine innovations. types, preemptible instances. 9

12 True or False: Google Cloud True. 13 Load Balancing lets you balance 14 HTTP traffic across multiple

15 Compute Engine regions.

16 Cloud OnBoard

5 6 Storage inStorage the in Cloud the Cloud 7

8 GCP Fundamentals: Core Infrastructure V4.0 GCP Fundamentals: Core Infrastructure

11 Getting Started with Cloud Storage and Cloud SQL 12

14Last modified 152017-11-13

16 Cloud OnBoard

1 Google Cloud Platform 2

6 Machine Operations 7 Compute Networking Big Data Storage Learning and Tools 8

13 Cloud Cloud Cloud Cloud Cloud

14 Storage SQL Spanner Datastore Bigtable

16 Cloud OnBoard

1 Separation of Storage and Compute 2

6 Cloud Dataflow BigQuery Cloud Dataproc Cloud ML 7 Engine

9 Processing 10

11 Storage

15 Cloud Storage BigQuery Storage Cloud Bigtable 16 (files) (tables) (NoSQL) Cloud OnBoard

Agenda Agenda 1

2 Cloud Storage

5 Cloud Bigtable

7 Cloud SQL and Cloud Spanner

8 Cloud Datastore 9

10 Comparing storage options 11

16 Cloud OnBoard

1 Cloud Storage is binary 2

3 large-object storage

5 ● High performance, internet-scale 6 ● Simple administration 7

8 ○ Does not require capacity

9 management

10 ● Data encryption at rest 11 ● Data encryption in transit by default 12 from Google to endpoint

13 ● Online and offline import services are 14 available 15

16 77 Cloud OnBoard

1 Your Cloud Storage files are organized into buckets 2

5 Bucket attributes: Bucket contents: 6

7 ● Globally unique name ● Files (in a flat namespace)

8 ● Storage class ● Access Control Lists

9 ● Location

10 ○ Region or multi-region

11 ● IAM policies or

12 Access Control Lists ● Object versioning setting 13 ● Object lifecycle management 14 rules 15

16 Cloud OnBoard

Choosing among Cloud Storage classes 1

2 Multi-regional Regional Nearline Coldline 3

5 Intended for data that is... Most frequently Accessed frequently Accessed less than Accessed less than 6 accessed within a region once a month once a year

7 Availability SLA 99.95% 99.90% 99.00% 99.00% 8

9 Access APIs Consistent APIs

10 Access time Millisecond access

11 Storage price 12 Price per GB stored per month 13 Retrieval price Total price per GB transferred 14 Use cases Content storage and In-region analytics, Long-tail content, Archiving, 15 delivery transcoding backups disaster recovery 16 Cloud OnBoard

Agenda Agenda 1

2 Cloud Storage

5 Cloud Bigtable

7 Cloud SQL and Cloud Spanner

8 Cloud Datastore 9

10 Comparing storage options 11

16 Cloud OnBoard

1 Cloud Bigtable is managed 2

3 NoSQL

6 ● Fully managed NoSQL, wide-column 7 database service for terabyte 8 applications

9 ● 10 Integrated

11 ○ Accessed using HBase API

12 ○ Native compatibility with big 13 data, Hadoop ecosystems 14

16 Cloud OnBoard

1 Why choose Cloud Bigtable? 2

5 ● Replicated storage 6 ● Data encryption in-flight and at 7 rest 8

9 ● Role-based ACLs 10 ● High throughput writes, low 11 latency reads 12

13 ● Drives major applications such as

14 Google Analytics and Gmail

16 Cloud OnBoard

1 Bigtable Access Patterns 2

3 Application API Data can be read from and written to Cloud 5 Bigtable through a data service layer like Managed

6 VMs, the HBase REST Server, or a Java Server using the HBase client. Typically this will be to serve 7 data to applications, dashboards, and data services. 8

9 Streaming

10 Data can be streamed in (written event by event) through a variety of popular stream processing 11 frameworks like Cloud Dataflow Streaming, Spark Streaming, and Storm. 12 Cloud Bigtable Batch Processing 13 Data can be read from and written to Cloud

14 Bigtable through batch processes like Hadoop MapReduce, Dataflow, or Spark. Often, summarized 15 or newly calculated data is written back to Cloud Bigtable or to a downstream database. 16 Cloud OnBoard

3 Follows 5

6 Row Key gwashington jadams tjefferson wmckinley

7 gwashington 1 1

8 jadams 1 1 9 tjefferson 1 10

11 wmckinley 1

13 multiple versions

16 Cloud OnBoard

3 Clients

8 LB / proxy 9

12 Bigtable Bigtable Bigtable Processing 13 node node node

16 Cloud OnBoard

3 Clients

8 Bigtable Bigtable Bigtable Processing 9 node node node

13 Storage Colossus file system 14

16 Cloud OnBoard

3 Clients

8 Bigtable Bigtable Bigtable Processing 9 node node node

13 Storage A B C D E 14 Colossus file system 15

16 Cloud OnBoard

3 Clients

8 Bigtable Bigtable Bigtable Processing 9 node node node

13 Storage A B C D E 14 Colossus file system 15

16 Cloud OnBoard

1 QPS 2 Nodes 3 80,000 5 Node Node Node

6 60,000

8 40,000

10 20,000

12 0 13 2 4 6 8

14 Bigtable nodes

16 Cloud OnBoard

1 QPS 2 Nodes 3 400,000 5 Node Node Node Node Node Node

6 Node Node Node Node Node Node 300,000 7 Node Node Node Node Node Node 8 200,000 9 Node Node Node Node Node Node

10 Node Node Node Node Node Node 100,000 11

12 Node Node Node Node Node Node 0 13 10 20 30 40

14 Bigtable nodes

16 Cloud OnBoard

1 QPS 2 Nodes 3 Node Node Node Node Node Node Node Node 4,000,000 Node Node Node Node Node Node Node Node 5 Node Node Node Node Node Node Node Node 6 Node Node Node Node Node Node Node Node 3,000,000

7 Node Node Node Node Node Node Node Node

Node Node Node Node Node Node Node Node 8 Node Node Node Node Node Node Node Node 2,000,000

9 Node Node Node Node Node Node Node Node

Node Node Node Node Node Node Node Node 10 Node Node Node Node Node Node Node Node 1,000,000 11 Node Node Node Node Node Node Node Node

12 Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node 0 13 100 200 300 400 Node Node Node Node Node Node Node Node

14 Node Node Node Node Node Node Node Node Bigtable nodes

16 Cloud OnBoard

Agenda Agenda 1

2 Cloud Storage

5 Cloud Bigtable

7 Cloud SQL and Cloud Spanner

8 Cloud Datastore 9

10 Comparing storage options 11

16 Cloud OnBoard

1 Cloud SQL is a managed RDBMS 2

3 ● 5 Offers MySQL and PostgreSQL DBaaS

6 ● Automatic data replication 7 ● High Availability (failover)

8 ● Managed backups and updates 9 ● Vertical scaling (read and write) 10

11 ● Horizontal scaling (read replicas)

12 ● Google security

16 Cloud OnBoard

1 Cloud Spanner is a 2

3 horizontally scalable RDBMS

6 supports:

7 ● 8 Global synchronous replication ● 9 Strong consistency

10 ● Automatic data sharding

11 ● Fully managed

12 ● Scaling with no downtime

13 ● SQL (ANSI 2011 with extensions)

14 ● JDBC

16 Cloud OnBoard

1 When is Cloud Spanner the Right Fit? 2

6 Transactional Database 7 Consistency Scale Global Data Consolidation 8

9 Companies that have Companies currently Companies and/or Companies that store outgrown a sharding databases developers building their business data in 10 single-instance RDBMS because they need more applications that rely multiple database 11 and moved to a NoSQL read or write on global data and products with variable solution but need throughput than can be require strong maintenance overheads and 12 transactional placed on a single node consistency for accuracy capabilities and need to 13 consistency, or are consolidate their data looking to move to a 14 scalable solution 15

16 Cloud OnBoard

1 Regional 2

6 Cloud Spanner Regional Instance 7

8 Europe region 1

9 Zone A Zone B Zone C

10 RW - Replica RW - Replica RW - Replica

16 Cloud OnBoard

1 Multi-Region 2

3 Write Quorum (US) Asia Region Europe Region 5 US region 1 (Default Leader) Asia region 1 Europe region 1 6

7 Zone A Zone B Zone A Zone A RW - Replica RW - Replica RO - Replica RO - Replica 8

9 US region 2 Asia region 2 Europe region 2 10 Zone A Zone A 11 Zone A Zone B RW - Replica RW - Replica RO - Replica RO - Replica

13 US region 3 (Witness)

14 Zone A 15 Witness

16 Cloud OnBoard

1 Data Replacement and Replication 2

3 Update

7 Zone 1 Zone 2 Zone 3

9 Table 1 Table 1 Table 1

10 Table 2 Table 2 Table 2

14 Data is synchronously replicated using Paxos consensus. 15

16 Cloud OnBoard

1 Multiregion Write Request 2

7 Geography - US RW Geography - ASIA RO

8 Multi-Region Spanner Multi-Region Spanner

9 Write

15 US Asia 16 Client Client Cloud OnBoard

1 Multi Region Strong Read Request 2

7 Geography - US RW Geography - ASIA RO

8 Multi-Region Spanner Multi-Region Spanner

9 RTT Strong Read 10

13 Asia Client 14

16 Cloud OnBoard

1 Multi Region Bounded Read Request 2

7 Geography - US RW Geography - ASIA RO

8 Multi-Region Spanner Multi-Region Spanner

9 Bounded Read 10

12 Asia 13 Serve from local Client 14 replica w/o RTT to

15 Leader

16 Cloud OnBoard

1 Relational Data Model 2

3 CREATE TABLE Singers ( Schema changes without downtime 5 SingerId INT64 NOT NULL, 6 SingerName STRING(MAX), ALTER TABLE Singers

7 ) PRIMARY KEY(SingerId); ADD COLUMN Age INT64;

8 CREATE TABLE Albums ( 9 SingerId INT64 NOT NULL, 10 AlbumId INT64 NOT NULL,

11 AlbumName STRING(MAX), ) PRIMARY KEY(SingerId, AlbumId), INTERLEAVE IN PARENT Singers; 12

14 Interleave = Ideal for hierarchical data; pre-join data for fast scans 15

16 Cloud OnBoard

1 Interleaving: Logical Data Layout 2

6 SingerId SingerName SingerId AlbumId AlbumName

7 1 Beatles 1 1 Help! 8

9 2 U2 1 2 Abbey Road 10

11 3 Pink Floyd 3 1 The Wall

16 Cloud OnBoard

1 Interleaving: Physical Data Layout 2

3 1 Beatles 5

6 1 1 Help!

8 1 2 Abbey Road

10 2 U2

12 3 Pink Floyd

13 3 1 The Wall 14

16 Cloud OnBoard

Agenda Agenda 1

2 Cloud Storage

5 Cloud Bigtable

7 Cloud SQL and Cloud Spanner

8 Cloud Datastore 9

10 Comparing storage options 11

16 Cloud OnBoard

1 Cloud Datastore is a 2

3 horizontally scalable NoSQL DB

6 NoSQL designed for application backends

7 ● Fully managed 8 ○ Uses a distributed architecture 9 to automatically manage scaling 10 ● Built-in redundancy 11

12 ● Supports ACID transactions

16 Cloud OnBoard

1 Google Cloud Datastore: 2 benefits 3

6 ● Schemaless access

7 ○ No need to think about 8 underlying data structure 9

10 ● Local development tools 11 ● Includes a free daily quota 12

13 ● Access from anywhere through

14 a RESTful interface

16 Cloud OnBoard

1 Google Cloud Firestore: some 2 key differences with Datastore 3

5 ● Data Model : Collections (Kinds), Documents 6 (Entities), References (Keys)

7 ● Client SDKs: Mobile/web clients - latency 8 compensation, write queuing and pipelining, offline

9 caching and more

10 ● Real-time queries: Clients will be able to

11 listen to changes for a query in "real-time"

12 ● Shallow queries: Shallow queries are the 13 default. Queries will be primarily around collections, and will not contain results from subcollections. 14

15 ● Easier Indexing: By default all document fields will be indexed 16 Cloud OnBoard

Agenda Agenda 1

2 Cloud Storage

5 Cloud Bigtable

7 Cloud SQL and Cloud Spanner

8 Cloud Datastore 9

10 Comparing storage options 11

16 Cloud OnBoard

Is your data No structured? 1

2 Do you need Yes Mobile SDK’s? 3

Is your workload 5 Yes No Yes analytics?

Cloud Firebase 7 No Storage Storage

8 Is your data Yes 9 relational?

No 11

12 Do you need Do you need Do you need updates horizontal scalability? Mobile SDK’s? or low-latency? 13

14 No Yes No Yes Yes No

16 Cloud Cloud Cloud Firebase Cloud BigQuery SQL Spanner Datastore Realtime DB Bigtable Cloud OnBoard

1 Comparing storage options: technical details

3 Cloud Bigtable Cloud Cloud Cloud BigQuery 5 Datastore Storage SQL Spanner

6 Type NoSQL NoSQL Blobstore Relational Relational Relational 7 document wide column SQL for OLTP SQL for OLTP SQL for OLAP 8

9 Transactions Yes Single-row No Yes Yes No

10 Complex No No No Yes Yes Yes 11 queries 12

13 Capacity Terabytes+ Petabytes+ Petabytes+ 500 GB Petabytes Petabytes+

14 Unit size 1 MB/entity ~10 MB/cell 5 TB/object Determined 10,240 MiB/ 10 MB/row 15 ~100 MB/row by DB engine row

16 Cloud OnBoard

1 Comparing storage options: use cases

3 Cloud Bigtable Cloud Cloud Cloud BigQuery 5 Datastore Storage SQL Spanner

7 Type NoSQL NoSQL Blobstore Relational Relational Relational document wide column SQL for OLTP SQL for OLTP SQL for OLAP 8

9 Best for Getting started, “Flat” data, Structured and Web Large-scale Interactive App Engine Heavy unstructured frameworks, database querying, offline 10 applications read/write, binary or object existing applications (> analytics 11 events, data applications ~2 TB)

12 analytical data

13 Use cases Getting started, AdTech, Images, large User credentials, Whenever high Data

14 App Engine Financial and media files, customer orders I/O, global warehousing applications IoT data backups consistency is 15 needed

16 Cloud OnBoard

1 Separation of Storage and Compute 2

6 Cloud Dataflow BigQuery Cloud Dataproc Cloud ML 7 Engine

9 Processing 10

11 Storage

15 Cloud Storage BigQuery Storage Cloud Bigtable 16 (files) (tables) (NoSQL) Cloud OnBoard

6 Containers in the Cloud

8 GCP Fundamentals: Core Infrastructure V4.0

14 15

16 Cloud OnBoard

2 IaaS and PaaS

7 Toward Compute Engine Kubernetes Engine App Engine Toward 8 managed managed 9 infrastructure services

10 IaaS PaaS Raw compute, storage, and network Preset run-times 11 More granular control Java, Go, PHP, Python...

12 Focus is application logic

14 Pay for what you allocate Pay for what you use More management overhead Less management overhead 15

16 Cloud OnBoard

Agenda Agenda 1

6 Introduction to Containers

7 Kubernetes 8

9 Kubernetes Engine 10

11 Quiz

16 Cloud OnBoard

1 Why use containers? 2

5 Consistency Loose coupling Workload Agility 6 migration 7 Across Between Agile

8 development, application Simplified development and

9 testing, and and operating between operations

10 production system layers on-premises

11 environments and cloud

12 environments

16 Cloud OnBoard

2 Virtual Machines VS Containers

16 Cloud OnBoard

1 Containers mean virtualization 2

3 inside the operating-system

5 layer

6 Container Container Container Code and Code and Code and 7 ● Separates operating system libraries libraries libraries 8 from application code and

9 Shared libraries dependencies

10 Operating system ● Isolates individual processes 11

12 ● Popular implementations Virtual Machine 13 include Docker

16 Cloud OnBoard

Agenda Agenda 1

6 Introduction to Containers

7 Kubernetes 8

9 Kubernetes Engine 10

11 Review

16 Cloud OnBoard

1 Kubernetes is a container 2

3 cluster orchestration system

6 ● Automates deployment, scaling,

7 and operations for container

8 clusters 9 ● Open source, based on Google’s 10 experience over 10+ years 11

12 ● Built for a multi-cloud world 13 ○ Public, private, hybrid 14

16 Cloud OnBoard

1 Kubernetes eases application 2

3 management

5 ● Workload portability 6

7 ○ You can run in many environments, across cloud providers. 8

9 ○ Implementation is open and modular.

10 ● Rolling updates

11 ○ You can upgrade applications without 12 downtime.

13 ● Persistent storage 14 ○ Details of how storage is provided are 15 abstracted from how it is consumed. 16 Cloud OnBoard

1 Kubernetes makes applications 2

3 more elastic

6 ● Multi-zone clusters

7 ○ Run a single cluster in multiple 8 zones

9 ● Load balancing 10 ○ External IP address routes traffic to 11 correct port 12

13 ● Autoscaling

14 ○ Automatically adapt to changes in 15 workload

16 Cloud OnBoard

Agenda Agenda 1

6 Introduction to Containers

7 Kubernetes 8

9 Kubernetes Engine 10

11 Review

16 Cloud OnBoard

1 Kubernetes Engine manages and 2

3 runs containers

6 ● Fully managed cluster management and 7 orchestration system for running containers 8 ○ Based on Kubernetes 9

10 ○ Uses Compute Engine instances and resources 11

12 ● Uses a declarative syntax to manage applications 13

14 ○ Declare desired application configuration, Kubernetes Engine 15 implements, manage 16 Cloud OnBoard

1 Why use Kubernetes Engine? 2

5 ● Decouples operational,

6 development concerns 7 ● Manages and maintains 8

9 ○ Logging, health management,

10 monitoring 11 ● Easily update Kubernetes 12 versions as they are released 13

16 Cloud OnBoard

1 Kubernetes Engine’s complementary services

11 Google Cloud Container Builder Google Container Registry

12 Create Docker container images Docker image storage 13

14 from app code in Google Cloud that’s private to your GCP

15 Storage project

16 Results Enabled by Google Cloud

● Creates a better lighting experience for customers - handles 200 million transactions every day, including 18 million remote lighting commands by customers, with Google Container Engine

● Changing traditional business models - Philips Hue offers a new approach to lighting, beyond just replacing bulbs, with GCP on the backend

“We chose Google Cloud Platform to power Philips Hue’s backend because it scales instantly, frees engineers to work on product development rather than managing infrastructure.”

George Yianni, Head of Technology-Connect Lighting for the Home, Philips Lighting & Inventor of Hue

Confidential & Proprietary 128 Cloud OnBoard

1 Deploying Apps: Kubernetes Engine vs App Engine 2

3 Kubernetes App Engine App Engine Flexible 5 Engine Standard 6

7 Language support Any Java, Python, Go Any 8 & PHP

10 Service model Hybrid PaaS PaaS 11 Primary use case Container-based Web and mobile Web and mobile 12 workloads applications applications, 13 container-based

14 workloads

16 Cloud OnBoard

1 Quiz

2 Name two reasons for deploying Consistency across development, testing, 3 applications using containers. and production environments; Simpler to 5 migrate workloads; Loose coupling; Agility

8 True or False: Kubernetes lets True you manage container clusters in 9 multiple cloud providers. 10

12 True or False: GCP provides a True 13 private, high-speed container

14 image storage service for use

15 with Kubernetes Engine.

16 Cloud OnBoard

1 2 Register NOW to get access to our Qwiklabs 3

5 1 Create Qwiklabs account with the email 6 you used to register for Cloud OnBoard

8 2 Open your email and confirm account

10 Return to Qwiklabs and log in 3 Username 11

12 4 Enroll in the GCP Essentials Quest Password

15 Start HERE!

32 Cloud OnBoard

6 Applications in the Cloud

8 GCP Fundamentals: Core Infrastructure V4.0

14 15

16 Cloud OnBoard

Agenda Agenda 1

3 Google App Engine 5

6 Google App Engine Standard Environment 7

8 Google App Engine Flexible Environment

10 Google Cloud Endpoints and Apigee Edge

12 Review

16 Cloud OnBoard

1 App Engine is a PaaS for 2 building scalable applications 3

6 ● App Engine makes deployment,

7 maintenance, and scalability easy

8 so you can focus on innovation

9 ● Especially suited for building 10 scalable web applications and 11 mobile backends 12

16 Cloud OnBoard

1 App Engine standard 2 environment 3

5 Requirements 6

7 ● Specific versions of Java, Python, PHP, 8 and Go are supported

9 ● Your application must conform to sandbox 10 constraints: 11

12 ○ No writing to local file system

13 ○ All requests time out at 60 seconds

14 ○ Third-party software installations 15 are limited

16 Cloud OnBoard

Agenda Agenda 1

3 Google App Engine 5

6 Google App Engine Standard Environment 7

8 Google App Engine Flexible Environment

10 Google Cloud Endpoints and Apigee Edge

12 Review

16 Cloud OnBoard

1 App Engine standard environment 2

5 ● Easily deploy your applications

6 ● Autoscale workloads to meet

7 demand

8 ● Economical

10 ○ Free daily quota

11 ○ Usage based pricing

13 ● SDKs for development, testing and

14 deployment

16 Cloud OnBoard

1 Example App Engine standard workflow: Web applications 2

3 App Engine automatically App Engine can access a 5 3 scales & reliably serves your variety of services using Develop & test the web 1 dedicated APIs 6 application locally web application 7 Project Memcache 8 App Engine 9 App Servers Task

10 queues Application Use the SDK to deploy to 11 2 instances Scheduled App Engine tasks 12 Application instances 13 Search Application 14 instances Logs 15

16 Cloud OnBoard

Agenda Agenda 1

3 Google App Engine 5

6 Google App Engine Standard Environment 7

8 Google App Engine Flexible Environment

10 Google Cloud Endpoints and Apigee Edge

12 Review

16 Cloud OnBoard

1 App Engine flexible 2

3 environment

5 ● Build and deploy containerized apps with a 6 click

7 ● No sandbox constraints 8

9 ● Can access App Engine resources

10 ● Standard runtimes: Python, Java, 11 Go, Node.js 12

13 ● Custom runtime support: Any language that supports HTTP requests 14 ○ Package your runtime as a Dockerfile 15

16 Cloud OnBoard

1 Comparing the App Engine environments

2 Standard Environment Flexible Environment 3

5 Instance startup Milliseconds Minutes

6 SSH access No Yes (although not by default) 7

8 Scaling Manual, basic, automatic Manual, automatic

9 Write to local disk No Yes (but writes are ephemeral) 10

11 Support for No Yes 3rd-party binaries 12

13 Network access Via App Engine services Yes

14 Pricing model After free daily use, pay per Pay for resource allocation per 15 instance class, with automatic hour; no automatic shutdown

16 shutdown Cloud OnBoard

Agenda Agenda 1

3 Google App Engine 5

6 Google App Engine Standard Environment 7

8 Google App Engine Flexible Environment

10 Google Cloud Endpoints and Apigee Edge

12 Review

16 Cloud OnBoard

1 Cloud Endpoints helps you 2 create and maintain APIs 3

5 ● 6 Distributed API management through an API console 7

8 ● Expose your API using a RESTful

9 interface

10 ● Control access and validate calls with

11 JSON Web Tokens and Google API keys

12 ○ Identify web, mobile users with 13 Auth0 and Firebase Authentication

14 ● Generate client libraries 15

16 Cloud OnBoard

1 Cloud Endpoints: Supported 2 platforms 3

6 ● Supports App Engine standard or

7 flexible environment, Compute Engine,

8 Kubernetes Engine

9 ● Use Java or Python open-source 10 Frameworks or any other framework and 11 language

12 ● Supports iOS, Android, and JavaScript 13 clients 14

16 Cloud OnBoard

1 Apigee Edge helps you secure and monetize APIs 2

6 1. A platform for making APIs available to your

7 customers

8 and partners

9 2. Contains analytics, monetization, and a 10 developer portal 11

16 Cloud OnBoard

1 Beta 2 Cloud Functions

5 ● Create single-purpose functions

6 that respond to events without a

7 server or runtime

8 ○ Event examples: New instance 9 created, file added to Cloud 10 Storage 11

12 ● Written in Javascript; execute in 13 managed Node.js environment on 14 Google Cloud Platform

16 Cloud OnBoard

Agenda Agenda 1

3 Google App Engine 5

6 Google App Engine Standard Environment 7

8 Google App Engine Flexible Environment

10 Google Cloud Endpoints and Apigee Edge

12 Review

16 Cloud OnBoard

1 Quiz 2

5 Name 3 advantages of using the The flexible environment allows SSH

6 App Engine flexible environment access, allows disk writes, and over App Engine standard. supports third-party binaries (also 7 allows stack customization and 8 background processes).

10 What is the difference between Cloud Endpoints helps you create Cloud Endpoints and Apigee Edge? and maintain APIs; Apigee Edge 11 helps you secure and monetize APIs. 12

16 Cloud OnBoard

6 Developing, Deploying, and 7 Monitoring in the Cloud 8

9 GCP Fundamentals: Core Infrastructure V4.0 10

14 15

16 Cloud OnBoard

Agenda Agenda 1

7 Development in the cloud

8 Deployment: Infrastructure as code 9

10 Monitoring: Proactive instrumentation

16 Cloud OnBoard

1 Deployment Manager 2

5 ● Infrastructure management 6 service 7

8 ● Create a .yaml template 9 describing your environment and 10 use Deployment Manager to create 11 resources

12 ● Provides repeatable deployments 13

16 Cloud OnBoard

Agenda Agenda 1

7 Development in the cloud

8 Deployment: Infrastructure as code 9

10 Monitoring: Proactive instrumentation

16 Cloud OnBoard

7 Monitoring Logging Debug

10 Error Reporting Trace 11

16 Cloud OnBoard

1 Stackdriver offers capabilities in five areas 2

5 Monitoring Logging Trace

6 Platform, system, and Platform, system, and Latency reporting and 7 application metrics application logs sampling 8 Uptime/health checks Log search, view, Per-URL latency and 9 filter, and export statistics 10 Dashboards and alerts

11 Log-based metrics

12 Error Reporting 13 Debugger 14 Error notifications Debug applications 15 Error dashboard

16 3 Finland

3 NOW London 3 2 Netherlands OPEN 3 3 Frankfurt Current region and # of zones Belgium Future region and # of zones Google Cloud Network

FASTER (US, JP, TW) 2016

Unity (US, JP) 2010

PLCN Unity (HK, LA) 2018

SJC (JP, HK, SG) 2013

Monet (US, BR) 2017

Junior (Rio, Santos) 2017

Network Tannat (BR, UY, AR) 2017 Network sea cable investments Indigo (SG, ID, AU) 2019

Edge points of presence 100+ Edge nodes 7500+

Cloud OnBoard

6 Big Data and Machine 7 Learning in the Cloud 8

9 GCP Fundamentals: Core Infrastructure V4.0 10

14 15

16 Cloud OnBoard

Agenda Agenda 1

6 Google Cloud Big Data Platform 7

8 Google Cloud Machine Learning Platform 9

16 Cloud OnBoard

Transform

1 Agenda Scalable Simple UI Partners Managed 2 No Code Serverless

5 Salesforce, Marketo, Facebook, Twitter, Dataprep Dataflow 6 Visualize CRM data etc... Extract Scalable Easy to 7 visualize Managed Load Firebase Export Serverless 8

9 DataStudio

10 GA360 Export BI/Analytics 11 BigQuery Tools

12 Automated More Efficient, accurate than 13 rule based Google BQ-Data 14 Transfer Service Machine Learning

15 DataLab

16 Cloud OnBoard

1 Google Cloud’s big data services are fully managed and scalable 2

8 Cloud 9 Cloud Cloud BigQuery Cloud Dataproc Dataflow Pub/Sub Datalab 10 Managed Stream and Scalable and Interactive 11 Analytics Hadoop batch database; flexible data 12 MapReduce, processing; stream data enterprise exploration 13 Spark, Pig, unified and at 100,000 messaging

14 and Hive simplified rows per service pipelines second 15

16 Proprietary + Confidential Cloud OnBoard

1 Cloud Dataproc is managed Hadoop 2

3 ● Fast, easy, managed way to run 5 Hadoop and Spark/Hive/Pig on 6 Google Cloud Platform

7 ● Create clusters in 90 seconds or 8 less

9 on average.

10 ● Scale clusters up and down even

11 when jobs are running.

16 Cloud OnBoard

1 Why use Cloud Dataproc? 2

3 ● Easily migrate on-premises Hadoop 5 jobs to the cloud. 6 ● Quickly analyze data (like log 7 data) stored in Cloud Storage; 8 create a cluster in 90 seconds or

9 less on average, and then delete it immediately. 10

11 ● Use Spark/Spark SQL to quickly perform data mining and analysis. 12

13 ● Use Spark Machine Learning Libraries (MLlib) to run 14 classification algorithms. 15

16 Cloud OnBoard

1 Cloud Dataproc under the hood 2

5 Dataproc Cluster

6 Dataproc Agent

7 Compute engine nodes

Cloud Dataproc API 8 Dataproc Image OSS

9 Clusters Apache Spark HDFS Cluster bucket Persistent Disk 10 Jobs Cloud Storage Apache Hadoop Clients11 Operations Apache Hive 12 (many others) 13

14 Raw Clients 15

16 Cloud OnBoard

1 Cloud Dataflow offers managed data 2 pipelines 3 ● Processes data using Compute 5 Engine instances. 6 ○ Clusters are sized for you 7 ○ Automated scaling, no instance 8 provisioning required

9 ● Write code once and get batch and 10 streaming.

11 ● Transform-based programming model. 12

16 Cloud OnBoard

1 Dataflow pipelines flow data from a source through transforms

3 Source

5 BigQuery 6

9 Transforms

14 Sink 15

16 Cloud Storage Cloud OnBoard

1 Why use Cloud Dataflow? 2

3 ● ETL (extract/transform/load) 5 pipelines to move, filter, enrich, 6 shape data

7 ● Data analysis: batch computation 8 or continuous computation using

9 streaming

10 ● Orchestration: create pipelines

11 that coordinate services, including external services 12

13 ● Integrates with GCP services like Cloud Storage, Cloud Pub/Sub, 14 BigQuery, and Bigtable 15 ○ Open source Java and Python SDKs 16 Cloud OnBoard

1 BigQuery is a fully managed data 2 warehouse 3

5 ● Provides near real-time interactive analysis of massive 6 datasets (hundreds of TBs) 7 ● Query using SQL syntax (SQL 2011) 8

9 ● No cluster maintenance is required. 10

16 Cloud OnBoard

1 BigQuery runs on Google’s 2 high-performance infrastructure 3

5 ● Compute and storage are separated with a terabit network in between 6 ● You only pay for storage and 7 processing used

8 ● Automatic discount for long-term 9 data storage

16 Cloud OnBoard

1 Cloud Pub/Sub is scalable, reliable 2 messaging 3

5 ● Supports many-to-many asynchronous 6 messaging

7 ○ Application components make

8 push/pull subscriptions to topics

9 ● Includes support for offline

10 consumers

11 ● Based on proven Google

12 technologies

13 ● Integrates with Cloud Dataflow for

14 data processing pipelines

16 Cloud OnBoard

1 Cloud Pub/Sub is scalable, reliable 2 messaging 3

5 ● Supports many-to-many asynchronous 6 messaging

7 ○ Application components make

8 push/pull subscriptions to topics

9 ● Includes support for offline

10 consumers

11 ● Based on proven Google

12 technologies

13 ● Integrates with Cloud Dataflow for

14 data processing pipelines

16 Cloud OnBoard

1 Why use Cloud Pub/Sub? 2

5 ● Building block for data ingestion 6 in Dataflow, Internet of Things

7 (IoT), Marketing Analytics

8 ● Foundation for Dataflow streaming

9 ● Push notifications for cloud-based 10 applications

11 ● Connect applications across Google 12 Cloud Platform (push/pull between

13 Compute Engine and App Engine)

16 Cloud OnBoard

1 Cloud Datalab offers interactive 2 data exploration 3

5 ● Interactive tool for large-scale data exploration, transformation, 6 analysis, and visualization 7 ● Integrated, open source 8

9 ○ Runs on App Engine

10 ○ Built on Jupyter (formerly IPython)

16 Cloud OnBoard

1 Why use Cloud Datalab? 2

5 ● Create and manage code, documentation, results, and 6 visualizations in intuitive 7 notebook format.

8 ○ Use Google Charts or matplotlib 9 for easy visualizations.

10 ● Analyze data in BigQuery, Compute

11 Engine, and Cloud Storage using Python, SQL, and JavaScript. 12

13 ● Easily deploy models to BigQuery.

16 Cloud OnBoard

Agenda Agenda 1

6 Google Cloud Big Data Platform 7

8 Google Cloud Machine Learning Platform 9

16 Cloud OnBoard

1 Machine Learning APIs enable apps that see, hear, and 2

3 understand

16 Cloud OnBoard

1 Open source tool to build and run neural network

2 models

3 ● Wide platform support: CPU or GPU; mobile, server, or

5 cloud

6 Fully managed machine learning service

7 ● Familiar notebook-based developer experience

8 ● Optimized for Google infrastructure; integrates with BigQuery 9 Cloud ML and Cloud Storage 10 Pre-trained machine learning models built by Google 11 ● Speech: Stream results in real time, detects 80 12 languages

13 ● Vision: Identify objects, landmarks, text, and content 14 Machine Learning APIs ● Translate: Language translation including detection 15 ● Natural language: Structure, meaning of text 16 Cloud OnBoard

31 ways...

2 Build your own models: Train our implemented models: Call our trained models as APIs: 3 your data + your models your data + our models our data + our models 5

7 AutoML

Image 9 RecSys ... Classifier Cloud Cloud Cloud TensorFlow Cloud ML Engine 10 Vision API Speech API Jobs API

11 Text ... 12 Classifier

Named 14 Entity Recog. Cloud Cloud Natural Cloud Video 15 Spark ML Cloud Dataproc Translation API Language API Intelligence API 16 Cloud OnBoard

1 Why use the Cloud Machine Learning platform? 2

3 For structured data For unstructured data 5

7 Classification and regression Image and video analytics 8

10 Recommendation Text analytics 11

14 Anomaly detection

16 Cloud OnBoard

1 Cloud Vision API 2

5 ● Analyze images with a simple REST 6 API

7 ○ Logo detection, label 8 detection, etc 9

10 ● With the Cloud Vision API, you 11 can:

12 ○ Gain insight from images 13 ○ Detect inappropriate content 14 ○ Analyze sentiment 15 ○ Extract text 16 Cloud OnBoard

1 Cloud Speech API 2

5 ● Recognizes over 80 languages and 6 variants

7 ● Can return text in real time 8

9 ● Highly accurate, even in noisy 10 environments

11 ● Access from any device 12

13 ● Powered by Google’s machine 14 learning

16 Cloud OnBoard

1 Cloud Natural Language API 2

5 ● Uses machine learning models to

6 reveal structure and meaning of

7 text.

8 ● 9 Extract information about items mentioned in text documents, news 10 articles, and blog posts. 11

12 ● Analyze text uploaded in request or 13 integrate with Cloud Storage. 14

16 Cloud OnBoard

1 Cloud Translation API 2

5 ● Translate arbitrary strings 6 between thousands of language 7 pairs

8 ● Programmatically detect a 9 document’s language 10

11 ● Support for dozens of languages

16 Cloud OnBoard

1 Cloud Video Intelligence APIBeta 2

5 ● Annotate the contents of 6 videos

7 ● Detect scene changes 8

9 ● Flag inappropriate content

10 ● Support for a variety of video 11 formats 12

16 Cloud OnBoard

See you next time