Google Cloud Storage. | 1

google cloud storage. | 1

*this is a guide that was created from lecture videos and is used to help you gain an understanding of how to use google cloud storage.

Understanding Google Cloud Storage

Google Cloud Platform- App Engine, Compute Engine, Cloud Storage, Cloud Datastore, Cloud SQL, BigQuery Cloud Storage- Global regional hosting of assets and data, edge caching = reduced latency = faster access, guaranteed up-time of 99.5% backup and restore options, no cap on storage limits, enhanced security, OAuth 2.0 authentication, group-based access controls such as ACL’s

Cloud Storage is based on buckets and objects, you create a bucket that holds one or more objects. Access to the buckets are handled with the API via XML in Java or Python or JSON through Java, JavaScript, Python, go, and PHP. You can also access through the Online Google Cloud Console or the gsutil Command line

Cloud Storage is great for ﬁles.

Cloud SQL is best for data not contained in a discrete ﬁle- relational database based on MySQL that can be hosted in US, Europe or Asia with up to 100GB storage and 16GB of RAM.

Cloud SQL has automatic database replication to multiple locations and point in time backup recovery. Mysqldump, MySQL Wire Protocol, JBDC and database instances can be spun up as needed. You access the database instances through Google Cloud Console interface, Standard PHP/MySQL connections, MySQL Client through command line or the JSON API

mind movement machine. google cloud storage. | 2

Cloud Datastore is for non relational data based oﬀ NoSQL. It has automatic replication.Supports ACID (Atomicity, Consistency, Isolation and Durability) transactions for reliable processing. You can access Cloud Datastore with Google Cloud Console interface, gcd command line tool or JSON API

BigQuery is used to analyze massive amounts of data. Access to BigQuery is through the UI or REST Interface. BigQuery is accessible through a host of client libraries for Java, .NET, Python, Go, Ruby, PhP, JavaScript. A SQL style interface is available which is accessed through the client libraries or web user interface

Google App Engine is a PaaS compared to Google Compute Engine which has the infrastructure as a service (IaaS) model.

App Engine can be used to build, deploy and maintain web and mobile applications and games

App Engine provides complete server side management and automatically scales to unlimited users. Auto scaling will allow instances as needed, databases shared on demand and bandwidth expanded when necessary

Google Cloud Storage is ideal for storing assets and data as a resource to Google App Engine. You can store, retrieve, display and delete assets with Cloud Storage. You can also read and write ﬂat datasheets that will be kept in cloud storage.

App Engine has full connectivity to Cloud SQL and Cloud Datastore and they both can be queried. You can manage your data with the Cloud SQL relational database or with Cloud Datastore non relational database.

mind movement machine. google cloud storage. | 3

App Engine can access BigQuery wia SQL dialect and you can utilize the analysis prowess of BigQuery through the language speciﬁc API REST interfaces.

Google Compute Engine is an IaaS for large scale computing needs. It has the same infrastructure as Google Search, Gmail and Ads. Compute Engine provides virtual machines on demand that give fast computations, has eﬃcient scaling and you can specify how many core instances (1, 2, 4, 8, 16) Each core has up to 3.75 GB memory per core.

Diﬀerence between App Engine vs Compute Engine

App Engine gives developers the ability to control Compute Engine cores, and provide a web front end for Google Compute Engine data processing applications.

Compute Engine oﬀers direct and complete operating system management for your virtual machines

Signup for Google Cloud -> Sign in or Create a Google account -> cloud.google.com -> get started -> enter information needed and payment method -> Create project -> Project ID is a unique identiﬁer that Google provides but you can change the name.

Enabled APIs under API & Auth -> API -> Enabled: BigQuery, Debuglet Controller, Cloud SQL, Cloud Storage, Cloud Storage JSON API -> you can enable any API as needed that are provided for Google Cloud. If you need more information on getting the foundations of how to use google cloud storage, visit our article laying the foundations of Google Cloud.

Google Cloud Storage Architecture

Unlimited file storage, unlimited file size, different pricing levels and are assigned on the

mind movement machine. google cloud storage. | 4

bucket level. Google Cloud Storage uses Google’s infrastructure, with high security and globally placed datacenters and fast retrieval due to edge caching for minimal latency

Google Cloud Storage is used for storage of digital content, retrieval from any capable computer/device, secure sharing that supports access control lists and OAuth 2.0 Authentication.

You can perform data analysis of stored data with Google Cloud integration with BigQuery.

You can integrate App Engine for mobile gaming and app development.

You can also host static website content in Google Cloud Storage.

Google Cloud Storage Access from the Google Cloud Storage Manager web interface, HTTP standard methods such as get, put and post, the REST interface or gsutil, a command line interface.

Google Cloud Storage is project based and have buckets within projects, And within buckets, there are objects. Projects -> Buckets -> Objects

Google Cloud Storage buckets are the main container, they provide access level control and are project speciﬁc. You cannot share buckets from one project to another. And you cannot NEST buckets. You cannot have a bucket inside of a bucket.

Google Cloud Storage objects are the content that resides in a bucket. Objects are bucket speciﬁc and cannot be shared between buckets. An object is made up of the object and the objects metadata, which is a series of name/value pairs that describe the object. Objects are

mind movement machine. google cloud storage. | 5

veriﬁed when uploaded and they must be readable to be acceptable Once read, the objects are available. Once deleted, they are gone.

Google Cloud Storage supports browser authenticated ﬁle transfer, pause and resume uploading and downloading, and automatic pause/resume when necessary. Google Cloud Storage can be programmatically user controlled.

Installing Google Cloud SDK on a Mac- cloud.google.com/sdk -> ensure you have Python on your machine -> follow installation instructions

Google Cloud Storage: Buckets

Ensure API for Google Cloud Storage is enabled -> Google Cloud Storage -> create bucket -> bucket name must be unique to Cloud Storage namespace and conform to DNS naming conventions (3 to 63 characters long) ->upload ﬁles -> select ﬁle you want to upload to the created object -> you can also upload folders Manipulating objects in buckets

Cloud Storage: Storage Browser -> select bucket you want to utilize objects for -> edit metadata ->

-> save -> edit permissions will allow permissions for user account management -> to access object go to storage.cloud.google.com/bucket name/object name -> ‘shared publicly’ will allow you to publish through storage.googleapis.com/bucketname/objectname

To get a better understanding of Google Cloud Buckets,, go to the googlecloudplatform on github.

mind movement machine. google cloud storage. | 6

Google Cloud Storage: gsutil

Authenticate with Google Cloud SDK -> specify project with gsutil: gcloud conﬁg set project (project id) -> gsutil ls will list buckets in project -> gsutil ls -L will give you the details about how many buckets you have in a JSON format, including who can access it, the access is the ACL

To see what is inside of a bucket with gsutil: gsutil ls gs://name of bucket -> you can use gsutil ls -l to see object information

Gsutil command to create a bucket- gsutil mb gs://unique bucket name -> check in Cloud Storage to ensure bucket was created

Files are moved from local to remote folders with cp command -> gsutil cp name-of-file gs://destination-of-file gs://unique-bucket-name -> will copy and upload files to buckets

Rearrange objects from bucket to bucket -> mv gs://source-bucket source-object gs://destination bucket cloud.google.com/storage/docs/gsutil for full list of commands and documentation

Build static website with Cloud Storage (not using server side language/not database driven)

Verify you are owner of website through google search console -> setup CNAME alias that points to the domain/subdomain to a Google API Service, you can do this under your current hosting provider in the DNS Zone ﬁle -> Add Record: CNAME and ﬁll out information -> ensure you have created a new CNAME alias -> Cloud Storage: Storage Browser -> Add new bucket -> bucket name MUST be the same name of the subdomain/domain that you are

mind movement machine. google cloud storage. | 7

going to use ex mind.mindmovementmachine.com -> Create bucket -> add website contents/files to your root -> open command line of root -> gsutil -m cp -r name-of-file–path gs://bucketname to move assets to Google Cloud Storage bucket -> ensure your files have downloaded into the appropriate Google Cloud Storage bucket -> gsutil -m acl set -r public- read gs://bucketname to make files publicly available -> specify which page is default (which should be homepage for example) command: gsutil web set -m homepage -e error-page gs://bucket-name to set the main homepage and the error page

Use gsutil Create new bucket copy ﬁles to bucket -> move ﬁles to gallery bucket -> delete empty bucket

Google Cloud SQL

Fully managed relational database service that uses MySQL Data automatically replicated across multiple datacenters.

MySQL is widely in used and you can import databases via mysqldump or export to mysqldump format

You can create multiple database instances with 100GB per instance and 16GB of RAM per instance

Cloud SQL Access can be done with Java, Python, the command line tool or the SQL prompt in the Google Cloud Console

Cloud SQL only supports MySQL 5.5 or higher, the instance size is limited to 500GB.

MySQL Statements not supported to Cloud SQL- Load Data in file, Select into outfile/dumpfile,

mind movement machine. google cloud storage. | 8

install/uninstall plugin, create function, load ﬁle

Cloud Storage for import/export and is accessible from App Engine and Compute Engine

Managing Google Cloud SQL Instances

Cloud SQL -> Create an instance -> ﬁll out information as needed -> show advanced options for more option such as billing, backup, MySQL ﬂags and app engine -> click on newly created database instance

Importing and exporting data in Google Cloud SQL

Everything goes through Cloud Storage.

Import- Cloud Storage: storage browser -> add bucket -> upload files -> select .sql file, ensure it is supported to import into Cloud SQL -> click import -> add cloud storage path with gs:// bucket-name imported-sql-file -> go to operations to see success of importing

Export- Google Cloud SQL -> Export -> enter gs://bucket-name sql-database-exporting

Working with Google Cloud SQL Prompt- Enable Google Cloud SQL API -> access Google Cloud SQL with listed instances -> click on database instance -> Access SQL Prompt -> choose database to query

Scheduling backups on Google Cloud SQL- Google Cloud SQL -> database instance -> edit Google Cloud instance -> enable backups

mind movement machine. google cloud storage. | 9

Asynchronous could possibly result in data loss

Create database instance -> add sql ﬁle to new bucket -> query data to see ﬁrst 200 rows

If you want to work with Google Cloud SQL and WordPress, go to Geekﬂare.

Google Cloud Storage: Datastore

For non relational data, schema less – no predeﬁned columns, supports transactions, supports robust queries, fully managed.

Based on App Engine Replication Datastore (HRD). Cloud Datastore oﬀers exceptional performance. There is over 1 petabyte for storage and has 4.5 trillion transactions per month and a 99.5 uptime.

Cloud Datastore Access by development environments: Java, Python, Node.js, SDK includes development development server and datastore simulator, and the command line tool gcd

Entities are the primary Datastore object and they are similar to a relational database table. An entity can have one or more properties. A property is like a relational database column. A property can have one or more values (values are similar to relational database rows). You cannot have multiple values with a relational database.

Properties can be diﬀerent types such as integer, string etc. Each entity is a particular kind, and a kind is a category. Kinds are used for queries for sorting and ﬁltering. Entities must have a unique ID, either a string or number. Entities can be hierarchical (referred to as ancestor path)

mind movement machine. google cloud storage. | 10

An app gets results from Cloud Datastore through a series of queries. Queries can include the entity kind, a ﬁlter to search entity property values, ID’s or ancestors, and one or more sort orders for sequencing results.

Indexes help queries be more eﬃcient and quicker. Each index is a table of entities and it is ordered by properties. It can be ordered by ancestors. When entities are modiﬁed, the index is automatically updated.

Deﬁning data in Google Cloud Datastore

Google Cloud Datastore -> create entity -> create kind (kind like a table) -> create properties (like a database column) -> you can easily ﬁlter on properties that are indexed

Setting up for local datastore

Enable you have Java Version 7 or 8 -> download gcd tools from cloud.google.com/datastore/docs/downloads -> place gcd tool in root folder -> go to command line and unzip gcd tools

GCD command line

Command line -> ./gcd.sh create dataset-id-or-project-id -> ./gcd.sh start project-id

-> *indexes can be created automatically based on your data or programmatically via XML -> place XML ﬁle into appropriate folder -> ./gcd.sh updateindexes project-id

Need more tips for Google Cloud Datastore? Visit Hackernoon.

mind movement machine. google cloud storage. | 11

Google Cloud Storage: BigQuery

Analyze massive datasets interactively, data storage scales to hundreds of TB, uses SQL like queries.

BigQuery can be accessed with the web interface, command line interface: bq or programmatically via a REST or an Excel connector

BigQuery is a fast data analyzer that is based on OLAP principles. It is CSV data based and reliant on small number of non related tables

BigQuery is not a relational database. It is not a database because you can’t insert, update or delete records, there is not support for indexes and BigQuery supports joins only when one side is smaller than the other.

Ensure you enable the BigQuery API and enable billing to work with BigQuery

To learn more in depth regarding Google Cloud BigQuery a go to medium.

Loading data into BigQuery

BigQuery -> Create new dataset -> create and import: ﬁll out destination, source data, schema -> check details of schema

Querying data in BigQuery

BigQuery -> select table -> compose query OR query table -> New Query

mind movement machine. google cloud storage. | 12

Exporting data with BigQuery

BigQuery -> run query -> you can download as csv or save as table

Python runtime environment

Download Python if not installed -> install Google API client library for Python -> download your notebook or IDE such as PyCharm. Go to JetBrains to download PyCharm

Authorizing requests to Cloud Storage

Setup consent screen- credentials ->create new Client ID for OAuth -> Download JSON ﬁle of ﬁle -> paste into google cloud python folder

To get started with Google Cloud Storage visit Google or reach out in the comments.