AI & Digital Innovation Using Artificial Intelligence to enhance the digital experience

Twitter: @RonBodkin

Slides © with many contributors. Not for redistribution.

© 2017 Google Inc. All rights reserved. Google and the are trademarks of Google Inc. All other company and product names may be trademarks of the respective companies with which they are associated. Agenda

● Artificial Intelligence at Google ● Advancing Techniques ● Industry Applications

Google Cloud Platform Confidential & Proprietary 2 is...

One branch of the field of Artificial Intelligence

A way of solving problems without explicitly codifying the solution

A way of building systems that improve themselves over time

Google Cloud Platform Confidential & Proprietary 3 Deep learning

Geoff Hinton’s work fascinated our team at Google X

We thought he was onto something with his ingenious simplifications and engineering of artificial neural networks

Google Cloud Platform Confidential & Proprietary 4 Deep Learning

Confidential & Proprietary 5 Data

Fei Fei Li sought to capture and label the same number of images a three-year-old child would witness

When combined with GPUs and deep learning, magic happened

Google Cloud Platform Confidential & Proprietary 6 Seeing with Seeing with Google Photos

[beach] [beach]

Confidential & Proprietary 7 Traditional vs Deep Machine Learning

Let a complex network build the right features

Bonuses: ● Generalization ● Transfer Learning ● Composition

Google Cloud Platform Confidential & Proprietary 8 Reading & Writing

AI now helps to reduce the strain of the email inbox by predicting responses based on your personal email history

Twelve percent of all responses are driven by AI

Confidential & Proprietary 9 10

San Francisco New York Machine learning scales better than hand-coded rules

query = ‘Giants’

user location = ‘Bay user location = user location = Area’ ? ‘New York’ ? ‘other’ ?

results about results about results about SF Giants NY Giants giants RankBrain (a deep neural network for search ranking) improved performance significantly #3 Search signal for Search ranking, out machine learning for search engines of hundreds #1 improvement to ranking quality in 2+ years Customer Acquisition for G Suite Traditional marketing model requires 45+ days to optimize

CAMPAIGN FREE TRIALS CONVERSIONS ANALYSIS

Channels

Start End Paid Seats

OPTIMIZE

45 Days

Proprietary + Confidential Customer Acquisition for G Suite 2-day marketing optimization model

CAMPAIGN FREE TRIALS CONVERSIONS ANALYSIS

Channels PREDICT

Paid Seats

COURSE CORRECT UPDATE MODEL

2 Days

Proprietary + Confidential Google is an AI company Used across products: Unique project directories

Time

Confidential & Proprietary Google Cloud Platform Copyright Google 2013-2017 15 Advancing Techniques

Google Cloud Platform Confidential & Proprietary 16 The most common ML models at Google are models that operate on structured data

Type of network # of network layers # of weights % of deployed models

MLP0 5 20M 61%

MLP1 4 5M

LSTM0 58 52M 29%

LSTM1 56 34M

CNN0 16 8M 5%

CNN1 89 100M https://cloud.google.com/blog/big-data/2017/05/an-in-depth-look-at--first-tensor-processing-unit-tpu

© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other company and product names may be trademarks of the respective companies with which they are associated. Open-source solutions

TensorFlow enables success at scale.

Google Cloud Platform Confidential & Proprietary 18 | Personalized recommendations

User Items Recommender Item 1 System Items Item 2 Item 3 Model ...

Logs Learner User Actions

Confidential & Proprietary 19 Memorization + Generalization

Wide Memorization: “Seagulls can fly.” “Pigeons can fly.”

Deep Generalization: “Animals with wings can fly.”

Wide + Generalization + memorizing exceptions: Deep “Animals with wings can fly, but penguins cannot fly.” P(Install) Wide Linear Memorization can be achieved by adding specific crossed features to learn Models frequent co-occurrence of items. Crosses add understandable non-linearity to the model.

+0.1 +0.5 +1.4

?

User Feature Impression Feature Crossed Feature installed_app=priceline impression_app=kayak AND(installed_app=priceline, impression_app=kayak) P(Install) Wide Linear Generalization can be achieved by adding Models less specific features like app category. The cross below can apply to all travel apps, not just Priceline or Kayak.

TRAVEL TRAVEL ? APP APP ?

General Crossed Feature AND(installed_category=travel, impression_category=travel) Wide Linear P(Install) Challenge: The feature space is wide and sparse. Most user-impression pairs never Models occurred in the training data. How to generalize to unseen pairs without manual feature engineering?

+0.6 +0.8 ?

?

Example: We can’t learn the weight if we've never shown Yelp to users who just installed Priceline. P(Install) Deep Neural Deep models can generalize prediction to that never or rarely Networks co-occurred via dense embeddings. Hidden layers can learn complex, but Hidden less interpretable feature interactions. Layers

Dense -1.0 +0.7 +1.8 +1.5 Embeddings +0.6 +0.5 +2.0 +1.4

User Feature Impression Feature installed_app=priceline impression_app=yelp Wide Model Weights Deep Model Embeddings Impression App (N)

[1.0, 1.0]

1.0 0.7 0.7 0.7 ? [0.3, 0.7] User 0.7 1.0 0.3 0.3 ? Installed [0.7, 0.3] App 0.7 0.3 1.0 0.3 0.3 (M) [0.1, 0.1] 0.7 0.3 0.3 1.0 0.3

0.3 0.3 ? 0.3 1.0 Deep model embeddings: Wide model nonzero weights: 22 D×(M+N) = 2 × (5+5) = 20 Wide Model Weights Deep Model Embeddings

Impression App (N)

1.0

User 1.0 Installed App 1.0 (M) 1.0

1.0 Deep model embeddings: Wide model nonzero weights: 5 D×(M+N) = 2 × (5+5) = 20 Wide & Deep memorization generalization relevance diversity Wide & Deep Joint Training of wide linear models and deep neural networks Learning by backpropagating to both sides. Combining the strengths of crossed features and embeddings.

? Wide & Deep Model for Google Play Logistic Loss

ReLU (256)

ReLU (512)

ReLU (1024) Cross Product Concatenated Embeddings (~1200 dimensions) Transformation

Embeddings Embeddings Embeddings Embeddings

... #App Device User Installed Impression Age Demographics ... Installs Class App App Continuous Features Categorical Features Experiment Results Train Wide & Deep models in 10 lines of code

# Define wide model features and crosses. query = sparse_column_with_hash_bucket("query", num_buckets) docid = sparse_column_with_hash_bucket("docid", num_buckets) query_x_docid = crossed_column([query, docid], num_buckets) wide_cols = [query_x_docid, ...]

# Define deep model features and embeddings. query_emb = embedding_column(query, dimension=32) docid_emb = embedding_column(docid, dimension=32) deep_cols = [query_emb, docid_emb, ...]

# Define model structure and start training. query docid query_x_docid m = DNNLinearCombinedClassifier( wide_cols, deep_cols, dnn_hidden_units=[500, 200, 100]) m.fit(train_data, labels, ...) TFX: TensorFlow-Based ML platform at Google

Integrated Frontend for Job Management, Monitoring, Debugging, Data/Model/Evaluation Visualization

Shared Configuration Framework and Job Orchestration

Tuner

Data Data Data Data Model Evaluation Trainer Serving Logging Ingestion Analysis Transformation Analysis and Validation

Shared Utilities for Garbage Collection, Data Access Controls

Pipeline Storage

Confidential & Proprietary 32 Rich Tooling Industry Applications

Google Cloud Platform Confidential & Proprietary 34 AI succeeds at very complicated tasks that programmers can’t write instructions for by hand

Confidential & Proprietary Three ways to deliver AI capabilities

Infrastructure AI building Blocks AI Solutions

Machine Learning Auto ML Cloud Engine Enterprise Edition Speech Differentiation Time to Value

Confidential & Proprietary 36 Customer Service

Ocado used Machine Learning to build on existing system and improved Natural Language Processing of customer service claims

Google Cloud Platform Confidential & Proprietary 37 Process Automation

Google Cloud Improves Productivity for the Largest Car Auction in Japan

Aucnet created a system that automatically recognizes the make, model, year, and estimated value of cars, serving 30,000 car dealers. Aucnet partners save thousands of hours of time through automatic classification of uploaded images.

Google Cloud Platform Confidential & Proprietary 38 Confidential & Proprietary “ Cloud AutoML’s technology is helping us build vision models to annotate our products with Disney characters, product categories, and colors. These annotations are being integrated into our to enhance the impact on Guest experience through more relevant search results, expedited discovery, and product recommendations on shopDisney.

Mike White, CTO and SVP, Disney Consumer Products and Interactive Media

Confidential & Proprietary AI does AI

Systematic exploration of the model space, using the techniques finessed in AlphaGo, yields super-human performance in AI network design

Google Cloud Platform Confidential & Proprietary 41 Computationally scalable Machine Learning

“The ML that has been successful has been the one that scales with computation” Richard Sutton

Google Cloud Platform Confidential & Proprietary 42 Rethinking hardware, not just software

TPU V2 PODs

Tensor Processing Unit (TPU) V2 Journey to “AI First”

Unify Actuate Augment

ITERATE

Confidential & Proprietary 44 Machine learning work requires all 4

Computational Algorithms resources + tools

Training Creativity + data ingenuity Thank You

Ron Bodkin @ronbodkin https://www.linkedin.com/in/ronbodkin

Google Cloud Platform Copyright Google 2013-2017 46