Senior Staff Software Engineer,

Google I/O Extended Boulder 8-May-2018

#io18extended ...like space :) #spaceishard Spaceflight is unforgiving and complicated. But it’s not a miracle. It can be done

#intelligenceishard ● Fundamentals of are complicated; ML is easily misapplied ● Doing “toy” things with ML is easy. ○ Doing useful things is a lot harder

How to get there from here? ● Start small, but real ● Measure. Be rigorous. Make sure you’re helping ● Have a compelling UX ● Launch. Iterate. Improve. ● Never forget the user

#io18extended Drive Quick Access

Main Idea: Prominently show the user's documents and they likely want to open right now

Benefit 1: Save Users Time ● Quick Access gets users to their files 50% faster… and with less cognitive friction

Benefit 2: Enable users to make better business decisions ● Show users relevant documents to their pending business decisions, including documents they may not be aware of. (The right information at the right time.)

#io18extended Drive Quick Access: Mobile (Android and iOS)

#io18extended What we’ve learned: Quick Access Feature

Intelligence works ● Training and using Machine Learning models results in improved metrics beyond a simple “Most Recently Used” across all metrics

Quick Access saves users time ● About 50% of opens come from QA, each one saves 50% on “finding time”

Starting Point for Future Work ● Quick Access has proven out our machine learning infrastructure for Drive and provides a framework for future intelligence features.

#io18extended Quick Access is a Large-Scale Project

Clients Web, Android, iOS Drive API Prediction service Experiment Evaluation pipeline Quick Access ML system framework BigQuery

Retrieve predictions Compute predictions Manage alternatives Compute accuracy

System Components

● TensorFlow, mapreduce Activity service Other inputs ML system ML Model Metrics ● Experiment framework Deep network backed ● Google BigQuery ● Servers and deployment ● Load balancing Data extraction Training ● APIs and protocols Flume pipeline Training data Flume pipeline ● Dashboards Collect data Build model ● Statistical evaluation

#io18extended Features and Data (a.k.a. “Inputs,” “Predictors,” “Signals”)

Features are the signals extracted from data to train models and to make predictions

Example feature types 1. Frequency and Recency: Ranks of documents by frequency and recency of access 2. Periodicity: Time of day, time of week an activity was performed on a document

Feature engineering: Create useful derived signals; e.g., histograms

Post-processing: Minimal post-processing done; some scaling, etc.

Feature Data Source: Activity Service, which receives and records events for documents

● E.g., when an item was created, shared, opened, edited, commented on, etc. ● Used during both model training time and model evaluation time

#io18extended Model: Deep Neural Network

● Framework: TensorFlow -- Open source ML toolkit ● Training: Conventional Back-Propagation / SGD (asynchronous stochastic gradient descent) ○ Distributed, parallel mapreduce task w/ 200+ workers ● Features: Approximately 20,000 - 40,000 features in use ● Setup: Two-class classification problem ○ Positive training examples ■ Documents the user opened (and their features) ○ Negative training examples ■ Documents the user did not open ● Evaluation: Model evaluated when user visits Drive ● Model output: probability_open from a candidate set of N docs

#io18extended Quality Metrics are how we optimize the model

1. Hit rate ○ Measures utility of Quick Access in getting users to their doc

2. Accuracy ○ Measures efficacy of machine learning predictions

3. Click-Through Rate (CTR) ○ Measures general engagement of users with Quick Access

#io18extended Production metrics track performance of the service

QPS (queries Per second) Latency (milliseconds per query)

● Slow Rollout ● Parallelization of backend calls ● Load Testing ● Increased capacity for reduced ● Platform growth overall load

#io18extended Primary Metrics Time-Savings Metrics

#io18extended Experiment Framework for Continuous Improvement

Question: How do we improve a system in a principled way? Answer: Science (experiments) ● You have an existing system ● You have an idea for how to make it better hypothesis( ) ● How will you know if it improves the system? (Or if it makes it worse?)

Approach ● Create an experiment with Experiment Framework ● Rigorously test the hypothesis ● Evaluate outcome, compare with hypothesis. Ship the improvements!

Result: Model accuracy, user benefit improves; increased domain understanding

#io18extended Improving Model Accuracy: One Idea

● Hypothesis: Actions on documents I own from my boss are more relevant, should be boosted higher

● Method: Introduce new features (signals) and see if we can improve model metrics through an experiment

● Feature Ideas: ○ Basic: ACTOR_WAS_BOSS = {True|False} ○ Extensed: ACTOR_CATEGORY = {Coworker|Report|Manager} ○ Generalized: Assign continuously valued weight to each user

● Run an experiment, test hypothesis, and ship the improvements!

#io18extended When it all goes right...

When we… ● Gather the correct signals ● Train correct models (apply ML properly) ● Measure the right thing ● Optimize for those correct metrics ● Build out the infrastructure ● Scale the system ● Create a beautiful, usable UX ● Make it all super-fast ● Methodically run experiments to constantly improve quality...

The Result: Magic. An intelligence feature that delivers real value to the user and to the business.

#io18extended Next Steps Get your learn on with Kaggle… ...then participate in competitions

kaggle.com #io17extended ● Hands-on Data Science Education ● Lessons in ML, Data visualization, SQL, R, Deep learning, and more

#io18extended ● Kaggle Competitions -- real-world practice! ● Build a model, make a submission ● See your model’s performance scored live on a leaderboard

#io18extended Thank you! Next up: Ali Beatty on Edge

Questions (5mins)?

#io17extended