Senior Staff Software Engineer, Google Google I/O Extended Boulder 8-May-2018 #io18extended ...like space :) #spaceishard Spaceflight is unforgiving and complicated. But it’s not a miracle. It can be done #intelligenceishard ● Fundamentals of machine learning are complicated; ML is easily misapplied ● Doing “toy” things with ML is easy. ○ Doing useful things is a lot harder How to get there from here? ● Start small, but real ● Measure. Be rigorous. Make sure you’re helping ● Have a compelling UX ● Launch. Iterate. Improve. ● Never forget the user #io18extended Drive Quick Access Main Idea: Prominently show the user's documents and files they likely want to open right now Benefit 1: Save Users Time ● Quick Access gets users to their files 50% faster… and with less cognitive friction Benefit 2: Enable users to make better business decisions ● Show users relevant documents to their pending business decisions, including documents they may not be aware of. (The right information at the right time.) #io18extended Drive Quick Access: Mobile (Android and iOS) #io18extended What we’ve learned: Quick Access Feature Intelligence works ● Training and using Machine Learning models results in improved metrics beyond a simple “Most Recently Used” across all metrics Quick Access saves users time ● About 50% of opens come from QA, each one saves 50% on “finding time” Starting Point for Future Work ● Quick Access has proven out our machine learning infrastructure for Drive and provides a framework for future intelligence features. #io18extended Quick Access is a Large-Scale Project Clients Web, Android, iOS Drive API Prediction service Experiment Evaluation pipeline Quick Access ML system framework BigQuery Retrieve predictions Compute predictions Manage alternatives Compute accuracy System Components ● TensorFlow, mapreduce Activity service Other inputs ML system ML Model Metrics ● Experiment framework Deep network Bigtable backed ● Google BigQuery ● Servers and deployment ● Load balancing Data extraction Training ● APIs and protocols Flume pipeline Training data Flume pipeline ● Dashboards Collect data Build model ● Statistical evaluation #io18extended Features and Data (a.k.a. “Inputs,” “Predictors,” “Signals”) Features are the signals extracted from data to train models and to make predictions Example feature types 1. Frequency and Recency: Ranks of documents by frequency and recency of access 2. Periodicity: Time of day, time of week an activity was performed on a document Feature engineering: Create useful derived signals; e.g., histograms Post-processing: Minimal post-processing done; some scaling, etc. Feature Data Source: Activity Service, which receives and records events for documents ● E.g., when an item was created, shared, opened, edited, commented on, etc. ● Used during both model training time and model evaluation time #io18extended Model: Deep Neural Network ● Framework: TensorFlow -- Open source ML toolkit ● Training: Conventional Back-Propagation / SGD (asynchronous stochastic gradient descent) ○ Distributed, parallel mapreduce task w/ 200+ workers ● Features: Approximately 20,000 - 40,000 features in use ● Setup: Two-class classification problem ○ Positive training examples ■ Documents the user opened (and their features) ○ Negative training examples ■ Documents the user did not open ● Evaluation: Model evaluated when user visits Drive ● Model output: probability_open from a candidate set of N docs #io18extended Quality Metrics are how we optimize the model 1. Hit rate ○ Measures utility of Quick Access in getting users to their doc 2. Accuracy ○ Measures efficacy of machine learning predictions 3. Click-Through Rate (CTR) ○ Measures general engagement of users with Quick Access #io18extended Production metrics track performance of the service QPS (queries Per second) Latency (milliseconds per query) ● Slow Rollout ● Parallelization of backend calls ● Load Testing ● Increased capacity for reduced ● Platform growth overall load #io18extended Primary Metrics Time-Savings Metrics #io18extended Experiment Framework for Continuous Improvement Question: How do we improve a system in a principled way? Answer: Science (experiments) ● You have an existing system ● You have an idea for how to make it better hypothesis( ) ● How will you know if it improves the system? (Or if it makes it worse?) Approach ● Create an experiment with Experiment Framework ● Rigorously test the hypothesis ● Evaluate outcome, compare with hypothesis. Ship the improvements! Result: Model accuracy, user benefit improves; increased domain understanding #io18extended Improving Model Accuracy: One Idea ● Hypothesis: Actions on documents I own from my boss are more relevant, should be boosted higher ● Method: Introduce new features (signals) and see if we can improve model metrics through an experiment ● Feature Ideas: ○ Basic: ACTOR_WAS_BOSS = {True|False} ○ Extensed: ACTOR_CATEGORY = {Coworker|Report|Manager} ○ Generalized: Assign continuously valued weight to each user ● Run an experiment, test hypothesis, and ship the improvements! #io18extended When it all goes right... When we… ● Gather the correct signals ● Train correct models (apply ML properly) ● Measure the right thing ● Optimize for those correct metrics ● Build out the infrastructure ● Scale the system ● Create a beautiful, usable UX ● Make it all super-fast ● Methodically run experiments to constantly improve quality... The Result: Magic. An intelligence feature that delivers real value to the user and to the business. #io18extended Next Steps Get your learn on with Kaggle… ...then participate in competitions kaggle.com #io17extended ● Hands-on Data Science Education ● Lessons in ML, Data visualization, SQL, R, Deep learning, and more #io18extended ● Kaggle Competitions -- real-world practice! ● Build a model, make a submission ● See your model’s performance scored live on a leaderboard #io18extended Thank you! Next up: Ali Beatty on Apigee Edge Questions (5mins)? #io17extended.