Scalable Deep Learning with TensorFlow and Apache Spark™ Schedule ▪ Keras and Neural Network Fundamentals ▪ MLflow and Spark UDFs ▪ Hyperparameter Tuning with Hyperopt ▪ Horovod: Distributed Model Training ▪ LIME, SHAP & Model Interpretability ▪ CNNs and ImageNet ▪ Transfer Learning ▪ Object Detection ▪ Generative Adversarial Networks (GANs) Survey ▪ Pandas/Spark? ▪ Machine Learning? Deep Learning? ▪ Expectations? Deep Learning Overview Why Deep Learning? ▪ Performs well on complex datasets like images, sequences, and natural language ▪ Scales better as data size increases ▪ Theoretically can learn any shape (universal approximation theorem) Open Source Landscape Where Does Dl Fit In? Image source What Is Deep Learning? ▪ “Composing representations of data in a hierarchical manner” Image source Image source Keras ▪ High-level Python API to build neural networks ▪ Official high-level API of TensorFlow ▪ Has over 250,000 users ▪ Released by François Chollet in 2015 Why Keras? Hardware Considerations ▪ GPUs are prefered for training due to speed of computation, but not good in data transfer ▪ CPUs are generally acceptable for inference Why Dl On Databricks? Neural Network Fundamentals Layers ▪ Input layer ▪ Zero or more hidden layers ▪ Output layer Image source Regression Evaluation ▪ Measure "closeness" between label and prediction ▪ When predicting someone's weight, better to be off by 2 lbs instead of 20 lbs ▪ Evaluation metrics: ▪ Loss: (y−ŷ) ▪ Absolute loss: |y−ŷ| ▪ Squared loss: (y−ŷ)2 Evaluation Metric: MSE Evaluation Metric: MSE Backpropagation ▪ Calculate gradients to update weights Linear Regression With Keras Activation Functions ▪ Provide non-linearity in our neural networks to learn more complex relationships ▪ Sigmoid ▪ Leaky ReLU ▪ Tangent ▪ PReLU ▪ ReLU ▪ ELU Sigmoid ▪ Saturates and kills gradients ▪ Not zero-centered Image source Hyperbolic Tanget (Tanh) ▪ Zero centered! ▪ BUT, like the sigmoid, its activations saturate Image source RELU ▪ BUT, gradients can still go to zero Image source Leaky RELU ▪ For x < 0: ▪ f(x) = α∗x ▪ For x > = 0: ▪ f(x) = x These functions are not differentiable at 0, so we set the derivative to 0 or average of left and right derivative. Image source Comparison Image source Optimizers Stochastic Gradient Descent (Sgd) ▪ Choosing a proper learning rate can be difficult. Image source Stochastic Gradient Descent ▪ Easy to get stuck in local minima Image source Momentum ▪ Accelerates SGD: Like pushing a ball down a hill ▪ Take average of direction we’ve been heading (current velocity and acceleration) ▪ Limits oscillating back and forth, gets out of local minima Image source ADAM ▪ Adaptive Moment Estimation (Adam) Image source Keras Keras Lab Hyperparameter Selection Hyperparameter Selection Which dataset should we use to select hyperparameters? Train? Test? Validation Dataset ▪ Split the dataset into three! ▪ Train on the training set ▪ Select hyperparameters based on performance of the validation set ▪ Test on test set Advanced Keras & Lab MLflow & Lab Hyperopt & Lab Horovod Image source Image source Horovod ▪ Created by Alexander Sergeev of Uber, open-sourced in 2017 ▪ Simplifies distributed neural network training ▪ Supports TensorFlow, Keras, PyTorch, and Apache MXNet Classical Parameter Server Image source All-Reduce # Only one line of code change! optimizer = hvd.DistributedOptimizer(optimizer) Image source Horovod Demo Model Interpretability LIME Image source SHAP ▪ Shapley Values Image source SHAP Image source Convolutional Neural Networks Convolutions ▪ Focus on Local Connectivity (fewer parameters to learn) ▪ Filter/kernel slides across input image (often 3x3) Image Kernels Visualization CS 231 Convolutional Networks Imagenet Challenge ▪ Classify images in one of 1000 categories ▪ 2012 Deep Learning breakthrough with AlexNet: 16% top-5 test error rate (next closest was 25%) VGG16 (2014) ▪ One of the most widely used architectures for its simplicity. Image source Max Vs Avg. Pooling Image source Inception Image source Residual Connection Image source What Do CNNS Learn? ▪ Breaking Convnets CNN Demo Transfer Learning Transfer Learning ▪ IDEA: Intermediate representations learned for one task may be useful for other related tasks Image source When To Use Transfer Learning? Reference: Andrej Karpathy’s Transfer Learning Transfer Learning Ok, so how do I find the optimal neural network architecture? Neural Architecture Search with Reinforcement Learning Generative Adversarial Networks Generative Adversarial Networks (GANS) ▪ Estimates generative models ▪ Simultaneously trains two models ▪ G: a generative model captures the data distribution ▪ D: a discriminative model predicts probability of data coming from G ▪ Used in generating art, deep fakes, up-scaling graphics, and astronomy research ▪ Paper Gans Architecture: 2 Models Image source The Algorithm ▪ G takes noise as input, outputs a counterfeit ▪ D takes counterfeits and real values as input, outputs P(counterfeit) ▪ To prevent overfitting… ▪ Alternate k steps of optimizing D and one step of optimizing G ▪ Start with k of at least 5 ▪ Use log(1 - D(G(z))) to provide stronger, non-saturated gradients Resources ▪ Horovod Meetup Talk ▪ MLflow ▪ Deep Learning with Python ▪ Stanford's CS 231 ▪ fast.ai ▪ Blog posts & webinars ▪ Databricks Runtime for ML Pyception Thank you! .