Deep Learning Software Security and Fairness of Deep Learning SP18 Today
● HW1 is out, due Feb 15th ● Anaconda and Jupyter Notebook ● Deep Learning Software ○ Keras ○ Theano ○ Numpy Anaconda
● A package management system for Python Anaconda
● A package management system for Python Jupyter notebook
● A web application that where you can code, interact, record and plot. ● Allow for remote interaction when you are working on the cloud ● You will be using it for HW1 Deep Learning Software Deep Learning Software
Caffe(UCB) Caffe2(Facebook) Paddle (Baidu) Torch(NYU/Facebook) PyTorch(Facebook) CNTK(Microsoft) Theano(U Montreal) TensorFlow(Google) MXNet(Amazon)
Keras (High Level Wrapper) Deep Learning Software: Most Popular
Caffe(UCB) Caffe2(Facebook) Paddle (Baidu) Torch(NYU/Facebook) PyTorch(Facebook) CNTK(Microsoft) Theano(U Montreal) TensorFlow(Google) MXNet(Amazon)
Keras (High Level Wrapper) Deep Learning Software: Today
Caffe(UCB) Caffe2(Facebook) Paddle (Baidu) Torch(NYU/Facebook) PyTorch(Facebook) CNTK(Microsoft) Theano(U Montreal) TensorFlow(Google) MXNet(Amazon)
Keras (High Level Wrapper) Mobile Platform
● Tensorflow Lite: ○ Released last November Why do we use deep learning frameworks?
● Easily build big computational graphs ○ Not the case in HW1 ● Easily compute gradients in computational graphs ● GPU support (cuDNN, cuBLA...etc) ○ Not required in HW1 Keras
● A high-level deep learning framework ● Built on other deep-learning frameworks ○ Theano ○ Tensorflow ○ CNTK ● Easy and Fun! Keras: A High-level Wrapper
● Pass on a layer of instances in the constructor
● Or: simply add layers. Make sure the dimensions match. Keras: Compile and train!
Epoch: 1 epoch means going through all the training dataset once Numpy
● The fundamental package in Python for: ○ Scientific Computing ○ Data Science ● Think in terms of vectors/Matrices ○ Refrain from using for loops! ○ Similar to Matlab Numpy
● Basic vector operations ○ Sum, mean, argmax…. ● Linear Algebra operations Numpy
● Indexing, Slicing, Iterating Numpy
● Broadcasting Numpy Example
● Find the nearest value from a given value in an array Computational Graphs
● f(x,y,z) = sum(x*y + z) ● x,y,z can be scalars, vectors, matrices, tensors. Computational Graphs- Numpy Computational Graphs - Theano
● Define a variable(input) ● Define new variables using Theano operations ● Define Output ● Define functions over these variables Computational Graph-Theano
Define Variables Computational Graph-Theano
Define New Variables Computational Graph-Theano
Define functions Theano: Shared Variable and Update
● Hybrid symbolic and non-symbolic variables ● Shared between multiple functions Theano: Shared Variable and Update
● Hybrid symbolic and non-symbolic variables ● Shared between multiple functions
Score Function Gradient Descent Tensorflow
Define Variables
Define New Variables(Gradients)
Define Functions Pytorch
Define Variables
Define Functions Comparison
Framework Pros/Cons
Theano Development completed, No development in progress, Static
Tensorflow Actively developed, big community, Static
PyTorch Better for Research, relatively new, Dynamic
Keras High-level, Easy, Not flexible Resources
● Great Documentation on all of the DL software ● Deeplearning.ai ● State-of-art result for machine learning problems ○ https://github.com/RedditSota/state-of-the-art-result-for-machine-learning-problems