Deep Learning Basics Lecture 1: Feedforward Princeton University COS 495 Instructor: Yingyu Liang Motivation I: representation learning Machine learning 1-2-3 • Collect data and extract features • Build model: choose hypothesis class 퓗 and loss function 푙 • Optimization: minimize the empirical loss Features 푥 Color Histogram Extract build 푇 features hypothesis 푦 = 푤 휙 푥 Red Green Blue Features: part of the model Nonlinear model build 푇 hypothesis 푦 = 푤 휙 푥 Linear model Example: Polynomial kernel SVM 푥1 푦 = sign(푤푇휙(푥) + 푏) 푥2 Fixed 휙 푥 Motivation: representation learning • Why don’t we also learn 휙 푥 ? Learn 휙 푥 Learn 푤 휙 푥 푦 = 푤푇휙 푥 푥 Feedforward networks • View each dimension of 휙 푥 as something to be learned … 푦 = 푤푇휙 푥 … 푥 휙 푥 Feedforward networks 푇 • Linear functions 휙푖 푥 = 휃푖 푥 don’t work: need some nonlinearity … 푦 = 푤푇휙 푥 … 푥 휙 푥 Feedforward networks 푇 • Typically, set 휙푖 푥 = 푟(휃푖 푥) where 푟(⋅) is some nonlinear function … 푦 = 푤푇휙 푥 … 푥 휙 푥 Feedforward deep networks • What if we go deeper? … … … … 푦 … … 푥 ℎ1 ℎ2 ℎ퐿 Figure from Deep learning, by Goodfellow, Bengio, Courville. Dark boxes are things to be learned. Motivation II: neurons Motivation: neurons Figure from Wikipedia Motivation: abstract neuron model • Neuron activated when the correlation between the input and a pattern 휃 exceeds some threshold 푏 푥1 • 푦 = threshold(휃푇푥 − 푏) 푥2 or 푦 = 푟(휃푇푥 − 푏) • 푟(⋅) called activation function 푦 푥푑 Motivation: artificial neural networks Motivation: artificial neural networks • Put into layers: feedforward deep networks … … … … 푦 … … 푥 ℎ1 ℎ2 ℎ퐿 Components in Feedforward networks Components • Representations: • Input • Hidden variables • Layers/weights: • Hidden layers • Output layer Components First layer Output layer … … … … 푦 … … Input 푥 Hidden variables ℎ1 ℎ2 ℎ퐿 Input • Represented as a vector • Sometimes require some preprocessing, e.g., Expand • Subtract mean • Normalize to [-1,1] Output layers Output layer • Regression: 푦 = 푤푇ℎ + 푏 • Linear units: no nonlinearity 푦 ℎ Output layers Output layer • Multi-dimensional regression: 푦 = 푊푇ℎ + 푏 • Linear units: no nonlinearity 푦 ℎ Output layers Output layer • Binary classification: 푦 = 휎(푤푇ℎ + 푏) • Corresponds to using logistic regression on ℎ 푦 ℎ Output layers Output layer • Multi-class classification: • 푦 = softmax 푧 where 푧 = 푊푇ℎ + 푏 • Corresponds to using multi-class logistic regression on ℎ 푧 푦 ℎ Hidden layers • Neuron take weighted linear combination of the previous … layer • So can think of outputting one value for the next layer … ℎ푖 ℎ푖+1 Hidden layers • 푦 = 푟(푤푇푥 + 푏) • Typical activation function 푟 푟(⋅) • Threshold t 푧 = 핀[푧 ≥ 0] • Sigmoid 휎 푧 = 1/ 1 + exp(−푧) 푥 푦 • Tanh tanh 푧 = 2휎 2푧 − 1 Hidden layers • Problem: saturation 푟(⋅) 푥 푦 Too small gradient Figure borrowed from Pattern Recognition and Machine Learning, Bishop Hidden layers • Activation function ReLU (rectified linear unit) • ReLU 푧 = max{푧, 0} Figure from Deep learning, by Goodfellow, Bengio, Courville. Hidden layers • Activation function ReLU (rectified linear unit) • ReLU 푧 = max{푧, 0} Gradient 1 Gradient 0 Hidden layers • Generalizations of ReLU gReLU 푧 = max 푧, 0 + 훼 min{푧, 0} • Leaky-ReLU 푧 = max{푧, 0} + 0.01 min{푧, 0} • Parametric-ReLU 푧 : 훼 learnable gReLU 푧 푧.