Linear Classifiers: Overfitting and Regularization CSE 446: Machine Learning Emily Fox University of Washington January 25, 2017

1/25/2017

Linear classifiers: Overfitting and regularization CSE 446: Machine Learning Emily Fox University of Washington January 25, 2017

Logistic regression recap

1 1/25/2017

Thus far, we focused on decision boundaries

Score(xi) = w0 h0(xi) + w1 h1(xi) + … + wD hD(xi) T = w h(xi)

Score(x) < 0 #awful … Relate Score(xi) to ⌃ 4 P(y=+1|x,ı )? 3

1 Score(x) > 0 0 0 1 2 3 4 … #awesome

Compare and contrast regression models

• Linear regression with Gaussian errors 2σ T 2 yi = w h(xi) + εi εi ~ N(0,σ )  p(y|x,w) = N(y; wTh(x), σ2) q w T h(x) y

• Logistic regression

2 1/25/2017

Understanding the logistic regression model

P(y=+1|xi,w) = sigmoid(Score(xi)) = 1 . 1 + e- wT h(x)

Score(xi) P(y=+1|xi,w)

Predict most likely class ⌃ P(y|x)= estimate of class probabilities

Sentence ⌃ from If P(y=+1| x) > 0.5: review ĳ = Else: Input: x ĳ =

⌃ Estimating P( y| x ) improves interpretability: - Predict ĳ = +1 and tell me how sure you are

3 1/25/2017

Gradient descent algorithm, cont’d

Step 5: Gradient over all data points

Sum over Feature data points value Difference between truth and prediction

P(y=+1|xi,w) Ş P(y=+1|xi,w) Ş

yi=+1

yi=-1

4 1/25/2017

Summary of gradient ascent for logistic regression

init w(1)=0 (or randomly, or smartly), t=1 while || ℓΔ (w(t))|| > ε for j=0,…,D partial[j] =

(t+1) (t) wj  wj + η partial[j] t  t + 1

Overfitting in classification

5 1/25/2017

Learned decision boundary

Coefficient Feature Value learned

h0(x) 1 0.23

h1(x) x[1] 1.12

h2(x) x[2] -1.07

Note: we are not including cross Quadratic features (in 2d) terms for simplicity

Coefficient Feature Value learned

h0(x) 1 1.68

h1(x) x[1] 1.39

h2(x) x[2] -0.59 2 h3(x) (x[1]) -0.17 2 h4(x) (x[2]) -0.96

6 1/25/2017

Note: we are not including cross Degree 6 features (in 2d) terms for simplicity

Coefficient Feature Value learned

h0(x) 1 21.6 Coefficient values h1(x) x[1] 5.3

h2(x) x[2] -42.7 getting large 2 h3(x) (x[1]) -15.9 Score(x) < 0 2 h4(x) (x[2]) -48.6 3 h5(x) (x[1]) -11.0 3 h6(x) (x[2]) 67.0 4 h7(x) (x[1]) 1.5 4 h8(x) (x[2]) 48.0 5 h9(x) (x[1]) 4.4 5 h10(x) (x[2]) -14.2 6 h11(x) (x[1]) 0.8 6 h12(x) (x[2]) -8.6

Note: we are not including cross Degree 20 features (in 2d) terms for simplicity

Coefficien Featur Value t e learned

h0(x) 1 8.7 h1(x) x[1] 5.1 Often, overfitting associated with h2(x) x[2] 78.7 … … … very large estimated coefficients ı 6 h11(x) (x[1]) -7.5 6 h12(x) (x[2]) 3803 7 h13(x) (x[1]) 21.1 7 h14(x) (x[2]) -2406 … … … 19 -6 h37(x) (x[1]) -2*10 19 h38(x) (x[2]) -0.15 20 -8 h39(x) (x[1]) -2*10 20 h40(x) (x[2]) 0.03 14 ©2017 Emily Fox CSE 446: Machine Learning

7 1/25/2017

Overfitting in classification

Error= Overfitting if there exists w*: • training_error(w*) > training_error(ı ) • true_error(w*) < true_error(ı ) Classification Classification Error

Model complexity

Overfitting in classifiers  Overconfident predictions

8 1/25/2017

Logistic regression model

wTh(x ) -∞ i +∞ 0.0

0.5 0.0 1.0 T P(y=+1|xi,w) = sigmoid(w h(xi))

The subtle (negative) consequence of overfitting in logistic regression

Overfitting  Large coefficient values

T ı h(xi) is very positive (or very negative)  T sigmoid(ı h(xi)) goes to 1 (or to 0)

Model becomes extremely overconfident of predictions

9 1/25/2017

Effect of coefficients on logistic regression model

Input x: #awesome=2, #awful=1

w0 0 w0 0 w0 0

w#awesome +1 w#awesome +2 w#awesome +6

w#awful -1 w#awful -2 w#awful -6

#awesome - #awful #awesome - #awful #awesome - #awful

Learned probabilities

Coefficient Feature Value learned

h0(x) 1 0.23

h1(x) x[1] 1.12

h2(x) x[2] -1.07

10 1/25/2017

Quadratic features: Learned probabilities

Coefficien Feature Value t learned

h0(x) 1 1.68

h1(x) x[1] 1.39

h2(x) x[2] -0.58 2 h3(x) (x[1]) -0.17 2 h4(x) (x[2]) -0.96

Overfitting  Overconfident predictions

Degree 6: Learned probabilities Degree 20: Learned probabilities Tiny uncertainty regions  Overfitting & overconfident about it!!! We are sure we are right, when we are surely wrong! 

11 1/25/2017

Overfitting in logistic regression: Another perspective

Linearly-separable data

Data are linearly separable if: • There exist coefficients ı such that: - For all positive training data

- For all negative training data

Note 1: If you are using D features, linear separability happens in a D-dimensional space Note 2: If you have enough features, data are (almost) always linearly separable training_error(ı ) = 0

12 1/25/2017

Effect of linear separability on coefficients

Score(x) < 0 Data are linearly separable with … ı 1=1.0 and ı 2=-1.5 awful 4 =# 3 [2] Data also linearly separable with X 2 ı 1=10 and ı 2=-15 1 Score(x) > 0 0 Data also linearly separable with 0 1 2 3 4 … ı =109 and ı =-1.5x109 x[1]=#awesome 1 2

MaximumEffect likelihoodof linear estimation separability (MLE) prefers most certain model  Coefficientson weights go to infinity for linearly-separable data!!!

P(y=+1|xi, ı ) Data is linearly separable with … ı 1=1.0 and ı 2=-1.5 awful 4 =# 3 x[1]= 2 [2] Data also linearly separable with X x[2]= 1 2 ı 1=10 and ı 2=-15 1 0 Data also linearly separable with 0 1 2 3 4 … ı =109 and ı =-1.5x109 x[1]=#awesome 1 2

13 1/25/2017

Overfitting in logistic regression is “twice as bad”

Learning tries If data are to find decision linearly boundary that separable separates data

Overly Coefficients complex go to boundary infinity!

9 ı 1=10 9 ı 2=-1.5x10

Penalizing large coefficients to mitigate overfitting

14 1/25/2017

Desired total cost format

Want to balance: i. How well function fits data ii. Magnitude of coefficients

want to balance Total quality = measure of fit - measure of magnitude of coefficients

Consider resulting objective

Select ı to maximize: 2 ℓ(w) - λ||w||2 tuning parameter = balance of fit and magnitude

Pick λ using: L2 regularized • Validation set (for large datasets) logistic regression • Cross-validation (for smaller datasets) (as in ridge/lasso regression)

15 1/25/2017

Degree 20 features, effect of regularization penalty λ

Regularizatio λ = 0 λ = 0.00001 λ = 0.001 λ = 1 λ = 10 n

Range of -3170 to 3803 -8.04 to 12.14 -0.70 to 1.25 -0.13 to 0.57 -0.05 to 0.22 coefficients

Decision boundary

Coefficient path j ı coefficients

16 1/25/2017

Degree 20 features: regularization reduces “overconfidence”

Regularization λ = 0 λ = 0.00001 λ = 0.001 λ = 1

Range of -3170 to 3803 -8.04 to 12.14 -0.70 to 1.25 -0.13 to 0.57 coefficients

Learned probabilities

Finding best L2 regularized linear classifier with gradient ascent

17 1/25/2017

Understanding contribution of L2 regularization

Term from L2 penalty

- 2 λ wj Impact on wj

wj > 0

wj < 0

Summary of gradient ascent for logistic regression + L2 regularization

init w(1)=0 (or randomly, or smartly), t=1 while not converged: for j=0,…,D partial[j] =

(t+1) (t) (t) wj  wj + η (partial[j] – 2λ wj ) t  t + 1

18 1/25/2017

Summary of overfitting in logistic regression

What you can do now…

• Identify when overfitting is happening • Relate large learned coefficients to overfitting • Describe the impact of overfitting on decision boundaries and predicted probabilities of linear classifiers • Motivate the form of L2 regularized logistic regression quality metric • Describe what happens to estimated coefficients as tuning parameter λ is varied • Interpret coefficient path plot • Estimate L2 regularized logistic regression coefficients using gradient ascent • Describe the use of L1 regularization to obtain sparse logistic regression solutions

19 1/25/2017

Linear classifiers: Scaling up learning via SGD CSE 446: Machine Learning Emily Fox University of Washington January 25, 2017

Stochastic gradient descent: Learning, one data point at a time

20 1/25/2017

Why gradient ascent is slow…

(t) (t+1) (t+2) w Update w Update w coefficients coefficients

Every update Δ

ℓΔ (w(t)) ℓ(w(t+1)) requires a full pass Compute over data Data Pass gradient over data

More formally: How expensive is gradient ascent?

Sum over data points

Contribution of data point xi,yi to gradient

21 1/25/2017

Every step requires touching every data point!!!

Sum over data points

Time to compute Total time to compute # of data points (N) contribution of xi,yi 1 step of gradient ascent 1 millisecond 1000 1 second 1000 1 millisecond 10 million 1 millisecond 10 billion

Stochastic gradient ascent

(t) (t+1) (t+2) (t+3) (t+4) w Update w Update w Update w Update w coefficients coefficients coefficients coefficients

Many updates for each pass Compute over data Data Use only gradient small subsets of data

22 1/25/2017

Example: Instead of all data points for gradient, use 1 data point only???

Sum over data points Gradient ascent

Each time, pick different data point i Stochastic gradient Ş ascent

Stochastic gradient ascent for logistic regression

init w(1)=0, t=1 Each time, pick Sum overdifferent data point i untilforconverged i=1,…,N data points for j=0,…,D partial[j] =

(t+1) (t) wj  wj + η partial[j] t  t + 1

23 1/25/2017

Stochastic gradient for L2-regularized objective

Total derivative = - 2 λ wj partial[j]

What about regularization term? Each time, pick different data point i

Stochastic Total gradient Ş - 2 λ wj partial[j] ascent derivative N Each data point contributes 1/N to regularization

Comparing computational time per step

Gradient ascent Stochastic gradient ascent Ş

Total time for Total time for Time to compute # of data points (N) 1 step of 1 step of contribution of x ,y i i gradient stochastic gradient 1 millisecond 1000 1 second 1 second 1000 16.7 minutes 1 millisecond 10 million 2.8 hours 1 millisecond 10 billion 115.7 days

24 1/25/2017

Which one is better??? Depends…

Total time to convergence for large data Time per Sensitivity to Algorithm In theory In practice iteration parameters Slow for Often Gradient Slower Moderate large data slower Stochastic Always Often Faster Very high gradient fast faster

Summary of stochastic gradient

Tiny change to gradient ascent

Much better scalability

Huge impact in real-world

Very tricky to get right in practice

25 1/25/2017

Why would stochastic gradient ever work???

Gradient is direction of steepest ascent

Gradient is “best” direction, but any direction that goes “up” would be useful

26 1/25/2017

In ML, steepest direction is sum of “little directions” from each data point

Sum over data points

Stochastic gradient: Pick a data point and move in direction

Most of the time, total likelihood will increase

27 1/25/2017

Stochastic gradient ascent: Most iterations increase likelihood, but sometimes decrease it  On average, make progress

until converged for i=1,…,N for j=0,…,D

(t+1) (t) wj  wj + η t  t + 1

Convergence path

28 1/25/2017

Convergence paths

Gradient

Stochastic gradient

Stochastic gradient convergence is “noisy” Stochastic gradient Stochastic gradient achieves higher makes “noisy” progress likelihood sooner, but it’s noisier

Gradient usually increases

Better likelihood smoothly Avg. log likelihood log Avg.

29 1/25/2017

Eventually, gradient catches up

Note: should only trust “average” quality of stochastic gradient

Stochastic gradient Gradient Better Avg. log likelihood log Avg.

The last coefficients may be really good or really bad!! 

Stochastic gradient will eventually oscillate around a solution Minimize noise: w(1005) was good don’t return last learned coefficients

How do we minimize risk of Output average: picking bad coefficients ı = 1 w(t) T w(1000) was bad

30 1/25/2017

Summary of why stochastic gradient works

Gradient finds direction of steeps ascent

Gradient is sum of contributions from each data point

Stochastic gradient uses direction from 1 data point

On average increases likelihood, sometimes decreases

Stochastic gradient has “noisy” convergence

Online learning: Fitting models from streaming data

31 1/25/2017

Batch vs online learning

Batch learning Online learning • All data is available at • Data arrives (streams in) over time start of training time - Must train model as data arrives! time t=1 t=2 t=3 t=4

ML Data Data Data Data Data algorithm

ı ML (1)(2)(3)(4) algorithm ı

Online learning example: Ad targeting

ĳ = Suggested ads Website …...... …...... …...... …...... Ad1 …...... …...... User clicked …...... …...... Ad2 …...... …...... on Ad2 …...... …......  …...... …...... Ad3 …...... …...... yt=Ad2

32 1/25/2017

Online learning problem • Data arrives over each time step t:

- Observe input xt • Info of user, text of webpage

- Make a prediction ĳt • Which ad to show

- Observe true output yt • Which ad user clicked on

Need ML algorithm to update coefficients each time step!

Stochastic gradient ascent can be used for online learning!!! • init w(1)=0, t=1 • Each time step t:

- Observe input xt

- Make a prediction ĳt

- Observe true output yt - Update coefficients: for j=0,…,D

(t+1) (t) wj  wj + η

33 1/25/2017

Summary of online learning

Data arrives over time

Must make a prediction every time new data point arrives

Observe true class after prediction made

Want to update parameters immediately

Summary of stochastic gradient descent

34 1/25/2017

What you can do now…

• Significantly speedup learning algorithm using stochastic gradient • Describe intuition behind why stochastic gradient works • Apply stochastic gradient in practice • Describe online learning problems • Relate stochastic gradient to online learning

Generalized linear models

35 1/25/2017

Models for data

• Linear regression with Gaussian errors 2σ T 2 yi = w h(xi) + εi εi ~ N(0,σ )  p(y|x,w) = N(y; wTh(x), σ2) qwTh(x) y

• Logistic regression

Examples of “generalized linear models”

A generalized linear model has E[y | x,w ] = g(wTh(x))

Mean Link function g Linear regression wTh(x) identity function Logistic regression P(y=+1|x,w) sigmoid function

(assuming y in {0,1} (mean is different if (need slight modification instead of {-1,1} or some y is not in {0,1}) if y is not in {0,1}) other set of values)

36 1/25/2017

Stochastic gradient descent more formally

Learning Problems as Expectations

• Minimizing loss in training data: - Given dataset: • Sampled iid from some distribution p(x) on features: - Loss function, e.g., hinge loss, logistic loss,… - We often minimize loss in training data:

• However, we should really minimize expected loss on all data:

• So, we are approximating the integral by the average on the training data

37 1/25/2017

Gradient Ascent in Terms of Expectations

• “True” objective function:

• Taking the gradient:

• “True” gradient ascent rule:

• How do we estimate expected gradient?

SGD: Stochastic Gradient Ascent (or Descent)

• “True” gradient:

• Sample based approximation:

• What if we estimate gradient with just one sample??? - Unbiased estimate of gradient - Very noisy!

- Called stochastic gradient ascent (or descent) • Among many other names - VERY useful in practice!!!