Overfitting in Decision Trees, Boosting Slides

1/29/17

Decision Trees: Overﬁtting CSE 446: Machine Learning Emily Fox University of Washington January 30, 2017

Decision tree recap

Loan status: Root poor 22 18 4 14 Safe Risky

Credit? Income?

excellent Fair 9 0 9 4 high low 4 5 0 9 For each leaf Safe Term? node, set Term? Risky ŷ = majority value 3 years 5 years 0 4 9 0 3 years 5 years 0 2 4 3 Risky Safe

Risky Safe

1 1/29/17

Greedy decision tree learning

• Step 1: Start with an empty tree

• Step 2: Select a feature to split data Pick feature split • For each split of the tree: leading to lowest • Step 3: If nothing more to, classiﬁcation error make predictions

Stopping • Step 4: Otherwise, go to Step 2 & conditions 1 & 2 continue (recurse) on this split Recursion

Scoring a loan application

xi = (Credit = poor, Income = high, Term = 5 years) Start

excellent poor Credit?

fair

Income? Safe Term? high Low 3 years 5 years

Risky Safe Term? Risky

3 years 5 years

Risky Safe ŷi = Safe

2 1/29/17

Decision trees vs logistic regression: Example

Logistic regression

Weight Feature Value Learned

h0(x) 1 0.22

h1(x) x[1] 1.12

h2(x) x[2] -1.07

3 1/29/17

Depth 1: Split on x[1]

y values Root - + 18 13

x[1]

x[1] < -0.07 x[1] >= -0.07 13 3 4 11

Depth 2

y values Root - + 18 13

x[1]

x[1] < -0.07 x[1] >= -0.07 13 3 4 11

x[1] x[2]

x[1] < -1.66 x[1] >= -1.66 x[2] < 1.55 x[2] >= 1.55 7 0 6 3 1 11 3 0

4 1/29/17

Threshold split caveat

y values Root - + 18 13

x[1]

For threshold splits, x[1] < -0.07 x[1] >= -0.07 same feature can be 13 3 4 11 used multiple times

x[1] x[2]

x[1] < -1.66 x[1] >= -1.66 x[2] < 1.55 x[2] >= 1.55 7 0 6 3 1 11 3 0

Decision boundaries

Depth 1 Depth 2 Depth 10

5 1/29/17

Comparing decision boundaries Decision Tree Depth 1 Depth 3 Depth 10

Logistic Regression Degree 1 features Degree 2 features Degree 6 features

Predicting probabilities with decision trees

Loan status: Root Safe Risky 18 12

Credit?

excellent fair poor P(y = Safe | x) 9 2 6 9 3 1

= 3 = 0.75 3 + 1 Safe Risky Safe

6 1/29/17

Depth 1 probabilities

Y values root - + 18 13

X1 < -0.07 X1 >= -0.07 13 3 4 11

Depth 2 probabilities

Y values root - + 18 13

X1 < -0.07 X1 >= -0.07 13 3 4 11

X1 X2

X1 < -1.66 X1 >= -1.66 X2 < 1.55 X2 >= 1.55 7 0 6 3 1 11 3 0

7 1/29/17

Comparison with logistic regression

Logistic Regression Decision Trees (Degree 2) (Depth 2)

Class

Probability

Overﬁtting in decision trees

8 1/29/17

What happens when we increase depth?

Training error reduces with depth

Tree depth depth = 1 depth = 2 depth = 3 depth = 5 depth = 10

Training error 0.22 0.13 0.10 0.03 0.00

Decision boundary

Two approaches to picking simpler trees

1. Early Stopping: Stop the learning algorithm before tree becomes too complex

2. Pruning: Simplify the tree after the learning algorithm terminates

9 1/29/17

Technique 1: Early stopping

• Stopping conditions (recap): 1. All examples have the same target value 2. No more features to split on

• Early stopping conditions: 1. Limit tree depth (choose max_depth using validation set) 2. Do not consider splits that do not cause a suﬃcient decrease in classiﬁcation error 3. Do not split an intermediate node which contains too few data points

Challenge with early stopping condition 1

Hard to know exactly when to stop

Simple Complex Also, might want trees trees some branches of

tree to go deeper while others remain shallow True error

Classiﬁcation Error Training error

max_depth Tree depth

10 1/29/17

Early stopping condition 2: Pros and Cons

• Pros: - A reasonable heuristic for early stopping to avoid useless splits

• Cons: - Too short sighted: We may miss out on “good” splits may occur right after “useless” splits - Saw this with “xor” example

Two approaches to picking simpler trees

1. Early Stopping: Stop the learning algorithm before tree becomes too complex

2. Pruning: Simplify the tree after the learning algorithm terminates

Complements early stopping

11 1/29/17

Pruning: Intuition Train a complex tree, simplify later

Complex Tree

Simpler Tree

Simplify

Pruning motivation

Simple Complex tree tree True Error Simplify after tree is built

Don’t stop too early

Classiﬁcation Error Training Error

Tree depth

12 1/29/17

Scoring trees: Desired total quality format

Want to balance: i. How well tree ﬁts data ii. Complexity of tree

want to balance Total cost = measure of ﬁt + measure of complexity

Simple measure of complexity of tree

L(T) = # of leaf nodes Start

excellent poor Credit?

good fair bad Safe Risky

Safe Safe Risky

13 1/29/17

Balance simplicity & predictive power

Too complex, risk of overﬁtting

Start

excellent poor Credit? Too simple, high fair classiﬁcation error Income? Safe Term? Start high low 3 years 5 years

Risky Risky Safe Term? Risky

3 years 5 years

Risky Safe

Balancing ﬁt and complexity

Total cost C(T) = Error(T) + λ L(T)

tuning parameter If λ=0:

If λ=∞:

If λ in between:

14 1/29/17

Tree pruning algorithm

Step 1: Consider a split

Start Tree T

excellent poor Credit?

fair

Income? Safe Term? high low 3 years 5 years

Risky Safe Term? Risky

3 years 5 years Candidate for Safe Risky pruning

15 1/29/17

Step 2: Compute total cost C(T) of split

Start Tree T

excellent poor λ = 0.3 Credit? Tree Error #Leaves Total fair T 0.25 6 0.43 Income? Safe Term? high low 3 years 5 years C(T) = Error(T) + λ L(T)

Risky Safe Term? Risky

3 years 5 years Candidate for Safe Risky pruning

Step 2: “Undo” the splits on Tsmaller

Start Tree Tsmaller

excellent poor λ = 0.3 Credit? Tree Error #Leaves Total fair T 0.25 6 0.43 Income? Safe Term? Tsmaller 0.26 5 0.41 high low 3 years 5 years C(T) = Error(T) + λ L(T) Risky Safe Safe Risky

16 1/29/17

Prune if total cost is lower: C(Tsmaller) ≤ C(T)

Worse training error but

Start Tree Tsmaller lower overall cost

excellent poor λ = 0.3 Credit? Tree Error #Leaves Total fair T 0.25 6 0.43 Income? Safe Term? Tsmaller 0.26 5 0.41 high low 3 years 5 years C(T) = Error(T) + λ L(T) Risky Safe Safe Risky

Step 5: Repeat Steps 1-4 for every split

Start Decide if each split can be “pruned” excellent poor Credit?

fair

Income? Safe Term? high low 3 years 5 years

Risky Safe Safe Risky

17 1/29/17

Summary of overﬁtting in decision trees

What you can do now…

• Identify when overfitting in decision trees • Prevent overfitting with early stopping - Limit tree depth - Do not consider splits that do not reduce classification error - Do not split intermediate nodes with only few points • Prevent overfitting by pruning complex trees - Use a total cost formula that balances classification error and tree complexity - Use total cost to merge potentially complex trees into simpler ones

18 1/29/17

Boosting CSE 446: Machine Learning Emily Fox University of Washington January 30, 2017

Simple (weak) classiﬁers are good!

Income>$100K?

Yes No

Safe Risky Logistic regression Shallow Decision w. simple features decision trees stumps

Low variance. Learning is fast!

But high bias…

19 1/29/17

Finding a classiﬁer that’s just right true error

error train error Classiﬁcation

Model complexity

Weak learner Need stronger learner

Boosting question

“Can a set of weak learners be combined to create a stronger learner?” Kearns and Valiant (1988)

Yes! Schapire (1990)

Boosting

Amazing impact: simple approach widely used in industry wins most Kaggle competitions

20 1/29/17

Ensemble classiﬁer

A single classiﬁer Input: x

Income>$100K?

Yes No

Safe Risky

21 1/29/17

Income>$100K?

Yes No

Safe Risky

Ensemble methods: Each classiﬁer “votes” on prediction

xi = (Income=$120K, Credit=Bad, Savings=$50K, Market=Good)

Income>$100K? Credit history? Savings>$100K? Market conditions?

Yes No Bad Good Yes No Bad Good

Safe Risky Risky Safe Safe Risky Risky Safe

f1(xi) = +1 f2(xi) = -1 f3(xi) = -1 f4(xi) = +1

Combine? Ensemble model Learn coeﬃcients

F(xi) = sign(w1 f1(xi) + w2 f2(xi) + w3 f3(xi) + w4 f4(xi))

22 1/29/17

Prediction with ensemble

Income>$100K? Credit history? Savings>$100K? Market conditions?

Yes No Bad Good Yes No Bad Good

Safe Risky Risky Safe Safe Risky Risky Safe

f1(xi) = +1 f2(xi) = -1 f3(xi) = -1 f4(xi) = +1

Combine? Ensemble w1 2 model Learn coeﬃcients w2 1.5

w3 1.5 F(xi) = sign(w1 f1(xi) + w2 f2(xi) + w3 f3(xi) + w4 f4(xi)) w4 0.5

Ensemble classiﬁer in general • Goal: - Predict output y • Either +1 or -1 - From input x • Learn ensemble model:

- Classiﬁers: f1(x),f2(x),…,fT(x)

- Coeﬃcients: ŵ1,ŵ2,…,ŵT • Prediction: T

23 1/29/17

Boosting

Training a classiﬁer

Training Learn f(x) Predict data classiﬁer ŷ = sign(f(x))

24 1/29/17

Learning decision stump

Credit Income y A $130K Safe B $80K Risky Income? C $110K Risky A $110K Safe A $90K Safe > $100K B $120K Safe ≤ $100K 3 1 4 3 C $30K Risky C $60K Risky B $95K Safe A $60K Safe ŷ = Safe ŷ = Safe A $98K Safe

Boosting = Focus learning on “hard” points

Training Learn f(x) Predict data classiﬁer ŷ = sign(f(x))

Evaluate Boosting: focus next classiﬁer on places where f(x) does less well Learn where f(x) makes mistakes

25 1/29/17

Learning on weighted data: More weight on “hard” or more important points

• Weighted dataset:

- Each xi,yi weighted by αi

• More important point = higher weight αi

• Learning:

- Data point i counts as αi data points

• E.g., αi = 2 è count point twice

Learning a decision stump on weighted data

Increase weight α of harder/misclassiﬁed points

CreditCredit IncomeIncome y y Weight α AA $130K$130K SafeSafe 0.5 BB $80K$80K RiskyRisky 1.5 Income? CC $110K$110K RiskyRisky 1.2 AA $110K$110K SafeSafe 0.8 AA $90K$90K SafeSafe 0.6 > $100K BB $120K$120K SafeSafe 0.7 ≤ $100K 2 1.2 3 6.5 CC $30K$30K RiskyRisky 3 CC $60K$60K RiskyRisky 2 BB $95K$95K SafeSafe 0.8 AA $60K$60K SafeSafe 0.7 ŷ = Safe ŷ = Risky AA $98K$98K SafeSafe 0.9

26 1/29/17

Learning from weighted data in general

• Usually, learning from weighted data

- Data point i counts as αi data points

• E.g., gradient ascent for logistic regression:

Sum over data points

NN ((tt+1)+1) ((tt)) ((tt)) ww ww ++⌘⌘ hhαjj((x xii)) 11[[yyii == +1] +1] PP((yy=+1=+1 xxii,,ww )) jj jj i || ii=1=1 XX ⇣⇣ ⌘⌘

Weigh each point by α i

Boosting = Greedy learning ensembles from data

Training data

Learn classiﬁer Higher weight f (x) for points where 1 Predict ŷ = sign(f1(x)) f1(x) is wrong

Weighted data

Learn classiﬁer & coeﬃcient

ŵ,f2(x)

Predict ŷ = sign(ŵ1 f1(x) + ŵ2 f2(x))

27 1/29/17

AdaBoost

AdaBoost: learning ensemble [Freund & Schapire 1999]

• Start with same weight for all points: αi = 1/N

• For t = 1,…,T

- Learn ft(x) with data weights αi

- Compute coeﬃcient ŵt

- Recompute weights αi

• Final model predicts by: T

28 1/29/17

Computing coeﬃcient ŵt

AdaBoost: Computing coeﬃcient ŵt of classiﬁer ft(x)

Yes ŵt large Is ft(x) good? No ŵt small

• ft(x) is good è ft has low training error

• Measuring error in weighted data? - Just weighted # of misclassiﬁed points

29 1/29/17

Weighted classiﬁcation error

Learned classiﬁer ŷ =

Data point Weight of correct 1.20 Correct!Mistake! Weight of mistakes (Food(SushiFoodSushi waswas OKOK,great,great ,,αα=0.5=1.2) ) 0.50

Hide label

Weighted classiﬁcation error

• Total weight of mistakes:

• Total weight of all points:

• Weighted error measures fraction of weight of mistakes:

weighted_error = .

- Best possible value is 0.0

30 1/29/17

AdaBoost:

Formula for computing coeﬃcient ŵt of classiﬁer ft(x) 1 1 weighted error(f ) wˆ = ln t t 2 weighted error(f ) ✓ t ◆

weighted_error(f ) 1 weighted error(ft) t on training data weighted error(ft) ŵt Yes

Is ft(x) good? No

AdaBoost: learning ensemble

• Start with same weight for all points: αi = 1/N

• For t = 1,…,T 1 1 weighted error(ft) - Learn ft(x) with data weights αi wˆ = ln t 2 weighted error(f ) ✓ t ◆ - Compute coeﬃcient ŵt

- Recompute weights αi

• Final model predicts by: T

31 1/29/17

Recompute weights αi

AdaBoost: Updating weights αi based on where classiﬁer ft(x) makes mistakes

Yes Decrease αi

Did ft get xi right? No Increase αi

32 1/29/17

AdaBoost: Formula for updating weights αi

-ŵt αi e , if ft(xi)=yi αi ç ŵt αi e , if ft(xi)≠yi

Multiply α by Implication ft(xi)=yi ? ŵt i

Yes

Did ft get xi right? No

AdaBoost: learning ensemble

• Start with same weight for all points: αi = 1/N

• For t = 1,…,T 1 1 weighted error(ft) - Learn ft(x) with data weights αi wˆ = ln t 2 weighted error(f ) ✓ t ◆ - Compute coeﬃcient ŵt

33 1/29/17

AdaBoost: Normalizing weights αi

If xi often mistake, If xi often correct, weight αi gets very weight αi gets very large small

Can cause numerical instability after many iterations

AdaBoost: learning ensemble

1 1 weighted error(f ) • Start with same weight for wˆ = ln t t 2 weighted error(f ) all points: αi = 1/N ✓ t ◆ • For t = 1,…,T -ŵ - Learn ft(x) with data weights αi t αi e , if ft(xi)=yi - Compute coeﬃcient ŵt αi ç ŵt - Recompute weights αi αi e , if ft(xi)≠yi

34 1/29/17

AdaBoost example

t=1: Just learn a classiﬁer on original data

Original data Learned decision stump f1(x)

35 1/29/17

Updating weights αi

Increase weight αi of misclassiﬁed points

Learned decision stump f1(x) New data weights αi Boundary

t=2: Learn classiﬁer on weighted data

f1(x)

α Weighted data: using i Learned decision stump f2(x) chosen in previous iteration on weighted data

36 1/29/17

Ensemble becomes weighted sum of learned classiﬁers

ŵ1 f1(x) ŵ2 f2(x)

0.61 + 0.53 =

Decision boundary of ensemble classiﬁer after 30 iterations

training_error = 0

37 1/29/17

Boosted decision stumps

• Start same weight for all points: αi = 1/N • For t = 1,…,T

- Learn ft(x): pick decision stump with lowest weighted training error according to α i

- Compute coeﬃcient ŵt

- Recompute weights αi

- Normalize weights αi • Final model predicts by: T

38 1/29/17

Finding best next decision stump ft(x) Consider splitting on each feature:

Income>$100K? Credit history? Savings>$100K? Market conditions?

Yes No Bad Good Yes No Bad Good

Safe Risky Risky Safe Safe Risky Risky Safe

weighted_error weighted_error weighted_error weighted_error = 0.2 = 0.35 = 0.3 = 0.4

Boosted decision stumps

• Start same weight for all points: αi = 1/N • For t = 1,…,T

- Learn ft(x): pick decision stump with lowest weighted training error according to αi

- Compute coeﬃcient ŵt

- Recompute weights αi

- Normalize weights αi • Final model predicts by: T

39 1/29/17

-ŵt Updating weights α -0.69 i α αi e = i e = αi /2 , if ft(xi)=yi Income>$100K? αi ç ŵt 0.69 Yes No α αi e = i e = 2αi , if ft(xi)≠yi Safe Risky Previous New Credit Income y ŷ weight α weight α A $130K Safe Safe A $130K Safe Safe 0.5 0.5/2 = 0.25 B $80K Risky Risky B $80K Risky Risky 1.5 0.75 C $110K Risky Safe C $110K Risky Safe 1.5 2 * 1.5 = 3 A $110K Safe Safe A $110K Safe Safe 2 1 A $90K Safe Risky A $90K Safe Risky 1 2 B $120K Safe Safe B $120K Safe Safe 2.5 1.25 C $30K Risky Risky C $30K Risky Risky 3 1.5 C $60K Risky Risky C $60K Risky Risky 2 1 B $95K Safe Risky B $95K Safe Risky 0.5 1 A $60K Safe Risky A $60K Safe Risky 1 2 A $98K Safe Risky A $98K Safe Risky 0.5 1 79 ©2017 Emily Fox CSE 446: Machine Learning

Summary of boosting

40 1/29/17

Variants of boosting and related algorithms

There are hundreds of variants of boosting, most important:

Gradient boosting • Like AdaBoost, but useful beyond basic classiﬁcation

Many other approaches to learn ensembles, most important:

• Bagging: Pick random subsets of the data

- Learn a tree in each subset Random - Average predictions forests • Simpler than boosting & easier to parallelize • Typically higher error than boosting for same # of trees (# iterations T)

Impact of boosting (spoiler alert... HUGE IMPACT)

Amongst most useful ML methods ever created

Extremely useful in • Standard approach for face detection, for example computer vision

Used by most winners of • Malware classiﬁcation, credit fraud detection, ads click through rate estimation, sales forecasting, ML competitions ranking webpages for search, Higgs boson (Kaggle, KDD Cup,…) detection,…

Most deployed ML systems use • Coeﬃcients chosen manually, with boosting, with model ensembles bagging, or others

41 1/29/17

What you can do now…

• Identify notion ensemble classifiers • Formalize ensembles as the weighted combination of simpler classifiers • Outline the boosting framework – sequentially learn classifiers on weighted data • Describe the AdaBoost algorithm - Learn each classifier on weighted data - Compute coefficient of classifier - Recompute data weights - Normalize weights • Implement AdaBoost to create an ensemble of decision stumps