The Naïve Bayes Classifier

Home , Prior probability

Machine Learning

1 Today’s lecture

• The naïve Bayes Classifier

• Learning the naïve Bayes Classifier

• Practical concerns

2 Today’s lecture

• The naïve Bayes Classifier

• Learning the naïve Bayes Classifier

• Practical concerns

3 Where are we?

We have seen Bayesian learning – Using a probabilistic criterion to select a hypothesis – Maximum a posteriori and maximum likelihood learning You should know what is the difference between them

We could also learn functions that predict probabilities of outcomes – Different from using a probabilistic criterion to learn

Maximum a posteriori (MAP) prediction as opposed to MAP learning

4 Where are we?

We have seen Bayesian learning – Using a probabilistic criterion to select a hypothesis – Maximum a posteriori and maximum likelihood learning You should know what is the difference between them

We could also learn functions that predict probabilities of outcomes – Different from using a probabilistic criterion to learn

Maximum a posteriori (MAP) prediction as opposed to MAP learning

5 MAP prediction

Using the Bayes rule for predicting � given an input �

� � = � � = � � � = � � � = � � = � = �(� = �)

Posterior probability of label being y for this input x

6 MAP prediction

Using the Bayes rule for predicting � given an input �

� � = � � = � � � = � � � = � � = � = �(� = �)

Predict the label � for the input � using

� � = � � = � � � = � argmax �(� = �)

7 MAP prediction

Using the Bayes rule for predicting � given an input �

� � = � � = � � � = � � � = � � = � = �(� = �)

Predict the label � for the input � using

argmax � � = � � = � � � = �

8 Don’t confuse with MAP learning: MAP prediction finds hypothesis by

Using the Bayes rule for predicting � given an input �

� � = � � = � � � = � � � = � � = � = �(� = �)

Predict the label � for the input � using

argmax � � = � � = � � � = �

9 MAP prediction

Predict the label � for the input � using

argmax � � = � � = � � � = �

Likelihood of observing this Prior probability of the label input x when the label is y being y

All we need are these two sets of probabilities

10 Example: Tennis again Play tennis P(Play tennis) Without any other information, what is the prior probability that I Prior Yes 0.3 should play tennis? No 0.7

Temperature Wind P(T, W |Tennis = Yes) On days that I do play tennis, what is the Hot Strong 0.15 probability that the Hot Weak 0.4 temperature is T and Cold Strong 0.1 the wind is W? Cold Weak 0.35 Likelihood Temperature Wind P(T, W |Tennis = No) On days that I don’t play Hot Strong 0.4 tennis, what is the probability that the Hot Weak 0.1 temperature is T and the Cold Strong 0.3 wind is W? Cold Weak 0.2

11 Example: Tennis again Play tennis P(Play tennis) Without any other information, what is the prior probability that I Prior Yes 0.3 should play tennis? No 0.7

12 Example: Tennis again Play tennis P(Play tennis) Without any other information, what is the prior probability that I Prior Yes 0.3 should play tennis? No 0.7

13 Example: Tennis again Input: Temperature = Hot (H) Play tennis P(Play tennis) Wind = Weak (W) Prior Yes 0.3 No 0.7 Should I play tennis?

Temperature Wind P(T, W |Tennis = Yes) Hot Strong 0.15 Hot Weak 0.4 Cold Strong 0.1 Cold Weak 0.35 Likelihood Temperature Wind P(T, W |Tennis = No) Hot Strong 0.4 Hot Weak 0.1 Cold Strong 0.3 Cold Weak 0.2

14 Example: Tennis again Input: Temperature = Hot (H) Play tennis P(Play tennis) Wind = Weak (W) Prior Yes 0.3 No 0.7 Should I play tennis?

Temperature Wind P(T, W |Tennis = Yes) argmaxy P(H, W | play?) P (play?) Hot Strong 0.15 Hot Weak 0.4 Cold Strong 0.1 Cold Weak 0.35 Likelihood Temperature Wind P(T, W |Tennis = No) Hot Strong 0.4 Hot Weak 0.1 Cold Strong 0.3 Cold Weak 0.2

15 Example: Tennis again Input: Temperature = Hot (H) Play tennis P(Play tennis) Wind = Weak (W) Prior Yes 0.3 No 0.7 Should I play tennis?

Temperature Wind P(T, W |Tennis = Yes) argmaxy P(H, W | play?) P (play?) Hot Strong 0.15 P(H, W | Yes) P(Yes) = 0.4 £ 0.3 Hot Weak 0.4 = 0.12 Cold Strong 0.1 Cold Weak 0.35 P(H, W | No) P(No) = 0.1 £ 0.7 Likelihood = 0.07 Temperature Wind P(T, W |Tennis = No) Hot Strong 0.4 Hot Weak 0.1 Cold Strong 0.3 Cold Weak 0.2

16 Example: Tennis again Input: Temperature = Hot (H) Play tennis P(Play tennis) Wind = Weak (W) Prior Yes 0.3 No 0.7 Should I play tennis?

Hot Strong 0.4 MAP prediction = Yes Hot Weak 0.1 Cold Strong 0.3 Cold Weak 0.2

17 How hard is it to learn probabilistic models?

O T H W Play? Outlook: S(unny), 1 S H H W - O(vercast), 2 S H H S - 3 O H H W + R(ainy) 4 R M H W + Temperature: H(ot), 5 R C N W + M(edium), 6 R C N S - 7 O C N S + C(ool) 8 S M H W - Humidity: H(igh), 9 S C N W + 10 R M N W + N(ormal), 11 S M N S + L(ow) 12 O M H S + 13 O H N W + Wind: S(trong), 14 R M H S - W(eak)

18 How hard is it to learn probabilistic models?

O T H W Play? Outlook: S(unny), 1 S H H W - O(vercast), 2 S H H S - 3 O H H W + R(ainy) We need to learn 4 R M H W + Temperature: H(ot), 5 R C N W + M(edium), 6 R C N S - 1.The prior �(Play? ) C(ool) 7 O C N S + 2.The likelihoods � x Play? ) 8 S M H W - Humidity: H(igh), 9 S C N W + 10 R M N W + N(ormal), 11 S M N S + L(ow) 12 O M H S + 13 O H N W + Wind: S(trong), 14 R M H S - W(eak)

19 How hard is it to learn probabilistic models?

20 How hard is it to learn probabilistic models?

21 How hard is it to learn probabilistic models?

O T H W Play? Prior P(play?) 1 S H H W - 2 S H H S - • A single number (Why only one?) 3 O H H W + 4 R M H W + Likelihood P(X | Play?) 5 R C N W + • There are 4 features 6 R C N S - 7 O C N S + • For each value of Play? (+/-), we 8 S M H W - need a value for each possible 9 S C N W + 10 R M N W + assignment: P(O, T, H, W | Play?) 11 S M N S + • (3 ⋅ 3 ⋅ 3 ⋅ 2 − 1) parameters in 12 O M H S + 13 O H N W + each case 14 R M H S - One for each assignment 3 3 3 2 Values for this feature 23 How hard is it to learn probabilistic models? In general O T H W Play? Prior P(Y) 1 S H H W - 2 S H H S - • If there are k labels, then k – 1 3 O H H W + parameters (why not k?) 4 R M H W + 5 R C N W + Likelihood P(X | Y) 6 R C N S - 7 O C N S + • If there are d features, then: 8 S M H W - • We need a value for each possible 9 S C N W + 10 R M N W + P(x1, x2, !, xd | y) for each y 11 S M N S + • k(2d – 1) parameters 12 O M H S + 13 O H N W + Need a lot of data to estimate these 14 R M H S - many numbers!

24 How hard is it to learn probabilistic models? In general O T H W Play? Prior P(Y) 1 S H H W - 2 S H H S - • If there are k labels, then k – 1 3 O H H W + parameters (why not k?) 4 R M H W + 5 R C N W + Likelihood P(X | Y) 6 R C N S - 7 O C N S + • If there are d Boolean features: 8 S M H W - • We need a value for each 9 S C N W + 10 R M N W + possible P(x1, x2, !, xd | y) for 11 S M N S + each y 12 O M H S + • d 13 O H N W + k(2 – 1) parameters 14 R M H S - Need a lot of data to estimate these many numbers! 25 How hard is it to learn probabilistic models? In general O T H W Play? Prior P(Y) 1 S H H W - 2 S H H S - • If there are k labels, then k – 1 3 O H H W + parameters (why not k?) 4 R M H W + 5 R C N W + Likelihood P(X | Y) 6 R C N S - 7 O C N S + • If there are d Boolean features: 8 S M H W - • We need a value for each 9 S C N W + 10 R M N W + possible P(x1, x2, !, xd | y) for 11 S M N S + each y 12 O M H S + • d 13 O H N W + k(2 – 1) parameters 14 R M H S - Need a lot of data to estimate these many numbers! 26 How hard is it to learn probabilistic models?

Prior P(Y) • If there are k labels, then k – 1 parameters (why not k?) High model complexity Likelihood P(X | Y) If there is very limited data, high variance in the • If there are d Boolean features: parameters • We need a value for each possible P(x1, x2, !, xd | y) for each y • k(2d – 1) parameters Need a lot of data to estimate these many numbers! 27 How hard is it to learn probabilistic models?

Prior P(Y) • If there are k labels, then k – 1 parameters (why not k?) High model complexity Likelihood P(X | Y) If there is very limited data, high variance in the • If there are d Boolean features: parameters • We need a value for each How can we deal with this? possible P(x1, x2, !, xd | y) for each y • k(2d – 1) parameters Need a lot of data to estimate these many numbers! 28 How hard is it to learn probabilistic models?

Prior P(Y) • If there are k labels, then k – 1 parameters (why not k?) High model complexity Likelihood P(X | Y) If there is very limited data, high variance in the • If there are d Boolean features: parameters • We need a value for each How can we deal with this? possible P(x , x , !, x | y) for 1 2 d Answer: Make independence each y assumptions • k(2d – 1) parameters Need a lot of data to estimate these many numbers! 29 Recall: Conditional independence

Suppose X, Y and Z are random variables

X is conditionally independent of Y given Z if the probability distribution of X is independent of the value of Y when Z is observed

Or equivalently

30 Modeling the features

d �(�, �, ⋯ , �|�) required k(2 – 1) parameters

What if all the features were conditionally independent given the label? The Naïve Bayes Assumption

That is, � �, �, ⋯ , � � = � � � � � � ⋯ � � �

Requires only d numbers for each label. kd features overall. Not bad!

31 Modeling the features

d �(�, �, ⋯ , �|�) required k(2 – 1) parameters

What if all the features were conditionally independent given the label? The Naïve Bayes Assumption

That is, � �, �, ⋯ , � � = � � � � � � ⋯ � � �

Requires only d numbers for each label. kd parameters overall. Not bad!

32 The Naïve Bayes Classifier

Assumption: Features are conditionally independent given the label Y

To predict, we need two sets of probabilities – Prior P(y)

– For each xj, we have the likelihood P(xj | y)

33 The Naïve Bayes Classifier

Assumption: Features are conditionally independent given the label Y

To predict, we need two sets of probabilities – Prior P(y)

– For each xj, we have the likelihood P(xj | y) Decision rule

ℎ � = argmax � � � �, �, ⋯ , � �)

34 The Naïve Bayes Classifier

Assumption: Features are conditionally independent given the label Y

To predict, we need two sets of probabilities – Prior P(y)

– For each xj, we have the likelihood P(xj | y) Decision rule

ℎ � = argmax � � � �, �, ⋯ , � �)

= argmax � � �(�|�)

35 Decision boundaries of naïve Bayes

What is the decision boundary of the naïve Bayes classifier?

Consider the two class case. We predict the label to be + if

� � = + � � � = + > � � = − � � � = −)

36 Decision boundaries of naïve Bayes

What is the decision boundary of the naïve Bayes classifier?

Consider the two class case. We predict the label to be + if

� � = + � � � = + > � � = − � � � = −)

� � = + ∏ � � � = +) > 1 � � = − ∏ �(�|� = −)

37 Decision boundaries of naïve Bayes

What is the decision boundary of the naïve Bayes classifier?

Taking log and simplifying, we get

�(� = −|�) log = �� + � �(� = +|�)

This is a linear function of the feature space!

Easy to prove. See note on course website

38 Today’s lecture

• The naïve Bayes Classifier

• Learning the naïve Bayes Classifier

• Practical Concerns

39 Learning the naïve Bayes Classifier

• What is the hypothesis function h defined by? – A collection of probabilities • Prior for each label: P(y)

• Likelihoods for feature xj given a label: P(xj| y)

If we have a data set D = {(xi, yi)} with m examples And we want to learn the classifier in a probabilistic way – What is the probabilistic criterion to select the hypothesis?

40 Learning the naïve Bayes Classifier

• What is the hypothesis function h defined by? – A collection of probabilities • Prior for each label: �(�)

• Likelihoods for feature xj given a label: �(��| �)

If we have a data set D = {(xi, yi)} with m examples And we want to learn the classifier in a probabilistic way – What is the probabilistic criterion to select the hypothesis?

41 Learning the naïve Bayes Classifier

• What is the hypothesis function h defined by? – A collection of probabilities • Prior for each label: �(�)

• Likelihoods for feature xj given a label: �(��| �)

Suppose we have a data set � = {(��, ��)} with m examples

42 Learning the naïve Bayes Classifier

• What is the hypothesis function h defined by? – A collection of probabilities • Prior for each label: �(�)

• Likelihoods for feature xj given a label: �(��| �)

Suppose we have a data set � = {(��, ��)} with m examples

A note on convention for this section: • Examples in the dataset are indexed by the subscript � (e.g. ��) • Features within an example are indexed by the subscript � • The � feature of the � example will be �

43 Learning the naïve Bayes Classifier

• What is the hypothesis function h defined by? – A collection of probabilities • Prior for each label: �(�)

• Likelihoods for feature xj given a label: �(��| �)

If we have a data set � = {(��, ��)} with m examples And we want to learn the classifier in a probabilistic way – What is a probabilistic criterion to select the hypothesis?

44 Learning the naïve Bayes Classifier

Maximum likelihood estimation

Here h is defined by all the probabilities used to construct the naïve Bayes decision

45 Maximum likelihood estimation

Given a dataset � = {(��, ��)} with m examples

Each example in the dataset is independent and identically distributed

So we can represent P(D| h) as this product

46 Maximum likelihood estimation

Given a dataset � = {(��, ��)} with m examples

Each example in the dataset is independent and identically distributed

So we can represent P(D| h) as this product

Asks “What probability would this particular h assign to the pair (xi, yi)?”

47 Maximum likelihood estimation

Given a dataset D = {(xi, yi)} with m examples

48 Maximum likelihood estimation

Given a dataset D = {(xi, yi)} with m examples

th xij is the j feature of xi

The Naïve Bayes assumption

49 Maximum likelihood estimation

Given a dataset D = {(xi, yi)} with m examples

How do we proceed?

50 Maximum likelihood estimation

Given a dataset D = {(xi, yi)} with m examples

51 Learning the naïve Bayes Classifier

Maximum likelihood estimation

What next?

52 Learning the naïve Bayes Classifier

Maximum likelihood estimation

What next?

We need to make a modeling assumption about the functional form of these probability distributions

53 Learning the naïve Bayes Classifier

Maximum likelihood estimation

For simplicity, suppose there are two labels 1 and 0 and all features are binary • Prior: P(y = 1) = p and P (y = 0) = 1 – p

That is, the prior probability is from the Bernoulli distribution.

54 Learning the naïve Bayes Classifier

Maximum likelihood estimation

For simplicity, suppose there are two labels 1 and 0 and all features are binary • Prior: P(y = 1) = p and P (y = 0) = 1 – p • Likelihood for each feature given a label

• P(xj = 1 | y = 1) = aj and P(xj = 0 | y = 1) = 1 – aj • P(xj = 1 | y = 0) = bj and P(xj = 0 | y = 0) = 1 - bj

55 Learning the naïve Bayes Classifier

Maximum likelihood estimation

For simplicity, suppose there are two labels 1 and 0 and all features are binary • That is, the Prior: P(y = 1) = p and P (y = 0) = 1 – p likelihood of each • feature is also is Likelihood for each feature given a label from the Bernoulli • P(xj = 1 | y = 1) = aj and P(xj = 0 | y = 1) = 1 – aj distribution. • P(xj = 1 | y = 0) = bj and P(xj = 0 | y = 0) = 1 - bj

56 Learning the naïve Bayes Classifier

Maximum likelihood estimation

For simplicity, suppose there are two labels 1 and 0 and all features are binary

• Prior: P(y = 1) = p and P (y = 0) = 1 – p h consists of p, all the a’s and b’s • Likelihood for each feature given a label

• P(xj = 1 | y = 1) = aj and P(xj = 0 | y = 1) = 1 – aj • P(xj = 1 | y = 0) = bj and P(xj = 0 | y = 0) = 1 - bj

57 Learning the naïve Bayes Classifier

Maximum likelihood estimation

• Prior: P(y = 1) = p and P (y = 0) = 1 – p

58 Learning the naïve Bayes Classifier

Maximum likelihood estimation

• Prior: P(y = 1) = p and P (y = 0) = 1 – p

[z] is called the indicator function or the Iverson bracket

Its value is 1 if the argument z is true and zero otherwise

59 Learning the naïve Bayes Classifier

Maximum likelihood estimation

Likelihood for each feature given a label

• P(xj = 1 | y = 1) = aj and P(xj = 0 | y = 1) = 1 – aj • P(xj = 1 | y = 0) = bj and P(xj = 0 | y = 0) = 1 - bj

60 Learning the naïve Bayes Classifier

Substituting and deriving the argmax, we get

P(y = 1) = p

61 Learning the naïve Bayes Classifier

Substituting and deriving the argmax, we get

P(y = 1) = p

P(xj = 1 | y = 1) = aj

62 Learning the naïve Bayes Classifier

Substituting and deriving the argmax, we get

P(y = 1) = p

P(xj = 1 | y = 1) = aj

P(xj = 1 | y = 0) = bj

63 Let’s learn a naïve Bayes classifier With the assumption that all O T H W Play? our probabilities are from the 1 S H H W - Bernoulli distribution 2 S H H S - 3 O H H W + 4 R M H W + 5 R C N W + 6 R C N S - 7 O C N S + 8 S M H W - 9 S C N W + 10 R M N W + 11 S M N S + 12 O M H S + 13 O H N W + 14 R M H S -

64 Let’s learn a naïve Bayes classifier

9 5 O T H W Play? � �� = + = � �� = − = 14 14 1 S H H W - 2 S H H S - 3 O H H W + 4 R M H W + 5 R C N W + 6 R C N S - 7 O C N S + 8 S M H W - 9 S C N W + 10 R M N W + 11 S M N S + 12 O M H S + 13 O H N W + 14 R M H S -

65 Let’s learn a naïve Bayes classifier

9 5 O T H W Play? � �� = + = � �� = − = 14 14 1 S H H W - 2 S H H S - 3 O H H W + 2 4 R M H W + �(� = � | �� = +) = 9 5 R C N W + 6 R C N S - 7 O C N S + 8 S M H W - 9 S C N W + 10 R M N W + 11 S M N S + 12 O M H S + 13 O H N W + 14 R M H S -

66 Let’s learn a naïve Bayes classifier

9 5 O T H W Play? � �� = + = � �� = − = 14 1 S H H W - 14 2 S H H S - 3 O H H W + 2 4 R M H W + �(� = � | �� = +) = 9 5 R C N W + 3 6 R C N S - �(� = � | �� = +) = 7 O C N S + 9 8 S M H W - 9 S C N W + 10 R M N W + 11 S M N S + 12 O M H S + 13 O H N W + 14 R M H S -

67 Let’s learn a naïve Bayes classifier

9 5 O T H W Play? � �� = + = � �� = − = 14 1 S H H W - 14 2 S H H S - 3 O H H W + 2 4 R M H W + �(� = � | �� = +) = 9 5 R C N W + 3 6 R C N S - �(� = � | �� = +) = 7 O C N S + 9 8 S M H W - 4 �(� = � | �� = +) = 9 S C N W + 9 10 R M N W + And so on, for other attributes and also for Play = - 11 S M N S + 12 O M H S + 13 O H N W + 14 R M H S -

68 Naïve Bayes: Learning and Prediction

• Learning – Count how often features occur with each label. Normalize to get likelihoods – Priors from fraction of examples with each label – Generalizes to multiclass

• Prediction – Use learned probabilities to find highest scoring label

69 Today’s lecture

• The naïve Bayes Classifier

• Learning the naïve Bayes Classifier

• Practical concerns + an example

70 Important caveats with Naïve Bayes

1. Features need not be conditionally independent given the label – Just because we assume that they are doesn’t mean that that’s how they behave in nature – We made a modeling assumption because it makes computation and learning easier

2. Not enough training data to get good estimates of the probabilities from counts

71 Important caveats with Naïve Bayes

1. Features are not conditionally independent given the label

All bets are off if the naïve Bayes assumption is not satisfied

And yet, very often used in practice because of simplicity Works reasonably well even when the assumption is violated

72 Important caveats with Naïve Bayes

2. Not enough training data to get good estimates of the probabilities from counts

The basic operation for learning likelihoods is counting how often a feature occurs with a label.

What if we never see a particular feature with a particular label? Eg: Suppose we never observe Temperature = cold with PlayTennis= Yes

Should we treat those counts as zero?

73 Important caveats with Naïve Bayes

2. Not enough training data to get good estimates of the probabilities from counts

The basic operation for learning likelihoods is counting how often a feature occurs with a label.

What if we never see a particular feature with a particular label? Eg: Suppose we never observe Temperature = cold with PlayTennis= Yes

Should we treat those counts as zero? But that will make the probabilities zero

74 Important caveats with Naïve Bayes

2. Not enough training data to get good estimates of the probabilities from counts

The basic operation for learning likelihoods is counting how often a feature occurs with a label.

What if we never see a particular feature with a particular label? Eg: Suppose we never observe Temperature = cold with PlayTennis= Yes

Should we treat those counts as zero? But that will make the probabilities zero

Answer: Smoothing • Add fake counts (very small numbers so that the counts are not zero) • The Bayesian interpretation of smoothing: Priors on the hypothesis (MAP learning)

75 Example: Classifying text

• Instance space: Text documents • Labels: Spam or NotSpam

• Goal: To learn a function that can predict whether a new document is Spam or NotSpam How would you build a Naïve Bayes classifier?

Let us brainstorm

How to represent documents? How to estimate probabilities? How to classify?

76 Example: Classifying text