Hmms and Crfs

School of Computer Science

10-701 Introduction to Machine Learning

HMMs and CRFs

Readings: Matt Gormley Bishop 13.1-13.2 Bishop 8.3-8.4 Lecture 19 Sutton & McCallum (2006) Laﬀerty et al. (2001) November 14, 2016

1 Reminders • Homework 4 – deadline extended to Wed, Nov. 16th – 10 extra points for submitting by Mon, Nov. 14th • Poster Sessions – two sessions on Fri, Dec. 2nd – session 1: 8 - 11:30 am – session 2: 2 - 6 pm

2 HIDDEN MARKOV MODEL (HMM)

3 Dataset for Supervised Part-of-Speech (POS) Tagging Data: = x(n), y(n) N D { }n=1

n v p d n y(1) Sample 1: time flies like an arrow x(1)

n n v d n y(2) Sample 2: time flies like an arrow x(2)

(3) n v p n n y Sample 3: (3) flies fly with their wings x

(4) p n n v v y Sample 4: (4) with time you will see x

4 Naïve Bayes for Time Series Data We could treat each word-tag pair (i.e. token) as independent. This corresponds to a Naïve Bayes model with a single feature (the word).

p(n, v, p, d, n, time, flies, like, an, arrow) = (.3 * .8 * .1 * .5 * …)

v .1 v .1 n .8 n .8 p .2 p .2 d .2 d .2

n v p d n … … like like like flies flies time time flies time like an arrow v .2 .5 .2 v .2 .5 .2 n .3 .4 .2 n .3 .4 .2 p .1 .1 .3 p .1 .1 .3 d .1 .2 .1 d .1 .2 .1 5 Hidden Markov Model A Hidden Markov Model (HMM) provides a joint distribution over the the sentence/tags with an assumption of dependence between adjacent tags.

p(n, v, p, d, n, time, flies, like, an, arrow) = (.3 * .8 * .2 * .5 * …)

v n p d v n p d v .1 .4 .2 .3 v .1 .4 .2 .3 n .8 .1 .1 0 n .8 .1 .1 0 p .2 .3 .2 .3 p .2 .3 .2 .3 d .2 .8 0 0 d .2 .8 0 0

n v p d n … … like like like flies flies time time flies time like an arrow v .2 .5 .2 v .2 .5 .2 n .3 .4 .2 n .3 .4 .2 p .1 .1 .3 p .1 .1 .3 d .1 .2 .1 d .1 .2 .1 6 From NB to HMM

Y Y Y Y Y 1 2 3 4 5

X1 X2 X3 X4 X5 K P (s, u)= P (X Y )p(Y ) “Naïve Bayes”: k| k k k=1

Y0 Y1 Y2 Y3 Y4 Y5

X X X X X 1 2 3 4 5 K P (s, u)= P (Xk Yk)p(Yk Yk 1) HMM: | | 7 k=1 Hidden Markov Model HMM Parameters: Emission matrix, , where P (X = w Y = t)=A , k k | k t,w Transition matrix, ", where P (Yk = t Yk 1 = s)=Bs,t, k | v n p d v n p d v 1 6 3 4 v 1 6 3 4 n 8 4 2 0.1 n 8 4 2 0.1 p 1 3 1 3 p 1 3 1 3 d 0.1 8 0 0 d 0.1 8 0 0

Y0 Y1 Y2 Y3 Y4 Y5 … … like like like flies flies time time X1 X2 X3 X4 X5 v 3 5 3 v 3 5 3 n 4 5 2 n 4 5 2 p 0.1 0.1 3 p 0.1 0.1 3 d 0.1 0.2 0.1 d 0.1 0.2 0.1 8 Hidden Markov Model HMM Parameters: Emission matrix, , where P (X = w Y = t)=A , k k | k t,w Transition matrix, ", where P (Yk = t Yk 1 = s)=Bs,t, k | Assumption: y0 = START Generative Story:

Yk Multinomial(Yk 1 ) k X Multinomial(" ) k k Yk

Y0 Y1 Y2 Y3 Y4 Y5

X1 X2 X3 X4 X5 9 Hidden Markov Model Joint Distribution: K p(t, v)= p(xk yk)p(yk yk 1) | | k=1 K

= Ayk,xk Byk 1,yk k=1

Y0 Y1 Y2 Y3 Y4 Y5

X1 X2 X3 X4 X5 10 From static to dynamic mixture models

Static mixture Dynamic mixture

Y1 Y1 Y2 Y3 ... YT

XA1 XA1 XA2 XA3 ... XAT N © Eric Xing @ CMU, 2006-2011 11 HMMs: History • Markov chains: Andrey Markov (1906) – Random walks and Brownian motion • Used in Shannon’s work on information theory (1948) • Baum-Welsh learning algorithm: late 60’s, early 70’s. – Used mainly for speech in 60s-70s. • Late 80’s and 90’s: David Haussler (major player in learning theory in 80’s) began to use HMMs for modeling biological sequences • Mid-late 1990’s: Dayne Freitag/Andrew McCallum – Freitag thesis with Tom Mitchell on IE from Web using logic programs, grammar induction, etc. – McCallum: multinomial Naïve Bayes for text – With McCallum, IE using HMMs on CORA • …

12 Slide from William Cohen Higher-order HMMs • 1st-order HMM (i.e. bigram HMM)

Y1 Y2 Y3 Y4 Y5

X1 X2 X3 X4 X5 • 2nd-order HMM (i.e. trigram HMM)

Y1 Y2 Y3 Y4 Y5

X1 X2 X3 X4 X5 • 3rd-order HMM

Y1 Y2 Y3 Y4 Y5

X X X X X 1 2 3 4 5

13 SUPERVISED LEARNING FOR BAYES NETS

14 Machine Learning

The data inspires Our model the structures deﬁnes a score we want to for each structure predict Domain Mathematical It also tells us Knowledge Modeling what to optimize ML

Inference ﬁnds Combinatorial Optimization Optimization {best structure, marginals, partition function} for a new observation Learning tunes the parameters of the (Inference is usually model called as a subroutine

in learning) 15 3 Alice saw Bob on a hill with a telescope

a telescope hill with on a Bob saw Machine Learning Alice Model 4 time ﬂies like an arrow X Data 1 X an arrow 3 like X2 ﬂies time

X4 X5 an arrow like ﬂies time

an arrow Objective like ﬂies time

an arrow like ﬂies timeInference

Learning

2 (Inference is usually called as a subroutine

in learning) 16 Recall… Learning Fully Observed BNs

X1 p(X1,X2,X3,X4,X5)= X X 3 2 p(X X )p(X X ,X ) 5| 3 4| 2 3

X4 X p(X3)p(X2 X1)p(X1) 5 |

17 Recall… Learning Fully Observed BNs

X1 p(X1,X2,X3,X4,X5)= X X 3 2 p(X X )p(X X ,X ) 5| 3 4| 2 3

X4 X p(X3)p(X2 X1)p(X1) 5 |

18 Recall… Learning Fully Observed BNs

X1 p(X1,X2,X3,X4,X5)= X X 3 2 p(X X )p(X X ,X ) 5| 3 4| 2 3

X4 X p(X3)p(X2 X1)p(X1) 5 |

How do we learn these conditional and marginal distributions for a Bayes Net?

19 Recall… Learning Fully Observed BNs Learning this fully observed Bayesian Network is p(X1,X2,X3,X4,X5)= p(X X )p(X X ,X ) equivalent to learning ﬁve 5| 3 4| 2 3 p(X )p(X X )p(X ) (small / simple) independent 3 2| 1 1 networks from the same data

X1 X1 X1

X X 3 2 X3 X2

X3 X3 X2 X4 X5

X4 X5 20 Learning Fully Observed BNs

How do we learn these conditional and marginal distributions for a Bayes Net? ✓✓⇤⇤ == argmax argmaxloglogpp((XX11,X,X22,X,X33,X,X44,X,X55)) ✓✓

== argmax argmaxloglogpp((XX55 XX33,,✓✓55)) ++ log logpp((XX44 XX22,X,X33,,✓✓44)) ✓✓ || || X1 ++ log logpp((XX3 ✓✓3)) ++ log logpp((XX2 XX1,,✓✓2)) 3|| 3 2|| 1 2 X3 ++ log logpp((XX11 ✓✓11)) X2 ||

✓1⇤ = argmax log p(X1 ✓1) ✓1 | X4 X5 ✓2⇤ = argmax log p(X2 X1, ✓2) ✓2 |

✓3⇤ = argmax log p(X3 ✓3) ✓3 |

✓4⇤ = argmax log p(X4 X2,X3, ✓4) ✓4 |

✓5⇤ = argmax log p(X5 X3, ✓5) ✓5 | 21 SUPERVISED LEARNING FOR HMMS

22 Hidden Markov Model HMM Parameters: Emission matrix, , where P (X = w Y = t)=A , k k | k t,w Transition matrix, ", where P (Yk = t Yk 1 = s)=Bs,t, k | v n p d v n p d v 1 6 3 4 v 1 6 3 4 n 8 4 2 0.1 n 8 4 2 0.1 p 1 3 1 3 p 1 3 1 3 d 0.1 8 0 0 d 0.1 8 0 0

Y0 Y1 Y2 Y3 Y4 Y5 … … like like like flies flies time time X1 X2 X3 X4 X5 v 3 5 3 v 3 5 3 n 4 5 2 n 4 5 2 p 0.1 0.1 3 p 0.1 0.1 3 d 0.1 0.2 0.1 d 0.1 0.2 0.1 23 Hidden Markov Model HMM Parameters: Emission matrix, , where P (X = w Y = t)=A , k k | k t,w Transition matrix, ", where P (Yk = t Yk 1 = s)=Bs,t, k | Assumption: y0 = START Generative Story:

Yk Multinomial(Yk 1 ) k X Multinomial(" ) k k Yk

Y0 Y1 Y2 Y3 Y4 Y5

X1 X2 X3 X4 X5 24 Hidden Markov Model Joint Distribution: K p(t, v)= p(xk yk)p(yk yk 1) | | k=1 K

= Ayk,xk Byk 1,yk k=1

Y0 Y1 Y2 Y3 Y4 Y5

X1 X2 X3 X4 X5 25 Whiteboard • MLEs for HMM

26 Representation of both directed and undirected graphical models FACTOR GRAPHS

27 Sampling from a Joint Distribution

A joint distribution deﬁnes a probability p(x) for each assignment of values x to variables X. This gives the proportion of samples that will equal x.

Sample 1: n v p d n

Sample 2: n n v d n

Sample 3: n v p d n

Sample 4: v n p d n

Sample 5: v n v d n

Sample 6: n v p d n

ψ ψ ψ ψ X0 ψ0 X1 2 X2 4 X3 6 X4 8 X5

ψ1 ψ3 ψ5 ψ7 ψ9 time flies like an arrow 28 Sampling from a Joint Distribution

A joint distribution deﬁnes a probability p(x) for each assignment of values x to variables X. This gives the proportion of samples that will equal x.

Sample 1: Sample 2:

ψ11 ψ11

ψ12 ψ12 ψ10 ψ10 ψ6 ψ6 ψ4 ψ4 ψ2 ψ2 ψ8 ψ8 ψ5 ψ5 ψ ψ 1 Sample 3: 1 Sample 4:

ψ7 ψ7 ψ ψ ψ ψ3 ψ11 9 ψ3 11 9 ψ ψ12 12 ψ ψ10 10 ψ ψ6 6 ψ ψ4 ψ 4 ψ2 2 ψ ψ8 ψ 8 ψ5 5 ψ ψ1 1

ψ ψ7 7 ψ ψ ψ9 ψ3 9 3

X7 X ψ11 6

X3 ψ12 X 1 ψ10 ψ6 ψ4 ψ2 X4 ψ8 ψ5 ψ X 1 X2 5

ψ7 ψ 29 ψ3 9 Sampling from a Joint Distribution

A joint distribution deﬁnes a probability p(x) for each assignment of values x to variables X. This gives the proportion of samples that will equal x.

n v p d n Sample 1: time flies like an arrow

n n v d n Sample 2: time flies like an arrow

n v p n n Sample 3: flies fly with their wings

p n n v v Sample 4: with time you will see

X0 ψ0 X1 ψ2 X2 ψ4 X3 ψ6 X4 ψ8 X5

ψ1 ψ3 ψ5 ψ7 ψ9

W1 W2 W3 W4 W5 30 Factors have local opinions (≥ 0)

Each black box looks at some of the tags Xi and words Wi

Note: We chose to reuse v n p d v n p d the same factors at v 1 6 3 4 v 1 6 3 4 different positions in the n 8 4 2 0.1 n 8 4 2 0.1 sentence. p 1 3 1 3 p 1 3 1 3 d 0.1 8 0 0 d 0.1 8 0 0

X0 ψ0 X1 ψ2 X2 ψ4 X3 ψ6 X4 ψ8 X5

ψ1 ψ3 ψ5 ψ7 ψ9 … … like like like flies flies time time W1 W2 W3 W4 W5 v 3 5 3 v 3 5 3 n 4 5 2 n 4 5 2 p 0.1 0.1 3 p 0.1 0.1 3 d 0.1 0.2 0.1 d 0.1 0.2 0.1 31 Factors have local opinions (≥ 0)

Each black box looks at some of the tags Xi and words Wi

p(n, v, p, d, n, time, flies, like, an, arrow) = ?

v n p d v n p d v 1 6 3 4 v 1 6 3 4 n 8 4 2 0.1 n 8 4 2 0.1 p 1 3 1 3 p 1 3 1 3 d 0.1 8 0 0 d 0.1 8 0 0

ψ0 n ψ2 v ψ4 p ψ6 d ψ8 n

ψ1 ψ3 ψ5 ψ7 ψ9 … … like like like flies flies time time flies time like an arrow v 3 5 3 v 3 5 3 n 4 5 2 n 4 5 2 p 0.1 0.1 3 p 0.1 0.1 3 d 0.1 0.2 0.1 d 0.1 0.2 0.1 32 Global probability = product of local opinions

Each black box looks at some of the tags Xi and words Wi

p(n, v, p, d, n, time, flies, like, an, arrow) = (4 * 8 * 5 * 3 * …)

v n p d v n p d Uh-oh! The probabilities of v 1 6 3 4 v 1 6 3 4 the various assignments sum n 8 4 2 0.1 n 8 4 2 0.1 up to Z > 1. p 1 3 1 3 p 1 3 1 3 So divide them all by Z. d 0.1 8 0 0 d 0.1 8 0 0

ψ0 n ψ2 v ψ4 p ψ6 d ψ8 n

ψ1 ψ3 ψ5 ψ7 ψ9 … … like like like flies flies time time flies time like an arrow v 3 5 3 v 3 5 3 n 4 5 2 n 4 5 2 p 0.1 0.1 3 p 0.1 0.1 3 d 0.1 0.2 0.1 d 0.1 0.2 0.1 33 Markov Random Field (MRF)

Joint distribution over tags Xi and words Wi The individual factors aren’t necessarily probabilities.

p(n, v, p, d, n, time, flies, like, an, arrow) = (4 * 8 * 5 * 3 * …)

v n p d v n p d v 1 6 3 4 v 1 6 3 4 n 8 4 2 0.1 n 8 4 2 0.1 p 1 3 1 3 p 1 3 1 3 d 0.1 8 0 0 d 0.1 8 0 0

ψ0 n ψ2 v ψ4 p ψ6 d ψ8 n

ψ1 ψ3 ψ5 ψ7 ψ9 … … like like like flies flies time time flies time like an arrow v 3 5 3 v 3 5 3 n 4 5 2 n 4 5 2 p 0.1 0.1 3 p 0.1 0.1 3 d 0.1 0.2 0.1 d 0.1 0.2 0.1 34 Bayesian Networks But sometimes we choose to make them probabilities. Constrain each row of a factor to sum to one. Now Z = 1.

p(n, v, p, d, n, time, flies, like, an, arrow) = (.3 * .8 * .2 * .5 * …)

v n p d v n p d v .1 .4 .2 .3 v .1 .4 .2 .3 n .8 .1 .1 0 n .8 .1 .1 0 p .2 .3 .2 .3 p .2 .3 .2 .3 d .2 .8 0 0 d .2 .8 0 0

n v p d n … … like like like flies flies time time flies time like an arrow v .2 .5 .2 v .2 .5 .2 n .3 .4 .2 n .3 .4 .2 p .1 .1 .3 p .1 .1 .3 d .1 .2 .1 d .1 .2 .1 35 Markov Random Field (MRF)

Joint distribution over tags Xi and words Wi

p(n, v, p, d, n, time, flies, like, an, arrow) = (4 * 8 * 5 * 3 * …)

v n p d v n p d v 1 6 3 4 v 1 6 3 4 n 8 4 2 0.1 n 8 4 2 0.1 p 1 3 1 3 p 1 3 1 3 d 0.1 8 0 0 d 0.1 8 0 0

ψ0 n ψ2 v ψ4 p ψ6 d ψ8 n

ψ1 ψ3 ψ5 ψ7 ψ9 … … like like like flies flies time time flies time like an arrow v 3 5 3 v 3 5 3 n 4 5 2 n 4 5 2 p 0.1 0.1 3 p 0.1 0.1 3 d 0.1 0.2 0.1 d 0.1 0.2 0.1 36 Conditional Random Field (CRF)

Conditional distribution over tags Xi given words wi. The factors and Z are now speciﬁc to the sentence w.

p(n, v, p, d, n | time, flies, like, an, arrow) = (4 * 8 * 5 * 3 * …)

v n p d v n p d v 1 6 3 4 v 1 6 3 4 n 8 4 2 0.1 n 8 4 2 0.1 p 1 3 1 3 p 1 3 1 3 d 0.1 8 0 0 d 0.1 8 0 0

ψ0 n ψ2 v ψ4 p ψ6 d ψ8 n

ψ ψ ψ ψ ψ v 3 1 3 v 5 5 7 9 n 4 n 5 p 0.1 p 0.1 d 0.1 d 0.2

time flies like an arrow 37 How General Are Factor Graphs? • Factor graphs can be used to describe – Markov Random Fields (undirected graphical models) • i.e., log-linear models over a tuple of variables – Conditional Random Fields – Bayesian Networks (directed graphical models)

• Inference treats all of these interchangeably. – Convert your model to a factor graph ﬁrst. – Pearl (1988) gave key strategies for exact inference: • Belief propagation, for inference on acyclic graphs • Junction tree algorithm, for making any graph acyclic (by merging variables and factors: blows up the runtime) Factor Graph Notation

X • Variables: 9

ψψ{1,8,9}{1,8,9}{1,8,9}

• Factors: ψ{2,7,8}

ψ{3,6,7}

ψψ Joint Distribution 101010

X1 ψψψ{1,2}222 X2 ψ{2,3} X3 ψ{3,4} X4 ψψ888 X5

ψψψ{1}111 ψψψ{2}333 ψψψ{3}555 ψψ777 ψψ99

time flies like an arrow 39 Factors are Tensors

s vp pp … s vp pp … s 0 2 .3 X9 s 0 s2 vp.3 pp … vp 3 4 2 vp s3 0 4 2 2 .3 ψψ pp .1 2 1 {1,8,9}{1,8,9}{1,8,9} ppvp .1 3 2 4 1 2 … X8 …pp .1 2 1 s • Factors: vp … ψ{2,7,8} pp X7 v n p d v 1 6 3 4 ψ{3,6,7} n 8 4 2 0.1 X p 1 3 1 3 6 d 0.1 8 0 0 ψψ101010

X1 ψψψ{1,2}222 X2 ψ{2,3} X3 ψ{3,4} X4 ψψ888 X5 v 3 n 4 p 0.1 ψψψ{1}111 ψψψ{2}333 ψψψ{3}555 ψψ777 ψψ99 d 0.1 time flies like an arrow 40 Converting to Factor Graphs Each conditional and Each clique in an marginal distribution in a undirected GM becomes a directed GM becomes a factor factor

X1 X1 X1 X1 X1 X1

X1 X1

X1 X1 X1 X1

X X X X1 X1 X1 1 1 1

X X1 1

X X X1 X1 1 1 41 Equivalence of directed and undirected trees • Any undirected tree can be converted to a directed tree by choosing a root node and directing all edges away from it

• A directed tree and the corresponding undirected tree make the same conditional independence assertions • Parameterizations are essentially the same.

– Undirected tree:

– Directed tree:

– Equivalence:

X X X X 1 5 fa 1 fd 5

X3 fc X3

fe X2 X4 fb X2 X4

P(X1) P(X2) P(X3|X1,X2) P(X5|X1,X3) P(X4|X2,X3)

fa(X1) fb(X2) fc(X3,X1,X2) fd(X5,X1,X3) fe(X4,X2,X3)

X1 X1

fa fc

X2 X3 X2 X3 f ψ(x1,x2,x3) = fa(x1,x2)fb(x2,x3)fc(x3,x1) b • Example 3 X1 X1

X2 X3 X2 X3

ψ(x1,x2,x3) = fa(x1,x2,x3) © Eric Xing @ CMU, 2005-2015 44 Tree-like Undirected GMs to Factor Trees

X2 X2

X X3 4

X3 X4

X5 X6

X5 X6 © Eric Xing @ CMU, 2005-2015 45 Poly-trees to Factor trees

X1 X2

X3 X4 X3 X4

• Because FG turns tree-like X1

X1 graphs to factor trees, • Trees are a data-structure that X 2 X2 guarantees correctness of BP !

X4 X3

X3 X4

X5 X6

X1 X2

X3 X4 X3 X4

X5 © Eric Xing @ CMU, 2005-2015 47 THE FORWARD-BACKWARD ALGORITHM

48 Learning and Inference Summary For discrete variables:

Learning Marginal MAP Inference Inference HMM Forward- Viterbi backward Linear-chain Forward- Viterbi CRF backward

49 Forward-Backward Algorithm

Y1 Y2 Y3

find preferred tags

Could be verb or noun Could be adjective or verb Could be noun or verb

50 Forward-Backward Algorithm

Y1 Y2 Y3

v v v

START n n n END

a a a

find preferred tags • Show the possible values for each variable

51 Forward-Backward Algorithm

Y1 Y2 Y3

v v v

START n n n END

a a a

find preferred tags • Let’s show the possible values for each variable • One possible assignment 52 Forward-Backward Algorithm

Y1 Y2 Y3

v v v

START n n n END

a a a

find preferred tags • Let’s show the possible values for each variable • One possible assignment • And what the 7 factors think of it … 53 Viterbi Algorithm: Most Probable Assignment

Y1 Y2 Y3

v v v

ψ{1}(v)

ψ{3,4}(a,END) START n n n END

ψ{3}(n)

a a a

ψ{2}(a)

find preferred tags • So p(v a n) = (1/Z) * product of 7 numbers • Numbers associated with edges and nodes of path • Most probable assignment = path with highest product 54 Viterbi Algorithm: Most Probable Assignment

Y1 Y2 Y3

v v v

ψ{1}(v)

ψ{3,4}(a,END) START n n n END

ψ{3}(n)

a a a

ψ{2}(a)

find preferred tags • So p(v a n) = (1/Z) * product weight of one path

55 Forward-Backward Algorithm: Finds Marginals

Y1 Y2 Y3

v v v

START n n n END

a a a

find preferred tags • So p(v a n) = (1/Z) * product weight of one path • Marginal probability p(Y2 = a) = (1/Z) * total weight of all paths through a 56 Forward-Backward Algorithm: Finds Marginals

Y1 Y2 Y3

v v v

START n n n END

a a a

find preferred tags • So p(v a n) = (1/Z) * product weight of one path • Marginal probability p(Y2 = a) = (1/Z) * total weight of all paths through n 57 Forward-Backward Algorithm: Finds Marginals

Y1 Y2 Y3

v v v

START n n n END

a a a

find preferred tags • So p(v a n) = (1/Z) * product weight of one path • Marginal probability p(Y2 = a) = (1/Z) * total weight of all paths through v 58 Forward-Backward Algorithm: Finds Marginals

Y1 Y2 Y3

v v v

START n n n END

a a a

find preferred tags • So p(v a n) = (1/Z) * product weight of one path • Marginal probability p(Y2 = a) = (1/Z) * total weight of all paths through n 59 Forward-Backward Algorithm: Finds Marginals

Y1 Y2 Y3

v v v

START n n n END

a a a

find preferred tags

α2(n) = total weight of these path preﬁxes

60 (found by dynamic programming: matrix-vector products) Forward-Backward Algorithm: Finds Marginals

Y1 Y2 Y3

v v v

START n n n END

a a a

find preferred tags

β2(n) = total weight of these path suﬃxes

61 (found by dynamic programming: matrix-vector products) Forward-Backward Algorithm: Finds Marginals

Y1 Y2 Y3

v v v

START n n n END

a a a

find preferred tags

α2(n) = total weight of these β2(n) = total weight of these path preﬁxes (a + b + c) path suﬃxes (x + y + z)

Product gives ax+ay+az+bx+by+bz+cx+cy+cz = total weight of paths 62 Forward-Backward Algorithm: Finds Marginals

Y1 Y2 Y3

Oops! The weight of a path v v v through a state also includes a weight at that state. “belief that Y = n” STARTSo α(n)·β(n) isn’t enough. n n n 2 END α2(n) β2(n) The extra weight is the opinion of the unigram a a a factor at this variable.

ψ{2}(n)

find preferred tags total weight of all paths through n

= α2(n) ⋅ ψ{2}(n)⋅ β2(n) 63 Forward-Backward Algorithm: Finds Marginals

Y1 Y2 Y3

v v “belief that v Y2 = v”

START n n “belief that n Y2 = n” END

α2(v) β2(v)

a a a

ψ{2}(v)

find preferred tags total weight of all paths through v

= α2(v) ⋅ ψ{2}(v)⋅ β2(v) 64 Forward-Backward Algorithm: Finds Marginals

v 1.8 Y1 Y2 Y3 n 0 a 4.2 v v “belief that v Y2 = v” divide v 0.3 by Z=6 to “belief that Y = n” STARTn 0 get n n n 2 END a 0.7 marginal α2(a) β2(a) probs a a “belief that a Y2 = a”

sum = Z ψ{2}(a) (total probability of all paths) find preferred tags total weight of all paths through a

= α2(a) ⋅ ψ{2}(a)⋅ β2(a) 65 CRF Tagging Model

Y1 Y2 Y3

find preferred tags

Could be verb or noun Could be adjective or verb Could be noun or verb

66 Whiteboard • Forward-backward algorithm • Viterbi algorithm

67 Conditional Random Fields (CRFs) for time series data LINEAR-CHAIN CRFS

68 Shortcomings of Hidden Markov Models

START Y1 Y2 … … … Yn

X1 X2 … … … Xn

• HMM models capture dependences between each state and only its corresponding observation – NLP example: In a sentence segmentation task, each segmental state may depend not just on a single word (and the adjacent segmental stages), but also on the (non- local) features of the whole line such as line length, indentation, amount of white space, etc. • Mismatch between learning objective function and prediction objective function – HMM learns a joint distribution of states and observations P(Y, X), but in a prediction task, we need the conditional probability P(Y|X) © Eric Xing @ CMU, 2005-2015 69 Conditional Random Field (CRF)

Conditional distribution over tags Xi given words wi. The factors and Z are now speciﬁc to the sentence w.

p(n, v, p, d, n | time, flies, like, an, arrow) = (4 * 8 * 5 * 3 * …)

v n p d v n p d v 1 6 3 4 v 1 6 3 4 n 8 4 2 0.1 n 8 4 2 0.1 p 1 3 1 3 p 1 3 1 3 d 0.1 8 0 0 d 0.1 8 0 0

ψ0 n ψ2 v ψ4 p ψ6 d ψ8 n

ψ ψ ψ ψ ψ v 3 1 3 v 5 5 7 9 n 4 n 5 p 0.1 p 0.1 d 0.1 d 0.2

time flies like an arrow 70 Conditional Random Field (CRF)

Recall: Shaded nodes in a graphical model are observed

v n p d v n p d v 1 6 3 4 v 1 6 3 4 n 8 4 2 0.1 n 8 4 2 0.1 p 1 3 1 3 p 1 3 1 3 d 0.1 8 0 0 d 0.1 8 0 0

ψ0 Y1 ψ2 Y2 ψ4 Y3 ψ6 Y4 ψ8 Y5

ψ ψ ψ ψ ψ v 3 1 3 v 5 5 7 9 n 4 n 5 p 0.1 p 0.1 d 0.1 d 0.2

X1 X2 X3 X4 X5 71 Conditional Random Field (CRF) This linear-chain CRF is just like an HMM, except that its factors are not necessarily probability distributions

1 K p(v t)= em(yk,xk)tr(yk,yk 1) | Z(t) k=1 1 K = 2tT( 7em(yk,xk))2tT( 7tr(yk,yk 1)) Z(t) · · k=1

ψ0 Y1 ψ2 Y2 ψ4 Y3 ψ6 Y4 ψ8 Y5

ψ1 ψ3 ψ5 ψ7 ψ9

X1 X2 X3 X4 X5 72 Quiz 1 K p(v t)= em(yk,xk)tr(yk,yk 1) | Z(t) k=1 1 K = 2tT( 7em(yk,xk))2tT( 7tr(yk,yk 1)) Z(t) · · k=1 Multiple Choice: Which model does the above distribution share the most in common with? A. Hidden Markov Model B. Bernoulli Naïve Bayes C. Gaussian Naïve Bayes D. Logistic Regression

73 Conditional Random Field (CRF) This linear-chain CRF is just like an HMM, except that its factors are not necessarily probability distributions

1 K p(v t)= em(yk,xk)tr(yk,yk 1) | Z(t) k=1 1 K = 2tT( 7em(yk,xk))2tT( 7tr(yk,yk 1)) Z(t) · · k=1

ψ0 Y1 ψ2 Y2 ψ4 Y3 ψ6 Y4 ψ8 Y5

ψ1 ψ3 ψ5 ψ7 ψ9

X1 X2 X3 X4 X5 74 Conditional Random Field (CRF)

• That is the vector X • Because it’s observed, we can condition on it for free • Conditioning is how we converted from the MRF to the CRF (i.e. when taking a slice of the emission factors)

v n p d v n p d v 1 6 3 4 v 1 6 3 4 n 8 4 2 0.1 n 8 4 2 0.1 p 1 3 1 3 p 1 3 1 3 d 0.1 8 0 0 d 0.1 8 0 0

ψ0 Y1 ψ2 Y2 ψ4 Y3 ψ6 Y4 ψ8 Y5

ψ ψ ψ ψ ψ v 3 1 3 v 5 5 7 9 n 4 n 5 p 0.1 p 0.1 d 0.1 d 0.2

X 75 Conditional Random Field (CRF)

• This is the standard linear-chain CRF deﬁnition • It permits rich, overlapping features of the vector X

1 K p(v t)= em(yk, t)tr(yk,yk 1, t) | Z(t) k=1 1 K = 2tT( 7em(yk, t))2tT( 7tr(yk,yk 1, t)) Z(t) · · k=1

ψ0 Y1 ψ2 Y2 ψ4 Y3 ψ6 Y4 ψ8 Y5

ψ1 ψ3 ψ5 ψ7 ψ9

X 76 Conditional Random Field (CRF)

• This is the standard linear-chain CRF deﬁnition • It permits rich, overlapping features of the vector X

1 K p(v t)= em(yk, t)tr(yk,yk 1, t) | Z(t) k=1 1 K = 2tT( 7em(yk, t))2tT( 7tr(yk,yk 1, t)) Z(t) · · k=1

ψ0 Y1 ψ2 Y2 ψ4 Y3 ψ6 Y4 ψ8 Y5

ψ1 ψ3 ψ5 ψ7 ψ9

Visual Notation: Usually we draw a CRF without showing the variable corresponding to X 77 Whiteboard • Forward-backward algorithm for linear-chain CRF

78 General CRF

Y The topology of the 9 ψψ graphical model for a CRF {1,8,9}{1,8,9}{1,8,9} Y doesn’t have to be a chain 8 ψ{2,7,8} Y 1 7 ψ p(v t)= (v, t; ) {3,6,7} | Z(t) Y 6 ψψ101010

Y1 ψψψ{1,2}222 Y2 ψ{2,3} Y3 ψ{3,4} Y4 ψψ888 Y5

ψψψ{1}111 ψψψ{2}333 ψψψ{3}555 ψψ777 ψψ99

time flies like an arrow 79 Standard CRF Parameterization 1 p (v t)= (v , t; ) | Z( ) t Deﬁne each potential function in terms of a ﬁxed set of feature functions: (v , t; )=2tT( 7 (v , t)) ·

Predicted Observed variables variables

80 Standard CRF Parameterization Deﬁne each potential function in terms of a ﬁxed set of feature functions: (v , t; )=2tT( 7 (v , t)) ·

n ψ2 v ψ4 p ψ6 d ψ8 n

ψ1 ψ3 ψ5 ψ7 ψ9

time flies like an arrow

81 Standard CRF Parameterization Deﬁne each potential function in terms of a ﬁxed set of feature functions: (v , t; )=2tT( 7 (v , t)) · s

ψ13 vp

ψ12 pp

ψ11 np

ψ10

ψ ψ ψ ψ n 2 v 4 p 6 d 8 n

ψ1 ψ3 ψ5 ψ7 ψ9 time flies like an arrow 82 Exact inference for tree-structured factor graphs BELIEF PROPAGATION

83 Inference for HMMs • Sum-product BP on an HMM is called the forward-backward algorithm • Max-product BP on an HMM is called the Viterbi algorithm

84 Inference for CRFs • Sum-product BP on a CRF is called the forward-backward algorithm • Max-product BP on a CRF is called the Viterbi algorithm

85 CRF Tagging by Belief Propagation

Forward algorithm = Backward algorithm = message passing belief message passing (matrix-vector products) v 1.8 (matrix-vector products) n 0 a 4.2 α message α β message β v 7 v n a v 3 v 2 v n a v 3 … … n 2 v 0 2 1 n 1 n v 1 0 2 1 n 6 a 1 n 2 1 0 a 6 a n 7 2 1 0 a 1 a 0 3 1 a 0 3 1 v 0.3 n 0 a 0.1

find preferred tags • Forward-backward is a message passing algorithm. • It’s the simplest case of belief propagation. 86 SUPERVISED LEARNING FOR CRFS

87 What is Training? That’s easy:

Training = picking good model parameters!

But how do we know if the model parameters are any “good”?

88 Machine Learning Log-likelihood Training

1. Choose model Such that derivative in #3 is ea 2. Choose objective: Assign high probability to the things we observe and low probability to everything else

3. Compute derivative by hand using the chain rule 4. Replace exact inference by approximate inference

89 Machine Learning Log-likelihood Training

1. Choose model Such that derivative in #3 is easy 2. Choose objective: Assign high probability to the things we observe and low probability to everything else

3. Compute derivative by hand using the chain rule 4. Compute the marginals by Note that these are factor marginals which are just the (normalized) exact inference factor beliefs from BP!

90 Recipe for Gradient-based Learning 1. Write down the objective function 2. Compute the partial derivatives of the objective (i.e. gradient, and maybe Hessian) 3. Feed objective function and derivatives into black box

Optimization

4. Retrieve optimal parameters from black box

91 Optimization Algorithms

What is the black box? Optimization • Newton’s method • Hessian-free / Quasi-Newton methods – Conjugate gradient – L-BFGS • Stochastic gradient methods – Stochastic gradient descent (SGD) – Stochastic meta-descent – AdaGrad

92 2.1.5 Newton-Raphson (Newton’s method, a second-order method) From our introductory example, we know that we can ﬁnd the solution to a quadratic function • analytically. Yet gradient descent may take many steps to converge to that optimum. The motivation behind Newton’s method is to use a quadratic approximation of our function to make a good guess where we should step next. Deﬁnition: the Hessian of an n-dimensional function is the matrix of partial second derivatives • with respect to each pair of dimensions.

d2f(~x) d2f(~x) 2 ... dx1 dx1dx2 2f(~x)= d2f(~x) d2f(~x) O 2 2 3 dx2dx1 dx2 6 ...... 7 6 7 4 5 Consider the secord order Taylor series expansion of f at x. • 1 gˆ(v)=fˆ(x + v)=f(x)+ f(x)T v + vT 2f(x)v O 2 O We want to ﬁnd the v that maximizes gˆ(v). This maximizer is called Newton’s step. x = • O nt arg maxv gˆ(v). Algorithm: • 1. Choose a starting point x. 2. While not converged: Compute Newton’s step x =( 2f(x)) 1 f(x) O nt O O Update x(k+1) = x(k) + x O nt Intuition: • If f(x) is quadratic, x + x exactly maximizes f. O nt gˆ(v) is a good quadratic approximation to the function f near the point x.Soiff(x) is locally quadratic, then f(x) is locally well approximated by gˆ(v). See Figure 9.17 in Boyd and Vandenberghe. In most presentations, Newton-Raphson would be presented a minimization algorithm, for • which we would negate the deﬁnition of Newton’s step from above.

2.1.6 Quasi-Newton methods (L-BFGS) What if we have n = millions of features? • The Hessian matrix H = 2f(x) is too large: n2 entries. • O quasi-Newton methods approximate the Hessian. • Limited memory BFGS stores only a history of the last k updates to ~x and f(~x). k is usually • O small (e.g. k = 10). This history is used to approximate the Hessian-vector product. • Optimization has nearly become a technology. Almost every language has many generic • optimization routines built in that you can use out of the box. Stochastic Gradient Descent 2.1.7 Stochastic Gradient Descent N Suppose we have N training examples s.t. f(x)= i=1 fi(x). • N GradientThis implies Descent: that Of(x)= i=1 Ofi(x). •• P N (k+1) (k) (k) ~x P = ~x + tOf(~x)=~x + t Ofi(x) i=1 6 X SGD Algorithm: • 1. Choose a starting point x. 2. While not converged: Choose a step size t. Choose i so that it sweeps through the training set. Update (k+1) (k) ~x = ~x + tOfi(~x) For CRF training Stochastic Meta Descent is even better (Vishwanathan, 2006). •

2.2 Additional readings 93 PGM Appendix A.5: Continuous Optimization • Boyd & Vandenberghe “Convex Optimization” http://www.stanford.edu/boyd/cvxbook/˜ • Chapter 9: Unconstrained Minimization ⌅ 9.3 Gradient Descent

⌅ 9.4 Steepest Descent

⌅ 9.5 Newton’s Method Intuitive explanation of Lagrange Multipliers (without the assumption of diﬀerentiability): • http://www.umiacs.umd.edu/˜resnik/ling848_fa2004/lagrange.html

2.3 Advanced readings “Overview of Quasi-Newton optimization methods” http://homes.cs.washington.edu/˜galen/ﬁles/quasi- • newton-notes.pdf Shewchuk (1994) “An Introduction to the Conjugate Gradient Method Without the Agonizing • Pain” http://www.cs.cmu.edu/˜quake-papers/painless-conjugate-gradient.pdf Conjugate Gradient Method: http://www.cs.iastate.edu/˜cs577/handouts/conjugate-gradient.pdf • 3 Third Review Session

3.1 Continuous Optimization (constrained) 3.1.1 Running example: MLE of a Multinomial Recall the pdf of the Categorical distribution: • support: X 0,...,k 2 { } pmf: p(X = k)=✓ k Let Xi Categorical(✓~) for 1 i N. • ⇠   N The likelihood of all these is: N ✓ = k ✓ l where N is the number of X = l. • i=1 Xi l=1 l l i The log-likelihood is then: LL(✓~)= k N log(✓ ) • Q l=1 Ql l Suppose we want to ﬁnd the maximum likelihood parameters: ✓~ = arg min LL(✓~) subject • P MLE ✓~ to the constraints k ✓ =1and 0 ✓ l. l=1 l  l8 P

7 Whiteboard • CRF model • CRF data log-likelihood • CRF derivatives

94 Practical Considerations for Gradient-based Methods • Overﬁtting – L2 regularization – L1 regularization – Regularization by early stopping • For SGD: Sparse updates

95 “Empirical” Comparison of Parameter Estimation Methods

• Example NLP task: CRF dependency parsing • Suppose: Training time is dominated by inference • Dataset: One million tokens • Inference speed: 1,000 tokens / sec • è 0.27 hours per pass through dataset

# passes through # hours to data to converge converge GIS 1000+ 270 L-BFGS 100+ 27 SGD 10 ~3

96 FEATURE ENGINEERING FOR CRFS

97 Slide adapted from 600.465 - Intro to NLP - J. Eisner Features

General idea: • Make a list of interesting substructures. th • The feature fk(x,y) counts tokens of k substructure in (x,y).

98 Slide courtesy of 600.465 - Intro to NLP - J. Eisner Features for tagging …