Dependency Grammar

Introducon Christopher Manning Dependency Grammar and Dependency Structure

Dependency syntax postulates that syntacc structure consists of relaons between lexical items, normally binary asymmetric relaons (“arrows”) called dependencies submitted

Bills were by

on Brownback

ports Senator Republican

and immigration of

Kansas Christopher Manning Dependency Grammar and Dependency Structure

Dependency syntax postulates that syntacc structure consists of relaons between lexical items, normally binary asymmetric relaons (“arrows”) called dependencies submitted nsubjpass auxpass prep The arrows are Bills were by commonly typed prep pobj with the name of on Brownback pobj nn appos grammacal ports Senator Republican relaons (subject, cc conj prep preposional object, and immigration of apposion, etc.) pobj Kansas Christopher Manning Dependency Grammar and Dependency Structure

Dependency syntax postulates that syntacc structure consists of relaons between lexical items, normally binary asymmetric relaons (“arrows”) called dependencies submitted The arrow connects a nsubjpass auxpass prep head (governor, Bills were by superior, regent) with a prep pobj dependent (modiﬁer, on inferior, subordinate) Brownback pobj nn appos

ports Senator Republican Usually, dependencies cc conj prep form a tree (connected, and immigration of acyclic, single-head) pobj Kansas Christopher Manning Dependency Grammar and Dependency Structure

ROOT Discussion of the outstanding issues was completed .

• Some people draw the arrows one way; some the other way! • Tesnière had them point from head to dependent… • Usually add a fake ROOT so every word is a dependent of precisely 1 other node

Christopher Manning

Dependency Grammar/Parsing History

• The idea of dependency structure goes back a long way • To Pāṇini’s grammar (c. 5th century BCE) • Basic approach of 1st millennium Arabic grammarians • Constuency is a new-fangled invenon • 20th century invenon (R.S. Wells, 1947) • Modern dependency work oen linked to work of L. Tesnière (1959) • Was dominant approach in “East” (Russia, China, …) • Good for free-er word order languages • Among the earliest kinds of parsers in NLP, even in the US: • David Hays, one of the founders of U.S. computaonal linguiscs, built early (ﬁrst?) dependency parser (Hays 1962) Christopher Manning Relaon between phrase structure and dependency structure

• A dependency grammar has a noon of a head. Oﬃcially, CFGs don’t. • But modern linguisc theory and all modern stascal parsers (Charniak, Collins, Stanford, …) do, via hand-wrien phrasal “head rules”: • The head of a Noun Phrase is a noun/number/adj/… • The head of a Verb Phrase is a verb/modal/…. • The head rules can be used to extract a dependency parse from a CFG parse

• The closure of dependencies give constuency from a dependency tree • But the dependents of a word must be at the same level (i.e., “ﬂat”) – there can be no VP! Christopher Manning

Dependency Condioning Preferences

What are the sources of informaon for dependency parsing? 1. Bilexical aﬃnies [issues à the] is plausible 2. Dependency distance mostly with nearby words 3. Intervening material Dependencies rarely span intervening verbs or punctuaon 4. Valency of heads How many dependents on which side are usual for a head?

ROOT Discussion of the outstanding issues was completed . Christopher Manning

Dependency Parsing

• A sentence is parsed by choosing for each word what other word (including ROOT) that it is a dependent of.

• Usually some constraints: • Only one word is a dependent of ROOT • Don’t want cycles A → B, B → A • This makes the dependencies a tree • Final issue is whether arrows can cross (non-projecve) or not

ROOT I ’ll give a talk tomorrow on bootstrapping 9 Christopher Manning

Methods of Dependency Parsing

1. Dynamic programming (like in the CKY algorithm) You can do it similarly to lexicalized PCFG parsing: an O(n5) algorithm Eisner (1996) gives a clever algorithm that reduces the complexity to O(n3), by producing parse items with heads at the ends rather than in the middle 2. Graph algorithms You create a Minimum Spanning Tree for a sentence McDonald et al.’s (2005) MSTParser scores dependencies independently using a ML classiﬁer (he uses MIRA, for online learning, but it can be something else) 3. Constraint Sasfacon Edges are eliminated that don’t sasfy hard constraints. Karlsson (1990), etc. 4. “Determinisc parsing” Greedy choice of aachments guided by good machine learning classiﬁers MaltParser (Nivre et al. 2008) DP for generave dependency grammars Christopher Manning Probabilistic dependency grammar: generative model

$ λw ρw 1. Start with left wall $ 0 0

2. Generate root w0 w0

3. Generate left children w-1, w w w-2, ..., w-ℓ from the FSA λw0 -1 1

4. Generate right children w1, w-2 w w2, ..., wr from the FSA ρw0 2 ... 5. Recurse on each wi for i in {- ... ℓ, ..., -1, 1, ..., r}, sampling αi λw-ℓ (steps 2-4) w-ℓ 6. Return αℓ...α-1w0α1...αr wr

w-ℓ.-1

These 5 slides are based on slides by Jason Eisner and Noah Smith Christopher Manning

Naïve Recognition/Parsing

goal O(n5) p combinations p c r i j k 0 n goal

takes

It takes

takes to

It takes two to tango

It takes two to tango Christopher Manning Dependency Grammar Cubic Recognition/Parsing (Eisner & Satta, 1999)

• Triangles: span over words, where tall side of triangle is the head, other } side is dependent, and no non-head words expecting more dependents

• Trapezoids: span over words, where larger side is head, smaller side is } dependent, and smaller side is still looking for dependents on its side of the trapezoid Christopher Manning Dependency Grammar Cubic Recognition/Parsing (Eisner & Satta, 1999)

A triangle is agoal head with some left (or right) subtrees. One trapezoid per dependency.

It takes two to tango Christopher Manning

Cubic Recognition/Parsing (Eisner & Satta, 1999)

goal O(n) combinations 0 i n

O(n3) combinations i j k i j k

O(n3) combinations i j k i j k Gives O(n3) dependency grammar parsing Graph Algorithms: MSTs

17 Christopher Manning McDonald et al. (2005 ACL) Online Large-Margin Training of Dependency Parsers

• One of two best-known recent dependency parsers • Score of a dependency tree = sum of scores of dependencies • Scores are independent of other dependencies • If scores are available, parsing can be solved as a minimum spanning tree problem • Chiu-Liu-Edmonds algorithm • One then needs a score for dependencies Christopher Manning McDonald et al. (2005 ACL): Online Large-Margin Training of Dependency Parsers

• Edge scoring is via a discriminave classiﬁer • Can condion on rich features in that context • Each dependency is a linear funcon of features mes weights • Feature weights were learned by MIRA, an online large-margin algorithm • But you could use an SVM, maxent, or a perceptron • Features cover: • Head and dependent word and POS separately • Head and dependent word and POS bigram features • Words between head and dependent • Length and direcon of dependency

Greedy Transion-Based Parsing

MaltParser Christopher Manning MaltParser [Nivre et al. 2008]

• A simple form of greedy discriminave dependency parser • The parser does a sequence of boom up acons • Roughly like “shi” or “reduce” in a shi-reduce parser, but the “reduce” acons are specialized to create dependencies with head on le or right • The parser has: • a stack σ, wrien with top to the right • which starts with the ROOT symbol • a buﬀer β, wrien with top to the le • which starts with the input sentence • a set of dependency arcs A • which starts oﬀ empty • a set of acons Christopher Manning

Basic transion-based dependency parser

Start: σ = [ROOT], β = w1, …, wn , A = ∅

1. Shi σ, wi|β, A è σ|wi, β, A

2. Le-Arcr σ|wi, wj|β, A è σ, wj|β, A∪{r(wj,wi)}

3. Right-Arcr σ|wi, wj|β, A è σ, wi|β, A∪{r(wi,wj)}

Finish: β = ∅

Notes: • Unlike the regular presentaon of the CFG reduce step, dependencies combine one thing from each of stack and buﬀer Christopher Manning

Acons (“arc-eager” dependency parser)

Start: σ = [ROOT], β = w1, …, wn , A = ∅

1. Le-Arcr σ|wi, wj|β, A è σ, wj|β, A∪{r(wj,wi)}

Precondion: r’ (wk, wi) ∉ A, wi ≠ ROOT

2. Right-Arcr σ|wi, wj|β, A è σ|wi|wj, β, A∪{r(wi,wj)}

3. Reduce σ|wi, β, A è σ, β, A

Precondion: r’ (wk, wi) ∈ A

4. Shi σ, wi|β, A è σ|wi, β, A

Finish: β = ∅

This is the common “arc-eager” variant: a head can immediately take a right dependent, before its dependents are found Christopher Manning

1. Le-Arcr σ|wi, wj|β, A è σ, wj|β, A∪{r(wj,wi)} Precondion: (wk, r’, wi) ∉ A, wi ≠ ROOT 2. Right-Arcr σ|wi, wj|β, A è σ|wi|wj, β, A∪{r(wi,wj)} Example 3. Reduce σ|wi, β, A è σ, β, A Precondion: (wk, r’, wi) ∈ A 4. Shi σ, wi|β, A è σ|wi, β, A Happy children like to play with their friends .

[ROOT] [Happy, children, …] ∅ Shi [ROOT, Happy] [children, like, …] ∅

LAamod [ROOT] [children, like, …] {amod(children, happy)} = A1

Shi [ROOT, children] [like, to, …] A1

LAnsubj [ROOT] [like, to, …] A1 ∪ {nsubj(like, children)} = A2

RAroot [ROOT, like] [to, play, …] A2 ∪{root(ROOT, like) = A3

Shi [ROOT, like, to] [play, with, …] A3

LAaux [ROOT, like] [play, with, …] A3∪{aux(play, to) = A4

RAxcomp [ROOT, like, play] [with their, …] A4∪{xcomp(like, play) = A5

Christopher Manning

RAxcomp [ROOT, like, play] [with their, …] A4∪{xcomp(like, play) = A5

RAprep [ROOT, like, play, with] [their, friends, …] A5∪{prep(play, with) = A6

Shi [ROOT, like, play, with, their] [friends, .] A6

LAposs [ROOT, like, play, with] [friends, .] A6∪{poss(friends, their) = A7

RApobj [ROOT, like, play, with, friends] [.] A7∪{pobj(with, friends) = A8

Reduce [ROOT, like, play, with] [.] A8

Reduce [ROOT, like, play] [.] A8

Reduce [ROOT, like] [.] A8

RApunc [ROOT, like, .] [] A8∪{punc(like, .) = A9

You terminate as soon as the buﬀer is empty. Dependencies = A9

1. Left-Arcr σ|wi, wj|β, A è σ, wj|β, A∪{r(wj,wi)} Precondition: (wk, r’, wi) ∉ A, wi ≠ ROOT 2. Right-Arc σ|w, w|β, A è σ|w|w, β, A∪{r(w,w)} Example r i j i j i j 3. Reduce σ|wi, β, A è σ, β, A Precondition: (wk, r’, wi) ∈ A 4. Shift σ, wi|β, A è σ|wi, β, A