Parsing English with a Link Grammar
Daniel D. Sleator* and Davy Temperleyt * School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 email: sleator©cs . emu . edu t Music Department Columbia University New York, NY 10027 email: dt3©cunixa. cc . columbia. edu
Abstract We define a new formal grammatical system called a link grammar. A sequence of words is in the language of a link grammar if there is a way to draw links between words in such a way that (1) the local requirements of each word are satisfied, (2) the links do not cross, and (3) the words form a connected graph. We have encoded English grammar into such a system, and written a program (based on new algorithms) for efficiently parsing with a link grammar. The formalism is lexical and makes no explicit use of constituents and categories. The breadth of English phenomena that our system handles is quite large. A number of sophisticated and new tecµniques were used to allow efficient parsing of this very complex grammar. Our program is written in C, and the entire system may be obtained via anonymous ftp. Several other researchers have begun to use link grammars in their own research. 1 Introduction Satisfaction: The links satisfy the linking re quirements of each word in the se Most sentences of most natural languages have quence. the property that if arcs are drawn connecting each pair of words that relate to each other, then theknown arcs will not cross [10, p. 36] . planarThis ity,well phenomelinknon, grammar whichs we call is the basis of our new formal lan guage system. words A link grammar consists of a set of ( the terminallinking symbols requir ofement. the grammar), each of which hassentencea A sequence of words is a of the language definedlinks by the gram mar if there exists a way to draw among the The linkingdictionary. requirements of each word are con words so as to satisfy the following conditions: tained in a To illustrate the linking requirements, the followingdiagram a, the, shows cat, snakea sim, Planarity: The links do not cross (when pleMary, dictionary ran, forchased the words. drawn above the words). and The linking requirement Connectivity: The links suffice to connect all the of each word is represented by the diagram above words of the sequence together. the word. 277 278 DANIEL SLEATOR AND DAVY TEMPERLEY
linkage. called a From now on we'll use simpler diagrams to illustrate linkages. Here is thethe sim cat plifiedchased forma snake of the diagram showing that a cat Mary is part of this language: the snake rYs� the cat chased a snake
ran chased We have a succinct, computer-readable nota tion for expressing the dictionary of linking re quirements. The following dictionary encodes the connector.Each of the intricately shaped labeled boxes is a A connector is satisfied by match linking requirements of the previous example. ing it to a compatible connector ( one with the words formula appropriate shape, facing in the opposite direc a the D+ - or tion). Exactly one of the connectors attached to snake cat D- & (0 S+) 0- or S+ a given black dot must be satisfiedcat ( the others, if Mary s- any, must not be used). Thus, requir�s a D ran connector to its left, and either an 0 connector to chased S- & 0+ its left or a S connector to its right. Plugging a pair of connectors together corresponds to draw The linking requirement for each word is ex ing a link between that pair of words. pressedor, as a formula involving the operators &, and parentheses, and connector names. The The following diagram shows how the Thelinking cat requirements are satisfiedin the sentence + or - suffix on a connector name indicates the chased a snake. direction (relative to the word being defined) in which the matching connector (if any) must lie. The & of two formulas is orsatisfied by satisfying both the formulas. The of two formulas re quires that exactly one of its formulas be satis the cat chased a snake fied. The order of the arguments of an & operator is significant. The farther left a connector is in (The unused connectors haveMary been chased suppressed the cat, the expression, the nearer the word tocat whic h it here.)the It catis easy ran to see that connects must be. Thus, when using as an and are also sentencesthe ofMary this chasedgram object, its determiner (to which it is connected mar.cat The sequence of words: with its D- connector) must be closer than the is not in this language. Any attempt to sat verb (to which it is connected with its 0- connec isfy the linking requirements leads to a violation tor). of one of the three rules. Here is one attempt: We can roughly divide our work on link gram mars into three parts: the link grammar formal ism and its properties, the construction of a wide coverage link grammar for English, and efficient cat algorithms and techniques for parsing link gram the Mary chased ran Mary, cat ran chased mars. We now touch briefly on all three of these Similarly and are not aspects. part of this language. Link grammars are a new12 and elegant context A set of links that prove that a sequence of free grammatical formalism , and have a unique words is in the language of a link grammar is combination of useful properties: 1 Link grammars resemble dependency grammars and categorial grammars. There are also many significant differ ences. Some light is shed on the relationship in section 6. 2The proof of the context-freeness of link grammars is not included in this paper, but appears in our technical PARSING ENGLISH WITH A LINK GRAMMAR 279
1. In a link grammar each word of the lexicon is be seen to emerge as contiguous connected col given a definitiondescribing how it can be used lections of words attached to the rest of the in a sentence. The grammar is distributed sentence by a particular type of link. For ex among the words. Such a system is said to ample in the dictionary above, s links always be lexical. This has several important advan attach a noun phrase ( the connected collection tages. It makes it easier to construct a large of words at the left end of the link) to a verb grammar, because a change in the definition of (on the right end of the link). 0 links work in a word only affects the grammaticality of sen a similar fashion. In these cases the links of a tences involving that word. The grammar can sentence can be viewed as an alternative way easily be constructed incrementally. Further of specifying the constituent structure of the more, expressing the grammar of the irregular sentence. On the other hand, this is not the verbs of English is easy - there's a separate way we think about link grammars, and we see definition for each word. no advantage in taking that perspective. Another nice feature of a lexical system is that it allows the construction of useful prob abilistic language models. This has led re Our second result is the construction of a link searchers to construct lexical versions of other grammar dictionary forEnglish. The goal we set grammatical systems, such as tree-adjoining for ourselves was to make a link grammar that grammars [13] . Lafferty and the present au can distinguish, as accurately as possible, syntac thors have also constructed such a probabilis tically correct English sentences from incorrect tic model for link grammars [11]. ones. We chose a formal or newspaper-style En glish as our model. The result is a link grammar 2. Unlike a phrase structure grammar, after pars of roughly eight hundred definitions (formulas) ing a sentence with a link grammar words that and 25000 words that captures many phenomena are associated semantically and syntactically of English grammar. It handles: noun-verb agree are directly linked. This makes it easy to ment, questions, imperatives, complex and irreg enforce agreement, and to gather statistical ular verbs, many types of nouns, past- or present information about the relationships between participles in noun phrases, commas, a variety words. of adjective types, prepositions, adverbs, relative 3. In English, whether or not a noun needs a de clauses, possessives, coordinating conjunctions, terminer is independent of whether it is used as unbounded dependencies, and many other things. a subject, an object, or even if it is part of a The third result described in this paper is a prepositional phrase. The algebraic notation program for parsing with link grammars. The we developed for expressing a link grammar program reads in a dictionary (in a form very takes advantage of this orthogonality. Any lex similar to the tables above) and will parse sen ical grammatical system, if it is to be used by tences according to the given grammar. The pro a human, must have such a capability. In our gram does an exhaustive search - it finds every current on-line dictionary the word cat can be way of parsing the given sequence with the given used in 369 different ways, and for time this link grammar. It is basedon our own O(n3 ) algo number is 1689. A compact link grammar for rithm ( n is the number of words in the sentence to mula captures this large number of possibil be parsed). The program also makes use of sev ities, and can easily be written and compre eral very effective data structures and heuristics hended by a human being. to speed up parsing. The program is comfortably 4. Another interesting property of link grammars fast - parsing typical newspaper sentences in a is that they have no explicit notion of con few seconds on a modern workstation. stituents or categories. In most sentences Both our program (written in ANSI-C) and parsed with our dictionaries, constituents can our dictionary are available via anonymous ftp report [14]. Note that context-free systems can differ in many ways, including the ease with which t_he same grammar can be expressed, the efficiency with which the same grammar can be parsed, and the usefulness of the output of the parser forfurther processing. 280 DANIEL SLEATOR AND DAVY TEMPERLEY
3 It is through the internet. Having the program avail Onelikely example that John is thewill non-ref go erential use ofThe it :cat" is able for experimentation may make it easier to likely that John will go is correct, but understand this paper. is wrong. It is possible - but awkward - to distinguish between these with The organization of this paper a link grammar. To deal with this ( and a num ber of other phenomena), we extendedpost-process the baor In section 2 we define link grammars more for sic link grammar formalism with a mally and explain the notation and terminology that begins with a linkage, analyzes its structure, used throughout the rest of the paper. In sec and determines if certain conditions are satisfied. tion 3 we describe the workingsO(n3 ) of a small link This allows the system to correctly judge a num grammar for English. Our algorithm is de ber of subtle distinctions (including that men scribed in section 4, and the data structures and tioned here). heuristics that make it run fast are described in section 5. In section 6 we explain the relationship 2 Notation and terminology between link grammars, dependency syntax,and categorial grammars. We show how to automat 2.1 Meta-rules ically construct a link grammar for a given cate gorial grammar. This construction allows our ef The link grammar dictionary consists of a collec ficientparsing algorithms and heuristics to be ap tion of entries, each of which defines the linking plied to categorial grammars. Section 7 mentions requirements of one or more words. Theseformula re several other research projects that are based on quirementsconnectors are specified by means of a link grammars. of combined by the binary associative Space limitations prevent us from presenting operators & and or. Presidence is specified by details of a number of other aspects of our work. means of parentheses. Without loss of general The following paragraphs mention a few of these. ity we may assume that a connector is simply a More details on all of these[14) .matters are contained character string ending in + or -. in our technical report When a link connects to a word, it is associ There are a number of common English phe ated with one of the connectorssatisfy of the formula of nomena that are not handled by our current sys that word, and it is said to that connector. tem. Our technical report contains a list of these, No two links may satisfy the same connector. The along with the reason forthis state of affairs. The connectors atmatch opposite, ends of a link must have reasons range from the fact that ours is a pre names that and the one on the left must liminary system to the fact that some phenom end in + and the one on the right must end in -. ena simply do not fit well into the link grammar In basic link grammars, two connectors match if framework. and and only if their strings are the same (up to but Coordinating conjunctions such as pose not including the final + or -) . A more general a problem for linkThe grammars. dog chased This and is bitbecause Mary form of matching will be introduced later. in a sentence like dog The connectors satisfied by the links must therebit should chasedlogically beMary links . between both serve to satisfy the whole formula. We define the and and and Such links would notion of satisfying a formularecursively. To sat cross. We have devised a scheme that handles the isfy the & of two formulas, both formulas must be vast majority of uses of such conjunctions and in satisfied. To satisfy the or of two formulas,no one corporated it into our program. The existence of the formulasmust be satisfied, and connec of such a conjunction in a sentence modifies the tors of the other formula may be satisfied. It is grammar of the words in it. The same parsing sometimes convenient to use the empty formula algorithm is then used on the resulting modified (" ()"), which is satisfied by being connected to grammar. no links. sentence Certain other constructs are difficult to han A sequence of words is a of the lan dle only using the basic link grammar framework. guage defined by the grammar if there exists a 3The directory is /usr/sleator/public on the host spade .pc.cs.cmu . edu (128.2.209.226). Our technical re ports [14, 11] are also available there. PARSING ENGLISH WITH A LINK GRAMMAR 281 way to draw links among the words so as to sat the direction is implicit in the form of the dis meta isfy each word's formula, and the following junct. rules: To satisfy the linking requirements of a word, Planarity: one of its disjuncts must be satisfied ( and no links The links are drawn above the sen may attach to any other disjunct). To satisfy a tence and do not cross. disjunct all of its connectors must be satisfied by Connectivity: 1 , 2 , ••• The links suffice to connect all appropriate links. The words to which L L the words of the sequence together. are linked are to the· left of the current word, and are monotonically increasing in distance from the Ordering: 1 , , When the connectors of a formulaare current word. The words to which R R2 ••• are traversed from left to right, the words to linked are to the right of the current word, and which they connect proceed from near to are monotonically increasing in distance from the far. In other words, consider a word, and current word. · consider two links connecting that word to It is_ easy to see how to translate a link gram words to its left. The link connecting the mar in disjunctive form to one in standard form. nearer word (the shorter link) must satisfy This can be done simply by rewriting each dis a connector appearing to the left (in the for junct as mula) of that of the other word. Similarly, a link to the right must satisfy a connector to the left (in the formula) of a longer link to the right. and combining all the disjuncts together with the Exclusion: or operator to make an appropriate formula. No two links may connect the same It is also easy to translate a formulainto a set pair of words. of disjuncts. This is done by enumerating all ways that the formula can be satisfied. For example, 2.2 Disjunctive form the formula: (A- D- (B+ (0- S+) The use of formulas to specify a link grammar or ()) & & or ()) & or dictionary is convenient for creating natural lan guage grammars, but it is cumbersome for mathe corresponds to the following eight disjuncts: matical analysis of link grammars, and in describ ((A,D) (S,B)) ing algorithms for parsing link grammars. We ((A,D,0) (B)) therefore introduce a different way of expressing ((A,D) (S)) disjunctive form. a link grammar called ((A,D,0) ()) In disjunctive form,each word of the grammar ((D) (S,B)) disjuncts has a set of associated with it. Each dis ((D,O) (B)) junct corresponds to one particular way of satis ((D) (S)) ((D,O) ()) fying the requirements of a word. A disjunct con sists of two ordered lists of connector names: the left list right list. and the The left list contains 2.3 Our dictionary language connectors that connect to the left of the current word ( those connectors end in -) , and the right To streamline the difficult process of writing the list contains connectors that connect to the right dictionary, we have incorporated several other of the current word. A disjunct will be denoted: features to the dictionary language. Examples ,Lm ) of all of these features can be found in section 3-. ((L1 ,L2,. · · (Rn,Rn 1 , · · · ,R1 )) - It is useful to consider connector matching 1 , L ••• , Lm Where L 2 , are the connectors that rules that are more powerful than simply requir n must connect to the left, and R1 , R2, ..., R are ing the strings of the connectors to be identical. connectors that must connect to the right. The The most general matching rule is simply a ta number of connectors in either list may be zero. ble - part of the link grammar - that specifies· The trailing + or - may be omitted from the con all pairs of connectors that match. The resulting nector names· when using disjunctive form, since link grammar is still context-free. DANIEL SLEATOR AND DAVY TEMPERLEY 282
In the dictionary presented later in this pa {©A-} &: Ds- per, and in our larger on-line dictionary, we use a &: {©M+ or (C+ &: Bs+)} matching rule that is slightly more sophisticated &: (J- or 0- or ({C- or CL-} &: Ss+) or Sis-) ; dogs cats parks bones sticks : than simple string matching. We shall now de {©A-} &: {Dm-} scribe this rule. &: {©M+ or (C+ &: Bp+)} A connector name begins with one or more &: (J- or 0- or ({C- or CL-} &: Sp+) or Sip-) ; upper case letters followed by a sequence of lower has : case letters or *S. Each lower case letter ( or *) (Sis+ or Ss- or (Z- &: B-)) subscript. is a To determine if two connectors &: (((B- or 0+) &: {©EV+}) or T+) ; match, delete the trailing + or -, and append an did: infinite sequence of *S to both connectors. The (SI+ &: I+) connectors match if and only if these two strings or ((S- or (Z- &: B-)) match under the proviso that * matches a lower &: (((B- or 0+) &: {©EV+}) or I+)); can may will must : case letter ( or *). (SI+ or S- or (Z- &: B-)) &: I+; For example, S matches both Sp and Ss, but is was : Sp does not match Ss. Similarly, D*u, matches (Ss- or (Z- &: Bs-) or Sis+) Dmu and Dm, but not Dmc. All four of these con &: (AI+ or O+ or B- or V+ or Mp+) ; nectors match Dm. "((A- B+) touch chase meet : The formula & or ())" is satis (Sp- or (Z- & Bp-) or I-) B+, fied either by using both A- and or by using &: (0+ or B-) &: {©EV+}; neither of them. Conceptually, then, the the ex touches chases meets: "(A+ B+)" pression & is optional. Since this oc (Ss- or (Z- &: Bs-)) &: (0+ or B-) &: {©EV+}; curs frequently, we denote it with curly braces, as touched chased met : {A+ B+}. follows: & (V- or M- or ((S- or (Z- &: B- ) or T-) &: (0+ or B-) )) It is useful to allow certain connectors to be &: {©EV+}; able to connect to one or more links. This makes touching chasing meeting: it easy, for example, to allow any number of ad (GI- or M-) &: (0+ or B-) &: {©EV+}; jectives to attach to a noun. We denote this by die arrive : putting an "©" before the connector name, and multi-connector. (Sp- or (Z- &: Bp-) or I-) &: {©EV+}; call the result a dies arrives : en Our dictionaries consist of a sequence of (Ss- or (Z- &: Bs-)) &: {©EV+}; tries, each of which is a list of words separated died arrived: by spaces, followed by a colon, followed by the (S- or (Z- &: B-) or T-) &: {©EV+}; formula defining the words, followed by a semi dying arriving : (GI- or M-) &: {©EV+}; colon. with in by : J+ &: (Mp- or EV-) ; 3 An example big black ugly: A+ or (AI- &: {©EV+}); who : Perhaps the best way to understand how to write (C- &: {Z+ or CL+}) or B+ or Ss+; a link grammar for English is to study an ex ample. The following dictionary does not cover 3.1 Some Simple Connectors the complete grammar of the words it contains, �ut it does handle a number of phenomena: verb We develop an explanation of how this works in noun agreement, adjectives, questions, infinitives, stages. Let's first restrict our attention to the prepositional phrases, and relative clauses. followingconnectors: S, O, A, D. (Imagine the dic the : D+; tionary with all of the other connectors removed.) a: Ds+; The Sis used to connect a noun to its verb. The 0 John Mary : connector is used to connect a verb to its object. J- or 0- or (({C- or CL-} &: S+) or SI-) ; The A connector is used to connect an adjective dog cat park bone stick : to its noun. The Dis for connecting a determiner PARSING ENGLISH WITH A LINK GRAMMAR 283 to its noun. Notice that this connector is omitted from proper nouns, is optional on plural nouns, and is mandatory on singular nouns. Also notice (os ss zv�� that the subscripts allow the to act as the de y y- terminer for both plural and singular nouns, but the dog arrived with a bone a can only work with the singular nouns. Sim ilarly, the S connector is subscripted to ensure verb-noun agreement. The ordering of the terms in these expressions (°�� is often important. For example, the fact that on the dog with a bone arrived nouns, the A- occurs to the left of the D- means that the adjective must be closer to the noun than Notice that, aswith A- connectors on nouns, a the determiner. © is used for M- connectors on nouns and EV- con Here are some judgements that can be ren nectors on verbs, allowing multiple prepositional dered by what we have described so far: phrases, such as John chased a dog in the park with a stick.
3.3 Participles The M- connector on chased allows it to act as a participle phrase modifying a noun, as shown in these examples.
the ugly black dog chased a big cat lf �s-@ [°•hEV1:� l rs� the dog chased in the park arrived dogs died
° ,s� / �� dogs chase cats the dog chased in the park arrivedl The I connector is used for infinitives, as in: *a dog chase a cat
*black the dog died S I *a/*the Mary chased the cat / Y � *a dogs died John must meet Mary *dog died Notice that the I connector is an alternative to the S connector on plural verb forms. Thus we 3. 2 Prepositions take advantage of the fact that plural verb forms are usually the same as infinitive forms, and in The J, M and EV connectors allow prepositional clude them both in a single dictionary entry. phrases. The J connector connects a preposition In a similar way, the T connector is used for to its object. Notice that in nouns, the J- is an pastparticiples. Pastparticiples have a T-; forms alternative to the 0-. This means that a noun of the verb have have a T+. The GI connector is cannot both be an object of a verb and of a prepo used for present participles. Present participles sition. The M connector is used when a preposi have a GI- connector; formsof the verb have a tional phrase modifies a noun and the EV connec GI+. The AI connector is used for predicativebe ad tor is used when a prepositional phrase modifies a jectives. Adjectives have a AI - connector; forms verb. The following two examples illustrate this: of have a AI+ connector. be 284 DANIEL SLEATOR AND DAVY TEMPERLEY
3.4 Questions S+, SI-, O+, and J+ connectors, meaning that one of these connectors must be used whether or not The SI connector is used for questions where the noun takes a relative clause. Verbs have a B there is subject-verb inversion. On nouns SI- is connector which is orred with their S- connec an alternative to S+, and on invertible verbs (is, tors; if a verb is in a subject-type relative clause, has, did, must, etc.) SI+ is an alternative to S-. it may not make an S connection as well. This allows For subject-type relative clauses, the relative pronoun who is mandatory. For this purpose, �x� verbs have a Z- connector anded with their B Who Z+ did John chase the dog connector. has a connector; thereforeit can fulfill this need. However, it also has a C Wh- questions work in various different ways; connector anded with its Z+ connector; this must only questions involving who will be discussed connect back to the C+ connector on nouns. This here. For subject-type questions, where who is allows the following: substituting for the subject, who simply has an S+ connector. This allows
(°-��� ss� the dog who chased John diedI who1 chased the dog For object-type relative clauses, the same B+ who For object-type questions, where is sub connector on nouns is used. However, this time B stituting for the object, the connector is used. it connects to the other B- connector on verbs, B- Transitive verbs have connectors as an alter the one which is orred with the O+ connector and native to their O+ connectors. Wh o hasa B+ con which is also used forobject-type wh- questions. nector. This allows In this case, the relative pronoun who is op tional. Notice that nouns have optional C+ and CL- connectors which are anded with their S+ con nectors. These are used when the noun is the who did John chase subject of an object-type relative clause. When who is not present, the C+ connector on the an The�7 following incorrect sentences are rejected: tecedent noun connects directly to the C- on the subject of the relative clause: *Did John chase *Who did John chase Mary *John did Mary chase *Chased John Mary (°"�� The following incorrect construction is ac the dog John chased died cepted. In our on-line system, post-processing is who used to eliminate this. When is present, the C+ on the antecedent connects to the C- on who; this forces the CL+ to *Who John chased connect to the CL- on the subject of the clause:
3.5 Relative Clauses For subject-type relative clauses, where the an r°"�Y� tecedent is acting as the subject of the clause, a B the dog who John chased diedI connector serves to connect the noun to the verb of the relative clause. Nouns have a B+ connector. This system successfully rejects the following Notice that this is optional; it is also &ed with the incorrect sentences: PARSING ENGLISH WITH A LINK GRAM MAR 285
*The dog chased cats died *The dog who chase cats died *The dog who John chased cats died *The dog John chased cats died *The dog who chased died
The following incorrect constructions are ac cepted, but can be weeded out in post-processing: L R Here the square boxes above the words L and R represent a data structure node corresponding *The dog did John chase died to the word. The rectangular box above each of *The dog who John died Mary chased these represents one of the (possibly many) dis died juncts for the word. The small squares pointed to by the disjuncts represent connectors. How do we go about extending the partial so lution into the region strictly between L and R? 4 The algorithm (This region will be denoted (L, ... , R).) First of all, if there are no words in this region (i.e. L = R + 1) then the partial solution we've built Our algorithm forparsing link grammars is based is certainly invalid if either l =/= NIL or r =/= NIL. If on dynamic programming. Perhaps its closest rel l = r = NIL then this region is ok, and we may ative in the standard literature is the dynamic proceed to construct the rest of the solution. programming algorithm for finding an optimal triangulation of a convex polygon [2, p. 320]. It Now suppose that the region between L and tries to build up a linkage (which we'll call a so R contains at least one word. In order to attach lution in this section) in a top down fashion: It the words of this region to the rest of the sen tence there must be at least one link either from will never add a link (to a partial solution) that is above a link already in the partial solution. L to some word in this region, or from R to some word in this region ( since no word in this region The algorithm is most easily explained by can link to a word outside of the [L, ..., R] range, specifying a data structure for representing dis and something must connect these words to the d juncts. A disjunct has pointers to two linked rest of the sentence). lists of connectors. These pointers are denoted Since the connector l' has already been used left[d] and right[d]. If c is a connector, then in the solution being constructed, this solution next[ c] will denote the next connector after c in must use the rest of the connectors of the dis its list. The next fieldof the last pointer of a list junct in which l' resides. The same holds for r'. has the value NIL. The only connectors of these disjuncts that can For example, suppose the disjunct d = ( (D, 0) be involved in the (L, ..., R) region are those in ()) (using the notation of section 2). Then left[d] the lists beginning with l and r. (The use of any would point to the connector □, next[left[d]] would other connector on these disjuncts in this region point to the connector D, and next[next[left[d]]] 'would violate the ordering requirement.) In fact, would be NIL. Similarly, right[d] = NIL. all of the connectors of these lists must be used To give some intuition of how the algorithm in this region in order to have a satisfactorysolu works, consider the situation after a link has been tion. proposed between a connector l' on word L and Suppose, for the moment, that l is not NIL. a connector r' on word R. (The words of the We know that this connector must link to some sequence to be parsed are numbered from O to disjunct on some word in the region (L, . . . , R). N - 1.) For convenience, we define l and r to be (It can't link to R because of the exclusion rule.) next[l'] and next[r'] respectively. The situation is The algorithm tries all possible such words and shown in the following diagram: disjuncts. Suppose it finds a word W and a dis- 286 DANIEL SLEATOR AND DAVY TEMPERLEY
junctleft[ d on W such that the connector l matches The running time is now bounded by the num d]. We can now add this link to our partial ber of different possible recursive calls multiplied solution. by the time used by each call. A recursive call is The situation is shown in the following dia completelyl r. determined by specifying theL pointersR.) gram. and (These uniquely determine and The cost of a given call is bounded by the total number of disjunctsd in the sequence of words. If we let be the number of disjuncts and c be theO(c number2 d). of connectors, then the runningd = O(N) time is c = O(N),For a fixed link grammar, 3 and so the running time is O(N ). L w R Our technical reports describe this algorithm in more detail.[14], They contain pseudo-code for the How do we determine if this partial solution algorithm(14] an argument for it's correctness can be extended to a full solution? We do this by and an elegant recurrence[11]. for the number of solving two problems similar to the problem we linkages of a sentence started with. In particular, we ask if the(L, solution..., W) After the algorithm above was implemented, can be extended to the word range next[l] we were interested in seeing how well it would using the connector lists beginning with work on sentences taken from newspapers and and next[left[d]]. We also ask if (theW, solutionR) can other natural sources. It quickly became clear be extended to the word range rig...ht ,[d] usingr. that something else was needed to make the al the connector lists beginning with and gorithm run faster on long sentences. Notice that in the latter case, the problem we are solving seems superficially different: the bound 5 Speeding it up ary words have not already been connected to gether by a link. This difference is actually of (L R As pointed out in the introduction, in a link gram no consequenceL W) because the pair of links to and to play the role that a direct link from mar dictionary with significant coverage of En W R glish grammar the number of disjuncts on many (W,to R)would play: ( 1) they separate the region d ... , fromall the other words, and (2) they words gets rather large. Thus, the constant in serve to connect the words W and R together. the analysis at the end of the last section is quite large. We devised and implemented several time We need to consider one other possibility. saving schemes that run in conjunction with the That is that there might be a solution with a link L W and algorithm of the previous section. betweenW R.words and a link between words and (This results in a solution where the 5.1 Pruning word/link graph is cyclic.) The algorithm han dles this possibilityright[d ]by alsor. attempting to form a Our firstapproach is based on the followingobser link between and If these two match, it vation: In any particular sequence of words to be does a third recursive call, solving a third problem parsed, most of the disjuncts are irrelevant forthe analogous to our original(W, problem.R) In this prob simple reason that they contain a connector that lem the word range is ... , and the connec does not match any other connector on a word in tor lists to be satisfied begin with next[right[d]] left[r]. the sequenceW . To be mored precise, suppose that and a word has a disjunct with a connector C in A very similarNIL. analysis suffices to handle the its right list. If no word to the right of W has a case when l is connector (pointingd to the left) that matches C, The algorithm described has an exponential then the disjunct cannot be in any linkage. This worst-case running time as a function of N, the disjunct can therefore be deleted without chang number of words in the sequence to be parsed. ing the set pruningof linkages. step .Deleting pruning such a disjunct This can easily be transformed into an efficientmem is called a consists of re dynamicoization programming algorithm by using peating the pruning step until it can no longer be ((2, p. 312]). applied. PARSING ENGLISH WITH A LINK GRAMMAR 287
The set of disjuncts left ( after pruning is com the connectors of d matches nothing in S, then plete) is independent of the order in which the we apply the pruning step to d ( we remove d). steps are applied. (The pruning operation has the Each right connector of each remaining disjunct Church-Rosser property.) We thereforechoose an of W is now incorporated into the set S. This ordering that can be efficiently implemented. It completes the processing of word W. would be ideal if we could achieve a running time The function computed by this left-to-right for pruning that is linear in the number of connec pass is idempotent, which is another way of say tors. The scheme we propose satisfies no useful ing that doing the operation twice in a row will a-priori bound on its running time, but in prac be the same as doing it once. Therefore if ( as we tice it appears to run in linear time. alternate left-to-right and right-to-left passes) a A series of sequential passes through the words pass ( after the first one) does nothing, then all is made, alternately left-to-right and right-to-left. further passes will do nothing. This is how the The two types of passes are analogous, so it algorithm decides when to stop. suffices to describe the left-to-right pass. The The data structure used for the set S is sim pass processes the words sequentially, starting ply a hash table, where the hash function only with word 1. Consider the situation after words uses the initial upper-case letters of the connec 1, ..., W - l have been processed. A set S of tor name. This ensures that if two connectors get connectors has been computed. This is the set of hashed to different locations, then they definitely connectors that exists on the right lists of the dis don't match. juncts of words 1, ..., W - l that have not been Although we know of no non-trivial bound on deleted. To process word W, we consider each the number of passes, we have never seen a case d W c disjunct of in turn. For each connector requiring more than five. Table 1 shows a typ on the left list of we search the set to see if ical example of the reduction in the number of d, c.S it contains a connector that matches If one of disjuncts achieved by pruning.
Initial: 3 12 18 391 296 9 10 3 81 391 20 423 104 391 after L-+R 3 8 11 67 160 6 10 3 81 163 8 381 25 357 after R-+L 3 6 5 25 21 3 3 3 25 25 3 25 4 12 after L-+R 3 6 5 25 21 3 3 3 25 21 3 21 3 8 after R-+L 3 6 5 25 21 3 3 3 25 21 3 21 3 8 After P.P.: 2 6 1 13 2 2 2 2 2 6 1 4 2 1
Now this Table 1: This table shows the number of disjuncts remaining on each word of the sentence vision is secular, but deteriorating economies will Javor Is lamic radicalism. (The first number is for the wall which hasnot been described in this paper. Of course the comma also counts as a word.) The fourth pass of pruning has no effect, so pruning stops. The last row in the table shows the number of disjuncts that remain after power pruning.
5.2 The fast-match data structure technique is roughly the number of different con nector types, which is roughly 30 in our current The inner loop in the algorithm described in sec dictionary. tion 4 searches for a word W and a disjunct d 5.3 Power pruning ofl, this word whose first left connector matches or whose first right connector matches r. If there were a fast way to find all such disjuncts, Power pruning is a refinement of pruning that significant savings might be achieved. The fast takes advantage of the ordering requirement of match data structure, which is based on hashing, the connectors of a disjunct, the exclusion rule, does precisely this. The speed-up afforded by this and other properties of any valid linkage. It also 288 DANIEL SLEATOR AND DAVY TEMPERLEY interacts with the fast-match data structure in a resulting link grammar is at most quadratic in beautiful way. Unfortunately, these details are the number of rules in the dependency grammar. beyond the scope of this paper 4• Table 1 shows None of the algorithms that have been described outcome of pruning and power pruning on a typ for dependency parsing [3, 15, 7] seem to bear ical sentence. any resemblance to ours. It is therefore plausible Each of the refinementsdescribed in this sec-_ to conjecture that our algorithms and techniques tion significantly reduced the time required to do could be very useful for directly parsing depen search for a linkage. The operations of pruning, dency grammars. power pruning, and searching for a linkage all take Gaifman's result shows that it is possible to roughly the same amount of time. represent a link grammar as a dependency gram mar (they're both context-free). But this corre 6 Dependency and catego- spondence is of little use if the parsed structures that result are totally different. rial grammars One problem with constructing a dependency grammar that is in direct correspondence with 6.1 Dependency formalisms a given link grammar is that a linkage in a link grammar my have cycles, whereascycles are not There is a large body of work based on the idea allowed in dependency grammar. If we restrict that linguistic analysis can be done by drawing ourselves to acyclic linkages, we run into another linksdependency between systems words. Thesedependency are variously syntax called problem. This is that there is an exponential dependency grammar[5] , word grammar[10] , blow-up in the number of rules required to express [3, 4] , or the same grammar. This is because each disjunct [6, 7] . of each word in the link grammar requires a sep In dependency grammar,dep aendency grammatical structur sene, arate rule in the dependency grammar. tence is endowed with a Gaifman 's model is not lexical. The method which is very similar to a linkage. This struc classifies the words into categories. One word can ture, as defined by Melcuk consists of a set [10), belong to many categories. Roughly speaking, for of planar directed arcs among theroot words word) that form each disjunct that occurs in the dictionary, there a tree. Each word ( except the has an is a category of all words that have that disjunct. arc out to exactly one other word, and no arc may The notation is therefore in a sense orthogonal to pass over the root word. In a linkage ( as opposed the link grammar notation. to a dependency structure) the links are labeled, We are not aware of any notation for depen undirected, and may form cycles, and there is no dency systems that is lexical, or that is as terse notion of a root word. and well suited for a natural language grammar Gaifman (5) was the first to actually give a as link grammars. There has been work on creat formal method of expressing a dependency gram ing dependency grammars for English [7, 3] , but mar. He shows that his model is context-free. we are not aware of an implementation of a de Melcuk's definition of a dependency structure, pendency grammar for any natural language that and Gaifman 's proof that dependency grammar is is nearly as sophisticated as ours. context free imply that there is a very close re- lationship between these systems and link gram roars. This is the case. 6.2 Categorial grammars catego It is easy to take a dependency grammar in Anotherrial grammar grammatical system, known as a Gaifman 's notation and generate a link grammar [1] bears some resemblance to link that accepts the same language. In this corre grammars. Below we show how to express any spondence, the linkage that results from parsing a cat egorial grammar concisely as a link grammar. sentence is the same as the corresponding depen It appears to be more difficult to express a link dency structure. This means that our algorithm grammar as a categorial grammar. for link parsing can easily be applied to depen Just as in a link grammar, each word of a cat dency grammars. The number of disjuncts in the egorial grammar is associated with one or more 4 Although they do appear in our technical report (14]. PARSING ENGLISH WITH A LINK GRAM MAR 289
symbolic expressions. An expression is either an grammar fora given categorial grammar. The re atomic symbol or a pair of expressions combined verse ( constructing a categorial grammar from a with one of two types of binary operators: / and given link grammar) seems to be more difficult, \. and we do not know of an elegant way to do this. A sentence is in the language defined by the To simplify the construction, we'll use a mod categorial grammar if, after choosing one expres ified definitionof a link grammar called a special sion associated with each word, there is a deriva link grammar. This differs from an ordinary link tion which transforms the chosen sequence of ex grammar in two ways: the links are not allowed pressions into S, a single expression consisting of to form cycles, and there is a special word at the a special atomic symbol. The derivation proceeds beginning of each sentence called the wall. The by combining two neighboring expressions into wall will not be viewed as being part of any sen one using one of the following rules: tence. Let d be a categorial grammar expression. e e\f f/e e We'll show how to build an equivalent link gram f f mar expression E(d). If a word w has the set Here and f are arbitrary expressions, and e { d1 , d2, .•., dk } of categorial expressions, then f\e and f /e are other expressions built using e we'll give that word the following link grammar and f. In both cases the two expressions being expression: combined ( the ones shown above the line) must be adjacent in the current sequence of expressions. E(d1 )orE(d2)or · · · orE(dk) Each combinational operation produces one ex pression (the one below the line), and reduces the The function E( •) is defined recursively as fol number of expressions by one. After n - 1 oper lows: ations have been applied, a sentence of length n has be reduced to one expression. E(J /e) = f /e- or f /e+ or (e+ & E(J)) For example, consider the followingcategorial E(e\f) = e\f- or e\f+ or (e- & E(J)) grammar [9] : E(A) = A- or A+ Harry : NP , S/(S\NP) likes : (S\NP)/NP Here A stands for any atomic symbol from the ..: peanuts: NP categorial grammar, and A, f /e and e\f are con passionately: (S\NP)\(S\NP) nector names in the link grammar formulas. Here is the derivation of Harry likes peanuts The wall has the formulaS+. passionately. Here is the link grammar corresponding to the categorial grammar above: Harry likes peanuts passionately S/(S\NP) (S\NP)/NP NP (S\NP)\(S\NP) WALL : S+ ; peanuts: NP+ or NP- ; S/NP Harry : (NP- or NP+) S\NP or (S/
(Here we have replaced parentheses in the cate 7 Remarks gorial gramm�r expressions with brackets when using them inside of a link grammar expression.) This link grammar gives the followinganalysis of the sentence shown above: Link grammars have become the basis for sev eral other research projects. John Lafferty [11] proposes to build and automatically tune a prob abilistic language model based on link grammars. WALL(� Harry likes�9 peanuts passionately The proposed model gracefully encompasses tri grams and grammatical constraints in one frame Notice that in this construction, both the size work. Andrew Hunt6 has developed a new model of the link grammar formula, and the number of of the relationship of prosody and syntax based on disjuncts it represents are linear in the size of link grammars. He has implemented the model, the original categorial grammar expressions. This and in preliminary tests, the results are much bet suggests that a very efficient way to parse a cate ter than with other models. Tom Brehony6 has gorial grammar would be to transformit to a link modified our parser to detect the kinds of errors grammar, then apply the algorithms and heuris that Francophonesmake when they write in En tics described in this paper. glish.
6Personal communication. PARSING ENGLISH WITH A LINK GRAM MAR 291
References Dependency Syntax: Theory [10] Melcand Prukactice,, I. A. State University of New York [1) Y. Bar-Hillel, Language and information; se Press 1988. lected essays on their theory and application. Addison-Wesley, 1964. [11] Lafferty, J, D. Sleator, D. Temperley, [2) Cormen, InT.troduction H., C. E. toLeiserson Algorithms,, and R. L. "Grammatical Trigrams: A Probabilistic Rivest, MIT Model of Link Grammar," Proc. of the Press and McGraw-Hill, 1990. 1992 AAAI Fall Symp. on Probabilistic Ap [3) Fraser, N., "Parsing and dependency gram proaches to Natural Language, and Techni UCL Work cal report CMU-CS-92-181, School _of Com mar," In: Carston, Robyn (ed.) ing Papers in Linguistics puter Science, Carnegie Mellon University, 1, University Col Sept 1992. lege London, London, 1989 Pages 296-319. [4) Fraser, N., "Prolegomena to a formal theory UCL Working [12) Oehrle, R. T., E. Bach, and D. Wheeler, Ed of dependency grammar," In: Categorial Grammars and Natural Lan Papers in Linguistics itorsguage Structures 2, University College D. Reidel Publishing Com London, London, 1990 Pages 298-319. pany, 1988. [5) Gaifman, H., "DependencyInfo systemsrmation andand phrase-structure systems," Control [13] Y. Schabes. "Stochastic lexicalizedProceedings tree of 8, 1965, Pages 304-337. adjoining grammars." In Word Grammar, COLING-92, [6) Hudson, R., Basil Black Nantes, France, July 1992. well, 1984. [7) Hudson, R., "Towards a computer testable [14] Sleator, D. D., D. Temperley, "Parsing En word grammar of English," In: Carston, glish with a Link Grammar," Technical re UCL Working Papers in Lin port CMU-CS-91-196, Carnegie Mellon Uni Robynguistics (ed.) 1, University College London, Lon versity, School of Computer Science, October don, 1989, Pages 321-339. 1991. English Word Grammar, [8] Hudson, R., Basil [15) van Zuijlen, J., M., "Probabilistic methods Blackwell, 1990. in dependency grammar parsing," Proceed [9) Joshi, ScienceA. K.,, "Natural Language Process ings of the International Workshop on Pars ing," Volume 253, No. 5025, (Sept. ing Technologies, Carnegie Mellon Univer 13, 1991), Pages 1242-1249. sity, 1989, Pages 142-250. 292 DANIEL SLEATOR AND DAVY TEMPERLEY