Joshua 2.0: A Toolkit for Parsing-Based Machine Translation with Syntax, Semirings, Discriminative Training and Other Goodies Zhifei Li, Chris Callison-Burch, Chris Dyer,y Juri Ganitkevitch, Ann Irvine, Lane Schwartz,? Wren N. G. Thornton, Ziyuan Wang, Jonathan Weese and Omar F. Zaidan Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD y Computational Linguistics and Information Processing Lab, University of Maryland, College Park, MD ? Natural Language Processing Lab, University of Minnesota, Minneapolis, MN Abstract authors and several other groups in their daily re- search, and has been substantially refined since the We describe the progress we have made in first release. The most important new functions in the past year on Joshua (Li et al., 2009a), the toolkit are: an open source toolkit for parsing based • Support for any style of synchronous context machine translation. The new functional- free grammar (SCFG) including syntax aug- ity includes: support for translation gram- ment machine translation (SAMT) grammars mars with a rich set of syntactic nonter- (Zollmann and Venugopal, 2006) minals, the ability for external modules to posit constraints on how spans in the in- • Support for external modules to posit transla- put sentence should be translated, lattice tions for spans in the input sentence that con- parsing for dealing with input uncertainty, strain decoding (Irvine et al., 2010) a semiring framework that provides a uni- fied way of doing various dynamic pro- • Lattice parsing for dealing with input un- gramming calculations, variational decod- certainty, including ambiguous output from ing for approximating the intractable MAP speech recognizers or Chinese word seg- decoding, hypergraph-based discrimina- menters (Dyer et al., 2008) tive training for better feature engineering, • A semiring architecture over hypergraphs a parallelized MERT module, document- that allows many inference operations to be level and tail-based MERT, visualization implemented easily and elegantly (Li and of the derivation trees, and a cleaner Eisner, 2009) pipeline for MT experiments. • Improvements to decoding through varia- 1 Introduction tional decoding and other approximate meth- ods that overcome intractable MAP decoding Joshua is an open-source toolkit for parsing-based (Li et al., 2009b) machine translation that is written in Java. The initial release of Joshua (Li et al., 2009a) was a • Hypergraph-based discriminative training for re-implementation of the Hiero system (Chiang, better feature engineering (Li and Khudan- 2007) and all its associated algorithms, includ- pur, 2009b) ing: chart parsing, n-gram language model inte- • A parallelization of MERT’s computations, gration, beam and cube pruning, and k-best ex- and supporting document-level and tail-based traction. The Joshua 1.0 release also included optimization (Zaidan, 2010) re-implementations of suffix array grammar ex- traction (Lopez, 2007; Schwartz and Callison- • Visualization of the derivation trees and hy- Burch, 2010) and minimum error rate training pergraphs (Weese and Callison-Burch, 2010) (Och, 2003; Zaidan, 2009). Additionally, it in- cluded parallel and distributed computing tech- • A convenient framework for designing and niques for salability (Li and Khudanpur, 2008). running reproducible machine translation ex- This paper describes the additions to the toolkit periments (Schwartz, under review) over the past year, which together form the 2.0 re- The sections below give short descriptions for lease. The software has been heavily used by the each of these new functions. 2 Support for Syntax-based Translation ticular span to be translated as an NP. We modi- fied Joshua’s chart-based decoder to support these The initial release of Joshua supported only constraints. Hiero-style SCFGs, which use a single nontermi- nal symbol X. This release includes support for ar- bitrary SCFGs, including ones that use a rich set 4 Semiring Parsing of linguistic nonterminal symbols. In particular we have added support for Zollmann and Venu- In Joshua, we use a hypergraph (or packed forest) gopal (2006)’s syntax-augmented machine trans- to compactly represent the exponentially many lation. SAMT grammar extraction is identical to derivation trees generated by the decoder for an Hiero grammar extraction, except that one side of input sentence. Given a hypergraph, we may per- the parallel is parsed, and syntactic labels replace form many atomic inference operations, such as the X nonterminals in Hiero-style rules. Instead of finding one-best or k-best translations, or com- extracting this Hiero rule from the bitext puting expectations over the hypergraph. For [X] ) [X,1] sans [X,2] j [X,1] without [X,2] each such operation, we could implement a ded- the nonterminals can be labeled according to icated dynamic programming algorithm. How- which constituents cover the nonterminal span on ever, a more general framework to specify these the parsed side of the bitext. This constrains what algorithms is semiring-weighted parsing (Good- types of phrases the decoder can use when produc- man, 1999). We have implemented the in- ing a translation. side algorithm, the outside algorithm, and the [VP] ) [VBN] sans [NP] j [VBN] without [NP] inside-outside speedup described by Li and Eis- [NP] ) [NP] sans [NP] j [NP] without [NP] ner (2009), plut the first-order expectation semir- Unlike GHKM (Galley et al., 2004), SAMT has ing (Eisner, 2002) and its second-order version (Li the same coverage as Hiero, because it allows and Eisner, 2009). All of these use our newly im- non-constituent phrases to get syntactic labels us- plemented semiring framework. ing CCG-style slash notation. Experimentally, we The first- and second-order expectation semi- have found that the derivations created using syn- rings can also be used to compute many interesting tactically motivated grammars exhibit more coher- quantities over hypergraphs. These quantities in- ent syntactic structure than Hiero and typically re- clude expected translation length, feature expec- sult in better reordering, especially for languages tation, entropy, cross-entropy, Kullback-Leibler with word orders that diverge from English, like divergence, Bayes risk, variance of hypothesis Urdu (Baker et al., 2009). length, gradient of entropy and Bayes risk, covari- 3 Specifying Constraints on Translation ance and Hessian matrix, and so on. Integrating output from specialized modules (like transliterators, morphological analyzers, and 5 Word Lattice Input modality translators) into the MT pipeline can improve translation performance, particularly for We generalized the bottom-up parsing algorithm low-resource languages. We have implemented that generates the translation hypergraph so that an XML interface that allows external modules it supports translation of word lattices instead of to propose alternate translation rules (constraints) just sentences. Our implementation’s runtime and for a particular word span to the decoder (Irvine memory overhead is proportional to the size of the et al., 2010). Processing that is separate from lattice, rather than the number of paths in the lat- the MT engine can suggest translations for some tice (Dyer et al., 2008). Accepting lattice-based set of source side words and phrases. The XML input allows the decoder to explore a distribution format allows for both hard constraints, which over input sentences, allowing it to select the best must be used, and soft constraints, which compete translation from among all of them. This is es- with standard extracted translation rules, as well pecially useful when Joshua is used to translate as specifying associated feature weights. In ad- the output of statistical preprocessing components, dition to specifying translations, the XML format such as speech recognizers or Chinese word seg- allows constraints on the lefthand side of SCFG menters, which can encode their alternative analy- rules, which allows constraints like forcing a par- ses as confusion networks or lattices. 6 Variational Decoding need to use an oracle translation (i.e., the transla- tion in the hypergraph that is most simmilar to the Statistical models in machine translation exhibit reference translation) as a surrogate for training. spurious ambiguity. That is, the probability of an We implemented the oracle extraction algorithm output string is split among many distinct deriva- described by Li and Khudanpur (2009a) for this tions (e.g., trees or segmentations) that have the purpose. same yield. In principle, the goodness of a string Given the current infrastructure, other training is measured by the total probability of its many methods (e.g., maximum conditional likelihood or derivations. However, finding the best string dur- MIRA as used by Chiang et al. (2009)) can also be ing decoding is then NP-hard. The first version of easily supported with minimum coding. We plan Joshua implemented the Viterbi approximation, to implement a large number of feature functions which measures the goodness of a translation us- in Joshua so that exhaustive feature engineering is ing only its most probable derivation. possible for MT. The Viterbi approximation is efficient, but it ig- nores most of the derivations in the hypergraph. 8 Minimum Error Rate Training We implemented variational decoding (Li et al., 2009b), which works as follows. First, given a for- Joshua’s MERT module optimizes parameter eign string (or lattice), the MT system produces a weights so as to maximize performance on a de-