MOSES Statistical Machine Translation System User Manual and Code Guide Philipp Koehn [email protected] University of Edinburgh Abstract This document serves as user manual and code guide for the Moses machine translation decoder. The decoder was mainly developed by Hieu Hoang and Philipp Koehn at the University of Edinburgh and extended during a Johns Hopkins University Summer Workshop and further developed under EuroMa- trix and GALE project funding. The decoder (which is part of a complete statistical machine translation toolkit) is the de facto benchmark for research in the field. This document serves two purposes: a user manual for the functions of the Moses decoder and a code guide for developers. In large parts, this manual is identical to documentation available at the official Moses decoder web site http://www.statmt.org/. This document does not describe in depth the underlying methods, which are described in the text book Statistical Machine Translation (Philipp Koehn, Cambridge University Press, 2009). August 8, 2016 2 Acknowledgments The Moses decoder was supported by the European Framework 6 projects EuroMatrix, TC-Star, the European Framework 7 projects EuroMatrixPlus, Let’s MT, META-NET and MosesCore and the DARPA GALE project, as well as several universities such as the University of Edin- burgh, the University of Maryland, ITC-irst, Massachusetts Institute of Technology, and others. Contributors are too many to mention, but it is important to stress the substantial contributions from Hieu Hoang, Chris Dyer, Josh Schroeder, Marcello Federico, Richard Zens, and Wade Shen. Moses is an open source project under the guidance of Philipp Koehn. Contents 1 Introduction 11 1.1 Welcome to Moses! . 11 1.2 Overview . 11 1.2.1 Technology . 11 1.2.2 Components . 12 1.2.3 Development . 13 1.2.4 Moses in Use . 14 1.2.5 History . 14 1.3 Get Involved . 14 1.3.1 Mailing List . 14 1.3.2 Suggestions . 15 1.3.3 Development . 15 1.3.4 Use . 15 1.3.5 Contribute . 15 1.3.6 Projects . 16 2 Installation 23 2.1 Getting Started with Moses . 23 2.1.1 Easy Setup on Ubuntu (on other linux systems, you’ll need to install packages that provide gcc, make, git, automake, libtool) . 23 2.1.2 Compiling Moses directly with bjam . 25 2.1.3 Other software to install . 26 2.1.4 Platforms . 27 2.1.5 Windows Installation . 27 2.1.6 OSX Installation . 28 2.1.7 Linux Installation . 28 2.1.8 Run Moses for the first time . 30 2.1.9 Chart Decoder . 31 2.1.10 Next Steps . 31 2.1.11 bjam options . 31 2.2 Building with Eclipse . 33 2.3 Baseline System . 34 2.3.1 Overview . 34 2.3.2 Installation . 35 2.3.3 Corpus Preparation . 36 2.3.4 Language Model Training . 37 2.3.5 Training the Translation System . 38 3 4 CONTENTS 2.3.6 Tuning . 39 2.3.7 Testing . 40 2.3.8 Experiment Management System (EMS) . 43 2.4 Releases . 44 2.4.1 Release 3.0 (3rd Feb, 2015) . 44 2.4.2 Release 2.1.1 (3rd March, 2014) . 44 2.4.3 Release 2.1 (21th Jan, 2014) . 44 2.4.4 Release 1.0 (28th Jan, 2013) . 45 2.4.5 Release 0.91 (12th October, 2012) . 50 2.4.6 Status 11th July, 2012 . 50 2.4.7 Status 13th August, 2010 . 52 2.4.8 Status 9th August, 2010 . 53 2.4.9 Status 26th April, 2010 . 53 2.4.10 Status 1st April, 2010 . 53 2.4.11 Status 26th March, 2010 . 53 2.5 Work in Progress . 54 3 Tutorials 55 3.1 Phrase-based Tutorial . 55 3.1.1 A Simple Translation Model . 55 3.1.2 Running the Decoder . 56 3.1.3 Trace . 57 3.1.4 Verbose . 58 3.1.5 Tuning for Quality . 62 3.1.6 Tuning for Speed . 63 3.1.7 Limit on Distortion (Reordering) . 66 3.2 Tutorial for Using Factored Models . 67 3.2.1 Train an unfactored model . 68 3.2.2 Train a model with POS tags . 69 3.2.3 Train a model with generation and translation steps . 72 3.2.4 Train a morphological analysis and generation model . 73 3.2.5 Train a model with multiple decoding paths . 74 3.3 Syntax Tutorial . 75 3.3.1 Tree-Based Models . 75 3.3.2 Decoding . 77 3.3.3 Decoder Parameters . 81 3.3.4 Training . 83 3.3.5 Using Meta-symbols in Non-terminal Symbols (e.g., CCG) . 86 3.3.6 Different Kinds of Syntax Models . 87 3.3.7 Format of text rule table . 92 3.4 Optimizing Moses . 93 3.4.1 Multi-threaded Moses . 93 3.4.2 How much memory do I need during decoding? . 94 3.4.3 How little memory can I get away with during decoding? . 96 3.4.4 Faster Training . 96 3.4.5 Training Summary . 98 3.4.6 Language Model . 99 3.4.7 Suffix array . 101 CONTENTS 5 3.4.8 Cube Pruning . 102 3.4.9 Minimizing memory during training . 102 3.4.10 Minimizing memory during decoding . 102 3.4.11 Phrase-table types . 104 3.5 Experiment Management System . 105 3.5.1 Introduction . 105 3.5.2 Requirements . 106 3.5.3 Quick Start . 107 3.5.4 More Examples . 110 3.5.5 Try a Few More Things . 113 3.5.6 A Short Manual . 118 3.5.7 Analysis . 126 4 User Guide 131 4.1 Support Tools . 131 4.1.1 Overview . ..