The Kleene Language for Weighted Finite-State Programming: User Documentation, Version 0.9.4.0 This Document is Work in Progress Corrections and Suggestions Are Welcome Kenneth R. Beesley SAP Labs, LLC P.O. Box 540475 North Salt Lake Utah 84054, USA [email protected] 19 May 2014 Copyright © 2014 SAP AG. Released under the Apache License, Version 2. http://www.apache.org/licenses/LICENSE-2.0.html Beesley i Preface What is Kleene? Kleene is a programming language that can be used to create many useful and efficient linguistic applications based on finite-state machines (FSMs). These applications include tokenizers, spelling checkers, spelling correc- tors, morphological analyzer/generators and shallow parsers. FSMs are also widely used in speech synthesis and speech recognition. Kleene allows programmers to define, build, manipulate and test finite- state machines using regular expressions and right-linear phrase-structure grammars; and Kleene supports variables, rule-like expressions, user-defined functions and familiar programming-language control syntax. The FSMs can include acceptors and two-projection transducers, either weighted un- der the Tropical Semiring or unweighted. If this makes no sense, the pur- pose of the book is to explain it. Pre-edited Kleene scripts can be run from the command line, and a graphical user interface is provided for interactive learning, programming and testing. Operating Systems Kleene runs on OS X and Linux, requiring Java version 1.5 or higher. Prerequisites This book assumes only superficial familiarity with regular languages, reg- ular relations and finite-state machines. While readers need not have any experience with finite-state program- ming, those who have no programming experience at all, e.g. in a language like Java, C++, Perl, Python, Ruby, etc., will find it difficult. ii The Kleene Language Implementation The Kleene parser is implemented in JavaCC/JJTree* and Java,† and the interpreter calls functions in the OpenFst library‡ via the Java Native In- terface (JNI).§ Kleene has its own syntax, based on familiar regular expres- sions, and it borrows much of its semantics from the Xerox/parc finite- state toolkit.¶ Language-restriction expressions and alternation rules are implemented using algorithms supplied by Måns Huldén.‖ Acknowledgments Thanks are due to the following people for advice and encouragement over the years. I have not always followed the advice I received, so I take full responsibility for any problems, failures or infelicities. Lauri Karttunen, Måns Huldén, Paola Nieddu, Phil Sours, Cyril Allauzen, André Kempe, Mike Wilkens, Kemal Oflazer, Natasha Lloyd Kleene is named after American mathematician Stephen Cole Kleene (1909–1994), who investigated the properties of regular sets and invented the metalan- guage of regular expressions. *http://java.net/projects/javacc †http://www.java.com/ ‡Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut and Mehryar Mohri, “OpenFst: A General and Efficient Weighted Finite-State Transducer Library,” Proceedings of the Ninth International Conference on Implementation and Application of Automata, (CIAA 2007), volume 4783 of Lecture Notes in Computer Science, pages 11-23. Springer, 2007. http://www.openfst.org. §http://www.ibm.com/developerworks/java/tutorials/j-jni/ ¶http://www.fsmbook.com ‖Måns Huldén, 2009 Finite-State Machine Construction Methods and Algorithms for Phonology and Morphology, Ph.D. dissertation, University of Arizona. Beesley iii License Kleene is free and open-source, released 4 May 2012 by SAP AG†† under the Apache License, Version 2.** The OpenFst library is available under the same license. ††http://www.sap.com/ **http://www.apache.org/licenses/LICENSE-2.0 Contents 1 Introduction 1 1.1 What is Kleene? . 1 1.2 The Kleene Name . 3 1.3 Possible Applications . 3 1.4 Design Criteria . 3 1.5 Terminology . 5 2 Getting Started with Kleene 7 2.1 Installation and Prerequisites . 7 2.1.1 Pre-compiled Binaries . 7 2.1.2 Compiling Kleene from Source Code . 8 2.2 Launching the Kleene gui ................... 9 2.3 OK, What can you do in Kleene? . 11 2.3.1 First, is it working? . 11 2.3.2 Visualizing Finite-State Machines . 11 2.3.3 Compile regular expressions into FSMs . 13 2.3.4 Optimization of FSMs . 23 2.3.5 Using Defined Variables . 26 2.3.6 Characters . 27 2.3.7 Regular Languages and Regular Relations . 31 2.4 Looking at FSMs . 41 2.5 Applications . 42 v vi The Kleene Language 3 Regular Expression Syntax 45 3.1 Regular Expressions . 45 3.1.1 Primary Regular Expressions . 46 3.1.2 Inherently Delimited Regular Expressions . 47 3.1.3 Regular-Expression Operators . 49 3.1.4 Precedence Issues . 49 3.1.5 Weights . 52 3.1.6 Whitespace in Regular Expressions . 54 3.1.7 Denoting the Empty String . 55 3.1.8 Denoting Any Symbol . 56 3.2 Abstraction Mechanisms . 57 3.2.1 Variables . 57 3.2.2 Pre-defined Functions . 58 3.2.3 User-defined Function Syntax . 60 3.2.4 Function Call Semantics . 64 3.2.5 Function Parameters with Default Values . 66 3.3 Right-linear Phrase-structure Grammars . 68 3.3.1 Right-linear Syntax . 68 3.3.2 Right-linear Semantics . 70 3.4 Scope . 72 3.4.1 Assignments, Declarations and Local Scope . 72 3.4.2 Stand-alone Code Blocks . 73 3.4.3 Function Blocks . 73 3.4.4 External Variables . 74 3.4.5 Free Variables . 77 3.4.6 Combining Free and Local Usage . 78 3.4.7 Export Statements . 79 3.4.8 Practical Use of Code Blocks . 80 3.5 Language Restriction Expressions . 83 4 Alternation Rules 87 4.1 What are Alternation Rules? . 87 Beesley vii 4.2 Mindtuning for Alternations . 88 4.2.1 Underlying and Surface, Upper and Lower . 88 4.2.2 Writing and Testing Alternation Rules . 92 4.2.3 Derivations or Cascades of Rules . 94 4.2.4 The Richness of Alternation-Rule Types . 99 4.3 Basic Mapping Rules . 100 4.3.1 Right-Arrow Rules . 100 4.3.2 Left-arrow Rules . 105 4.4 Optional Alternation Rules . 106 4.5 Maximum and Minimum Matching . 107 4.6 Deletion Rules . 108 4.7 Epenthesis Rules . 108 4.8 Markup Rules . 109 4.9 Two-Level Rule Contexts . 110 4.10 Parallel Rules . 112 4.10.1 Vertical Derivations vs. Parallel Rules . 112 4.10.2 The Built-In Parallel Rule-Compilation Function . 115 4.10.3 Rules with Where-Clauses . 119 4.10.4 Rules where the Input can Match the Empty String . 124 4.11 Transducer-Style Rules . 125 4.11.1 Traditional Rules vs. Transducer-Style Rules . 125 4.11.2 Simple Mapping Rules in the Transducer Style . 127 4.11.3 Transducer-Style Rules vs. Where-Clause Rules . 133 4.11.4 Transducer-Style Rules vs. Traditional Markup Rules 134 4.11.5 Transducer-Style Rules Not Expressible as Traditional Rules . 135 5 Examples without Weights 137 5.1 Introduction . 137 5.2 Spanish Morphology . 137 5.2.1 Spanish Verbs . 137 5.2.2 Spanish Nouns . 150 viii The Kleene Language 5.3 Latin Morphology . 150 5.4 Aymara Morphology . 150 6 Examples with Weights 151 6.1 Introduction . 151 6.2 Weighted Edit Distance for Spelling Correction . 151 7 Challenges in Morphology 163 7.1 Overview . 163 7.2 Infixation . 164 7.3 Reduplication . 164 7.3.1 Range of Reduplicative Phenomena . 164 7.3.2 Contiguous Reduplication . 165 7.3.3 Discontiguous Reduplication . 174 7.4 Semitic Interdigitation . 174 7.5 Constraining Long-distance Dependencies . 174 8 Arithmetic Expressions 175 8.1 Mindtuning for Arithmetic Expressions . 175 8.2 Primary Arithmetic Expressions . 176 8.3 Arithmetic Expression Operators . 177 8.4 Arithmetic Functions . 177 8.5 Boolean Functions . 179 8.5.1 Special Case of Arithmetic Functions . 179 8.5.2 Assert and Require Statements . 181 9 Lists 183 9.1 Lists of FSMs and Numbers . 183 9.2 List Literals, Identifiers, and Assignment . .184 9.3 Pre-Defined Functions Operating on Lists . .185 9.3.1 Functions Joining the Elements of a List . 188 9.3.2 Functions Returning the Map of a List . 190 9.3.3 Functions Returning the Alphabet (Sigma) of an FSM 191 Beesley ix 9.3.4 Functions Returning the Size of a List . 192 9.4 Iteration through Members of a List . 192 9.5 User-Defined Functions and Lists . .193 9.6 Traditional Array-indexing Syntax . 194 10 Other Syntax 197 10.1 Void Functions with Names . 197 10.2 Anonymous Void Functions . 197 10.3 Control Syntax . 198 10.4 Input/Output . 198 10.4.1 Scripts . 198 10.4.2 XML Input/Output . 201 10.4.3 dot Output . 202 10.4.4 Interactive Testing in the gui . 203 10.5 Memory Management . 205 10.5.1 Garbage Collection . 205 10.5.2 Java Memory Usage . 206 10.5.3 Java Fst Objects . 206 10.5.4 Contents of the Main Symbol Table . 207 10.5.5 C++ Memory Space . 207 11 Code Generation and Runtime Code 209 11.1 Using Kleene FSMs in Real Projects . 209 11.2 The fst2java Experiment . 210 11.2.1 Stand-Alone Java Code Generation . 210 11.2.2 How it Works . 211 11.2.3 How to Create the Java Class Files for an FSM . 213 11.2.4 FSM to Java Examples . 221 11.2.5 Generated Java Code API . 231 11.3 Runtime Code . 234 11.3.1 Status of Runtime Code . 234 11.3.2 Basic Functionality of Runtime Code . 235 11.3.3 Kinds of Runtime Code . 236 x The Kleene Language 12 rtn Support 239 12.1 OpenFst Recursive Transition Networks (rtns) . 239 12.1.1 Status .