DOCSLIB.ORG

The Kleene Language for Weighted Finite-State Programming: User Documentation, Version 0.9.4.1

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

The Kleene Language for Weighted Finite-State Programming: User Documentation, Version 0.9.4.1 This Document is Work in Progress Corrections and Suggestions Are Welcome Kenneth R. Beesley SAP Labs, LLC P.O. Box 540475 North Salt Lake Utah 84054, USA [email protected] 7 July 2014 Copyright © 2014 SAP AG. Released under the Apache License, Version 2. http://www.apache.org/licenses/LICENSE-2.0.html Beesley i Preface What is Kleene? Kleene is a programming language that can be used to create many useful and efficient linguistic applications based on finite-state machines (FSMs). These applications include tokenizers, spelling checkers, spelling correc- tors, morphological analyzer/generators and shallow parsers. FSMs are also widely used in speech synthesis and speech recognition. Kleene allows programmers to define, build, manipulate and test finite- state machines using regular expressions and right-linear phrase-structure grammars; and Kleene supports variables, rule-like expressions, user-defined functions and familiar programming-language control syntax. The FSMs can include acceptors and two-projection transducers, either weighted un- der the Tropical Semiring or unweighted. If this makes no sense, the pur- pose of the book is to explain it. Pre-edited Kleene scripts can be run from the command line, and a graphical user interface is provided for interactive learning, programming and testing. Operating Systems Kleene runs on OS X and Linux, requiring Java version 1.6 or higher. Prerequisites This book assumes only superficial familiarity with regular languages, reg- ular relations and finite-state machines. While readers need not have any experience with finite-state program- ming, those who have no programming experience at all, e.g. in a language like Java, C++, Perl, Python, Ruby, etc., will find it difficult. ii The Kleene Language Implementation The Kleene parser is implemented in JavaCC/JJTree* and Java,† and the interpreter calls functions in the OpenFst library‡ via the Java Native In- terface (JNI).§ Kleene has its own syntax, based on familiar regular expres- sions, and it borrows much of its semantics from the Xerox/parc finite- state toolkit.¶ Language-restriction expressions and alternation rules are implemented using algorithms supplied by Måns Huldén.‖ Acknowledgments Thanks are due to the following people for advice and encouragement over the years. I have not always followed the advice I received, so I take full responsibility for any problems, failures or infelicities. Lauri Karttunen, Måns Huldén, Paola Nieddu, Phil Sours, Cyril Allauzen, André Kempe, Mike Wilkens, Kemal Oflazer, Natasha Lloyd Kleene is named after American mathematician Stephen Cole Kleene (1909–1994), who investigated the properties of regular sets and invented the metalan- guage of regular expressions. *http://java.net/projects/javacc †http://www.java.com/ ‡Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut and Mehryar Mohri, “OpenFst: A General and Efficient Weighted Finite-State Transducer Library,” Proceedings of the Ninth International Conference on Implementation and Application of Automata, (CIAA 2007), volume 4783 of Lecture Notes in Computer Science, pages 11-23. Springer, 2007. http://www.openfst.org. §http://www.ibm.com/developerworks/java/tutorials/j-jni/ ¶http://www.fsmbook.com ‖Måns Huldén, 2009 Finite-State Machine Construction Methods and Algorithms for Phonology and Morphology, Ph.D. dissertation, University of Arizona. Beesley iii License Kleene is free and open-source, released 4 May 2012 by SAP AG†† under the Apache License, Version 2.** The OpenFst library is available under the same license. ††http://www.sap.com/ **http://www.apache.org/licenses/LICENSE-2.0 Contents 1 Introduction 1 1.1 What is Kleene? . 1 1.2 The Kleene Name . 3 1.3 Possible Applications . 3 1.4 Design Criteria . 3 1.5 Terminology . 5 2 Getting Started with Kleene 7 2.1 Installation and Prerequisites . 7 2.1.1 Pre-compiled Binaries . 7 2.1.2 Compiling Kleene from Source Code . 8 2.2 Launching the Kleene gui ................... 9 2.3 OK, What can you do in Kleene? . 11 2.3.1 First, is it working? . 11 2.3.2 Visualizing Finite-State Machines . 11 2.3.3 Compile regular expressions into FSMs . 13 2.3.4 Optimization of FSMs . 23 2.3.5 Using Defined Variables . 26 2.3.6 Characters . 27 2.3.7 Regular Languages and Regular Relations . 31 2.4 Looking at FSMs . 41 2.5 Applications . 42 v vi The Kleene Language 3 Regular Expression Syntax 45 3.1 Regular Expressions . 45 3.1.1 Primary Regular Expressions . 46 3.1.2 Inherently Delimited Regular Expressions . 47 3.1.3 Regular-Expression Operators . 49 3.1.4 Precedence Issues . 49 3.1.5 Weights . 52 3.1.6 Whitespace in Regular Expressions . 54 3.1.7 Denoting the Empty String . 55 3.1.8 Denoting Any Symbol . 56 3.2 Abstraction Mechanisms . 57 3.2.1 Variables . 57 3.2.2 Pre-defined Functions . 58 3.2.3 User-defined Function Syntax . 60 3.2.4 Function Call Semantics . 64 3.2.5 Function Parameters with Default Values . 66 3.3 Right-linear Phrase-structure Grammars . 68 3.3.1 Right-linear Syntax . 68 3.3.2 Right-linear Semantics . 70 3.4 Scope . 72 3.4.1 Assignments, Declarations and Local Scope . 72 3.4.2 Stand-alone Code Blocks . 73 3.4.3 Function Blocks . 73 3.4.4 External Variables . 74 3.4.5 Free Variables . 77 3.4.6 Combining Free and Local Usage . 78 3.4.7 Export Statements . 79 3.4.8 Practical Use of Code Blocks . 80 3.5 Language Restriction Expressions . 83 4 Alternation Rules 87 4.1 What are Alternation Rules? . 87 Beesley vii 4.2 Mindtuning for Alternations . 88 4.2.1 Underlying and Surface, Upper and Lower . 88 4.2.2 Writing and Testing Alternation Rules . 92 4.2.3 Derivations or Cascades of Rules . 94 4.2.4 The Richness of Alternation-Rule Types . 99 4.3 Basic Mapping Rules . 100 4.3.1 Right-Arrow Rules . 100 4.3.2 Left-arrow Rules . 105 4.4 Optional Alternation Rules . 106 4.5 Maximum and Minimum Matching . 107 4.6 Deletion Rules . 108 4.7 Epenthesis Rules . 108 4.8 Markup Rules . 109 4.9 Two-Level Rule Contexts . 110 4.10 Parallel Rules . 112 4.10.1 Vertical Derivations vs. Parallel Rules . 112 4.10.2 The Built-In Parallel Rule-Compilation Function . 115 4.10.3 Rules with Where-Clauses . 119 4.10.4 Rules where the Input can Match the Empty String . 124 4.11 Transducer-Style Rules . 125 4.11.1 Traditional Rules vs. Transducer-Style Rules . 125 4.11.2 Simple Mapping Rules in the Transducer Style . 127 4.11.3 Transducer-Style Rules vs. Where-Clause Rules . 133 4.11.4 Transducer-Style Rules vs. Traditional Markup Rules 134 4.11.5 Transducer-Style Rules Not Expressible as Traditional Rules . 135 5 Examples without Weights 137 5.1 Introduction . 137 5.2 Spanish Morphology . 137 5.2.1 Spanish Verbs . 137 5.2.2 Spanish Nouns . 150 viii The Kleene Language 5.3 Latin Morphology . 150 5.4 Aymara Morphology . 150 6 Examples with Weights 151 6.1 Introduction . 151 6.2 Weighted Edit Distance for Spelling Correction . 151 7 Challenges in Morphology 163 7.1 Overview . 163 7.2 Infixation . 164 7.3 Reduplication . 164 7.3.1 Range of Reduplicative Phenomena . 164 7.3.2 Contiguous Reduplication . 165 7.3.3 Discontiguous Reduplication . 174 7.4 Semitic Interdigitation . 174 7.5 Constraining Long-distance Dependencies . 174 8 Arithmetic Expressions 175 8.1 Mindtuning for Arithmetic Expressions . 175 8.2 Primary Arithmetic Expressions . 176 8.3 Arithmetic Expression Operators . 177 8.4 Arithmetic Functions . 177 8.5 Boolean Functions . 179 8.5.1 Special Case of Arithmetic Functions . 179 8.5.2 Assert and Require Statements . 181 9 Lists 183 9.1 Lists of FSMs and Numbers . 183 9.2 List Literals, Identifiers, and Assignment . .184 9.3 Pre-Defined Functions Operating on Lists . .185 9.3.1 Functions Joining the Elements of a List . 188 9.3.2 Functions Returning the Map of a List . 190 9.3.3 Functions Returning the Alphabet (Sigma) of an FSM 191 Beesley ix 9.3.4 Functions Returning the Size of a List . 192 9.4 Iteration through Members of a List . 192 9.5 User-Defined Functions and Lists . .193 9.6 Traditional Array-indexing Syntax . 194 10 Other Syntax 197 10.1 Void Functions with Names . 197 10.2 Anonymous Void Functions . 197 10.3 Control Syntax . 198 10.4 Input/Output . 198 10.4.1 Scripts . 198 10.4.2 XML Input/Output . 201 10.4.3 dot Output . 202 10.4.4 Interactive Testing in the gui . 203 10.5 Memory Management . 205 10.5.1 Garbage Collection . 205 10.5.2 Java Memory Usage . 206 10.5.3 Java Fst Objects . 206 10.5.4 Contents of the Main Symbol Table . 207 10.5.5 C++ Memory Space . 207 11 Code Generation and Runtime Code 209 11.1 Using Kleene FSMs in Real Projects . 209 11.2 The fst2java Experiment . 210 11.2.1 Stand-Alone Java Code Generation . 210 11.2.2 How it Works . 211 11.2.3 How to Create the Java Class Files for an FSM . 213 11.2.4 FSM to Java Examples . 221 11.2.5 Generated Java Code API . 231 11.3 Runtime Code . 234 11.3.1 Status of Runtime Code . 234 11.3.2 Basic Functionality of Runtime Code . 235 11.3.3 Kinds of Runtime Code . 236 x The Kleene Language 12 rtn Support 239 12.1 OpenFst Recursive Transition Networks (rtns) . 239 12.1.1 Status .
Recommended publications
  • HFST—A System for Creating NLP Tools

    HFST—A System for Creating NLP Tools

    HFST—a System for Creating NLP Tools Krister Lindén, Erik Axelson, Senka Drobac, Sam Hardwick, Juha Kuokkala, Jyrki Niemi, Tommi A Pirinen, and Miikka Silfverberg University of Helsinki Department of Modern Languages Unioninkatu 40 A FI-00014 University of Helsinki, Finland {krister.linden, erik.axelson, senka.drobac, sam.hardwick, juha.kuokkala, jyrki.niemi, tommi.pirinen, miikka.silfverberg}@helsinki.fi Abstract. The paper presents and evaluates various NLP tools that have been created using the open source library HFST–Helsinki Finite- State Technology and outlines the minimal extensions that this has re- quired to a pure finite-state system. In particular, the paper describes an implementation and application of Pmatch presented by Karttunen at SFCM 2011. Keywords: finite-state technology, language identification, morpholog- ical guessers, spell-checking, named-entity recognition, language genera- tion, parsing, HFST, XFST, Pmatch Introduction In natural language processing, finite-state string transducer methods have been found useful for solving a number of practical problems ranging from language identification via morphological processing and generation to part-of-speech tag- ging and named-entity recognition, as long as the problems lend themselves to a formulation based on matching and transforming local context. In this paper, we present and evaluate various tools that have been created using HFST–Helsinki Finite-State Technology1 and outline the minimal exten- sions that this has required to a pure FST system. In particular, we describe an implementation of Pmatch presented by Karttunen at SFCM 2011 [7] and its application to a large-scale named-entity recognizer for Swedish. The paper is structured as follows: Section 1 is on applications and their evaluation.
  • Using Finite State Transducers for Multilingual Rule-Based Phonetic

    Using Finite State Transducers for Multilingual Rule-Based Phonetic

    Thesis submitted for the degree of Bachelor of Arts Using finite state transducers for multilingual rule-based phonetic transcription Author: Thora Daneyko Matrikelnr.: 3822667 Supervisor: Prof. Gerhard Jäger Eberhard Karls Universität Tübingen Philosphische Fakultät Seminar für Sprachwissenschaft Eidesstattliche Erklärung Hiermit versichere ich, dass ich die Arbeit selbständig verfasst, keine anderen als die angegebenen Hilfsmittel und Quellen benutzt, alle wörtlich oder sinngemäß aus anderen Werken übernommenen Aussagen als solche gekennzeichnet habe und dass die Arbeit weder vollständig noch in wesentlichen Teilen Gegenstand eines anderen Prüfungsverfahrens gewesen ist und dass die Arbeit weder voll- ständig noch in wesentlichen Teilen bereits veröffentlicht wurde sowie dass das in Dateiform eingereichte Exemplar mit den eingereichten gebundenen Exemplaren übereinstimmt. Tübingen, den Thora Daneyko Abstract EVOLAEMP is a current research project on comparative historical linguistics which is compiling a large lexical database called NorthEuraLex covering most languages of Northern Eurasia. To be able to compare the lexemes of these languages with their different writing systems and orthographies, NorthEuraLex aims to provide phonetic transcription in IPA for all entries. Because expert transcriptions are hard to obtain for smaller languages, automatic rule-based transcriptors have been developed. However, the current system is quite inefficient, resulting in bad performance especially on longer words. Thus, I develop and test a revision of this imple- mentation using finite state transducers (FST), which have previously proven to be very suitable for modeling phonological processes. The new implementation should perform faster and be able to operate on the system-independent rule sets already developed for 107 languages. The FST interface used in my system is the Helsinki Finite State Toolkit (HFST).
  • This Document Is Sorted by Transfer Institution Name, Transfer Course

    This Document Is Sorted by Transfer Institution Name, Transfer Course

    TRANSFER COURSE EQUIVALENCIES AS OF 5/2/2018 This document is sorted by Transfer Institution Name, Transfer Course Subject, and finally by JCU Equivaleny Course Subject Transfer Transfer Course JCU Equivalent JCU Equivalent JCU EquSCIalent Course Effective Institution Course Transfer Course Title JCU Equivalent Course Title Number Course Subject Course Number Attrutes (CORE) Term Subject Academy Of The Canyons FRNCH 101 ELEMENTARY FRENCH I FR 101 BEGINNING FRENCH 1 I 201610 Academy Of The Canyons HLHSCI 100 HEALTH EDUCATION EPA 2XX EXERCISE SCIENCE ELECTIVE 201610 Adrian College ACCT 203 PRIN OF ACCOUNTING I AC 201 ACCOUNTING PRINCIPLES 1 201610 Adrian College CHEM 105 GENERAL CHEMISTRY CH 141 GENERAL CHEMISTRY 1 SCI 201610 Adrian College CHEM 117 INTRO CHEMISTRY LAB I CH 143 GENERAL CHEMISTRY LAB 1 SCI 201610 Adrian College COMM 102 PRIN & PRAC OF PUBLIC SPEAKING CO 125 SPEECH COMMUNICATION 201610 Adrian College COMM 109 TELEVISION & RADIO ANNOUNCING CO 215 FUNDAMENTALS OF MEDIA PERFORM CA 201610 Adrian College MATH 115 PRE-CALCULUS GE 1XX GENERAL ELECTIVE 201610 Adrian College MLCS 102 SPANISH II SP 102 BEGINNING SPANISH 2 201610 Adrian College MUS 140 ADRIAN COLLEGE CHOIR FA 1XX FINE ARTS ELECTIVE CA 201610 Ajman Univ of Sci & Technology BASIC 102110 ISLAMIC CULTURE TRS 2XX THEOL & RELIGIOUS STUDIES ELEC 201625 Al Ma'arifaa International Sch CH 9CH01 AS LEVEL CHEMISTRY CH 142 GENERAL CHEMISTRY 2 SCI 201625 Al Ma'arifaa International Sch CH 9CH01 AS LEVEL CHEMISTRY CH 141 GENERAL CHEMISTRY 1 SCI 201625 Alabama State University CMS 205
  • A Java Tool for Compiling Finite-State Transducers from Full-Form Lexicons

    A Java Tool for Compiling Finite-State Transducers from Full-Form Lexicons

    Proceedings of Symposium in Information and Human Language Technology. Natal, RN, Brazil, November 4–7, 2015. c 2015 Sociedade Brasileira de Computac¸ao.˜ JCLexT: A Java Tool for Compiling Finite-State Transducers from Full-Form Lexicons Leonel F. de Alencar, Philipp B. Costa, Mardonio J. C. França, Alexander Ewart, Katiuscia M. Andrade, Rossana M. C. Andrade* Group of Computer Networks, Software Engineering, and Systems (GREat) – Universidade Federal do Ceará (UFC) Campus do Pici, Bloco 942-A – CEP 60455-760 – Fortaleza – CE – Brazil [email protected] Abstract. JCLexT is a compiler of finite-state transducers from full-form lexicons, this tool seems to be the first Java implementation of such functionality. A comparison between JCLexT and Foma was performed based on extensive data from Portuguese. The main disadvantage of JCLexT is the slower compilation time, in comparison to Foma. However, this is negated by the fact that a large transducer compiled with JCLexT was shown to be 8.6% smaller than the Foma created counterpart. 1. Introduction Finite-state transducers (FSTs) have been the preferred devices used to implement morphological parsers (Trommer, 2004). They store information in a compact manner and allow for quick lookup times, being superior to concurring alternatives. Computing is increasingly moving towards mobile platforms. Due to this, the need for natural language processing tools which are designed for the specifics of this setting has emerged. Alencar et al. (2014), for example, addresses this issue with the proposal of JMorpher, a finite-state morphological parser in Java able to natively run on Android devices. However, the JMorpher tool is limited in its functionality, as it was designed to apply an existing finite-state transducer to an input text.