DOCSLIB.ORG

Principled Procedural Parsing Nicolas Laurent

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Principled Procedural Parsing Nicolas Laurent Principled Procedural Parsing Nicolas Laurent August 2019 Thesis submitted in partial fulfillment of the requirements for the degree of Doctor of Applied Science in Engineering Institute of Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM) Louvain School of Engineering (EPL) Université catholique de Louvain (UCLouvain) Louvain-la-Neuve Belgium Thesis Committee Prof. Kim Mens, Advisor UCLouvain/ICTEAM, Belgium Prof. Charles Pecheur, Chair UCLouvain/ICTEAM, Belgium Prof. Peter Van Roy UCLouvain/ICTEAM Belgium Prof. Anya Helene Bagge UIB/II, Norway Prof. Tijs van der Storm CWI/SWAT & UG, The Netherlands Contents Contents3 1 Introduction7 1.1 Parsing............................7 1.2 Inadequacy: Flexibility Versus Simplicity.........8 1.3 The Best of Both Worlds.................. 10 1.4 The Approach: Principled Procedural Parsing....... 13 1.5 Autumn: Architecture of a Solution............ 15 1.6 Overview & Contributions.................. 17 2 Background 23 2.1 Context Free Grammars (CFGs).............. 23 2.1.1 The CFG Formalism................. 23 2.1.2 CFG Parsing Algorithms.............. 25 2.1.3 Top-Down Parsers.................. 26 2.1.4 Bottom-Up Parsers.................. 30 2.1.5 Chart Parsers..................... 33 2.2 Top-Down Recursive-Descent Ad-Hoc Parsers....... 35 2.3 Parser Combinators..................... 36 2.4 Parsing Expression Grammars (PEGs)........... 39 2.4.1 Expressions, Ordered Choice and Lookahead... 39 2.4.2 PEGs and Recursive-Descent Parsers........ 42 2.4.3 The Single Parse Rule, Greed and (Lack of) Ambi- guity.......................... 43 2.4.4 The PEG Algorithm................. 44 2.4.5 Packrat Parsing.................... 45 2.5 Expression Parsing...................... 47 2.6 Error Reporting........................ 50 2.6.1 Overview....................... 51 2.6.2 The Furthest Error Heuristic............ 52 2.6.3 Associating Errors With Custom Messages.... 53 3 4 CONTENTS 2.6.4 Error Recovery.................... 54 2.7 Further Related Work.................... 59 3 Autumn’s Basics 61 3.1 Introductory Example.................... 62 3.1.1 DSL, rule, parsers and combinators......... 66 3.1.2 Whitespace Handling & String Literals....... 67 3.1.3 lazy and sep ..................... 67 3.1.4 Launching the Parse................. 68 3.2 Parser and Parse ....................... 68 3.3 Building An Abstract Syntax Tree (AST)......... 72 3.3.1 AST-Building Combinators............. 74 3.3.2 Value Stack as Context............... 76 3.4 Beyond Basics........................ 76 3.5 Java 8 Grammar....................... 77 4 Infix Expression Parsing 79 4.1 Outline............................ 80 4.2 Grammatical Encodings of Expression Syntax....... 82 4.3 The Semantics of Left-Recursion.............. 84 4.4 The Semantics of Left-Associativity............. 86 4.4.1 Naive Formulation.................. 87 4.4.2 Ambiguous Recursion................ 87 4.4.3 Restating the Problem................ 88 4.4.4 A Pragmatic Way Out................ 89 4.4.5 Related Work..................... 90 4.5 Transparent Left-Recursion Handling in PEG....... 91 4.5.1 The Left-Recursive Algorithm............ 91 4.5.2 Transparent Left-Recursion in Autumn...... 94 4.5.3 Performance Woes.................. 97 4.6 Automatic Left-Recursion Discovery............ 98 4.7 Expression Clusters...................... 102 4.8 Left-Associativity via Left-Folding............. 106 4.8.1 Syntax Trees and Left Folds............. 106 4.8.2 Left-Folding by Hand................ 109 4.8.3 Left-Folding in Autumn............... 112 4.9 Defining Expression Families................ 113 4.10 Discussion........................... 118 4.11 Related Work......................... 120 5 Principled Stateful Parsing 121 5.1 Context-Sensitive Parsing.................. 122 5.1.1 Recall and Context-Sensitive Features....... 122 CONTENTS 5 5.1.2 Context-Sensitive Grammars............ 123 5.1.3 Context-Sensitivity & Parser Combinators..... 124 5.2 State of The Art....................... 125 5.2.1 Backtracking Semantic Actions........... 125 5.2.2 Data-Dependent Grammars............. 127 5.2.3 Monadic Parsers................... 128 5.2.4 Attribute Grammars................. 129 5.2.5 Unprincipled Stateful Parsing............ 129 5.2.6 Rats!.......................... 130 5.2.7 Marpa and Ruby Slippers.............. 131 5.3 Context Transparency.................... 131 5.4 Intuition............................ 133 5.5 Formalization......................... 135 5.5.1 Parse State...................... 136 5.5.2 Parsers........................ 137 5.5.3 Primitive Operations................. 138 5.5.4 Parser Invocation Semantics............. 140 5.6 Implementation........................ 141 5.6.1 From Theory to Practice.............. 141 5.6.2 Primitive Operations and the Log......... 145 5.6.3 Operationalization and Usability Concerns.... 148 5.6.4 Examples....................... 152 5.6.5 Alternatives...................... 157 5.7 Conclusion.......................... 159 6 Engineering Aspects 161 6.1 Performance Considerations................. 163 6.1.1 The Conspicuous Absence of Exponentiality.... 164 6.1.2 Inefficient Idioms in Simple PEG Parsing..... 164 6.1.3 Packrat Parsing.................... 165 6.1.4 Memoization in Autumn............... 168 6.1.5 Megamorphic Call Sites............... 169 6.2 Performance Comparison.................. 172 6.2.1 Run Times and Hot Spots.............. 173 6.2.2 Virtual Machine Effects............... 177 6.2.3 Memory Footprint.................. 178 6.2.4 Discussion....................... 178 6.3 Lexical Analysis....................... 179 6.3.1 Motivation...................... 180 6.3.2 Autumn’s Lexical Analysis Emulation....... 181 6.4 Error Reporting & Recovery................. 184 6.4.1 Performance & Error Reporting........... 185 6.4.2 Autumn’s Error-Reporting Capabilities...... 185 6 CONTENTS 6.4.3 Custom Error States & Introspection........ 187 6.4.4 Longest-Match Analysis............... 187 6.4.5 Error-Recoverable Parsers.............. 189 6.4.6 Lexical-Level Error Reporting............ 190 6.5 Grammar Traversal & Parser Visitors........... 192 6.5.1 Grammar Traversal................. 192 6.5.2 The Expression Problem............... 193 6.5.3 A Clean But Verbose Partial Solution....... 195 6.5.4 A User-Friendly Partial Solution.......... 198 6.5.5 Well-Formedness Checking Using Built-In Visitors 201 6.5.6 Grammar Transformation Using CopyVisitor ... 202 6.5.7 Abstract Parsers................... 203 6.5.8 Potential Further Applications........... 204 6.6 Debugging & Tracing..................... 206 6.6.1 Parser Invocation Stack Traces........... 206 6.6.2 Custom Parsers for State Inspection........ 208 6.6.3 Testing Support................... 208 6.6.4 Performance Tracing................. 210 6.7 Grammar Composition.................... 212 6.8 Conclusion.......................... 214 7 Conclusions and Future Directions 217 Bibliography 223 Appendices 235 A Autumn Java 8 Grammar 237 Chapter 1 Introduction This thesis is concerned with the limitations of currently available parsing systems, and how to overcome them. In this chapter, we introduce parsing, some of the challenges and trade-offs we perceive in the field, and our approaches to tackle these challenges. 1.1 Parsing Parsing is the process of analysing an input string in order to extract a structured representation of its content (a syntax tree) with respect to a specific language. In the thesis, we focus on parsing formal languages, such as programming or markup languages — as opposed to natural spoken languages. Unlike natural languages, formal languages are never ambiguous: there is only a single correct interpretation of the input. Parsing is a pervasive activity: every time a source file must be turned into executable code, a parser is required. Similarly, parsers are used to convert input files into relevant data structures. It is fair to say that most programs include a parser — sometimes many. As such, making parsers easier to write, use, and modify is a broadly beneficial endeavour. In particular, parsers should be written using a simple yet expressive notation. It should be easy to modify existing parsers, as the language definition may evolve. Finally, the parsers should be able to support diverse practical non-linguistic requirements, such as the generation of intelligible error messages when the input does not conform to expectations, permissive parsing in the presence of such errors, 7 8 CHAPTER 1. INTRODUCTION and the ability to derive various artifacts from the parser specification. Broadly, the practice of parsing can be divided between ad-hoc parsing, where a programmer writes code to implement a parser for a single language; and the use of grammarware [49] — parsing tools that derive from or use a grammar. Grammarware also includes other grammar-based tools, such as syntax highlighters and pretty-printers. Parsing is one of the oldest disciplines in computer science, yet even today, practical challenges and inadequacies remain — whether one uses ad-hoc parsing or grammarware. 1.2 Inadequacy: Flexibility Versus Simplicity Ad-hoc parsers have many advantages: they can be fast and can be cus- tomized to do exactly what the programmer requires. On the other hand, they are verbose and do not constitute a readable language description. Nor is it easy to verify that the parser conforms to an existing description of the language. Additionally, there is very little possibility of reusing the parsing code to perform other tasks that depend on the structure
Load more

Recommended publications
  • Derivatives of Parsing Expression Grammars
    Derivatives of Parsing Expression Grammars Aaron Moss Cheriton School of Computer Science University of Waterloo Waterloo, Ontario, Canada [email protected] This paper introduces a new derivative parsing algorithm for recognition of parsing expression gram- mars. Derivative parsing is shown to have a polynomial worst-case time bound, an improvement on the exponential bound of the recursive descent algorithm. This work also introduces asymptotic analysis based on inputs with a constant bound on both grammar nesting depth and number of back- tracking choices; derivative and recursive descent parsing are shown to run in linear time and constant space on this useful class of inputs, with both the theoretical bounds and the reasonability of the in- put class validated empirically. This common-case constant memory usage of derivative parsing is an improvement on the linear space required by the packrat algorithm. 1 Introduction Parsing expression grammars (PEGs) are a parsing formalism introduced by Ford [6]. Any LR(k) lan- guage can be represented as a PEG [7], but there are some non-context-free languages that may also be represented as PEGs (e.g. anbncn [7]). Unlike context-free grammars (CFGs), PEGs are unambiguous, admitting no more than one parse tree for any grammar and input. PEGs are a formalization of recursive descent parsers allowing limited backtracking and infinite lookahead; a string in the language of a PEG can be recognized in exponential time and linear space using a recursive descent algorithm, or linear time and space using the memoized packrat algorithm [6]. PEGs are formally defined and these algo- rithms outlined in Section 3.
    [Show full text]
  • Ece351 Lab Manual
    DEREK RAYSIDE & ECE351 STAFF ECE351 LAB MANUAL UNIVERSITYOFWATERLOO 2 derek rayside & ece351 staff Copyright © 2014 Derek Rayside & ECE351 Staff Compiled March 6, 2014 acknowledgements: • Prof Paul Ward suggested that we look into something with vhdl to have synergy with ece327. • Prof Mark Aagaard, as the ece327 instructor, consulted throughout the development of this material. • Prof Patrick Lam generously shared his material from the last offering of ece251. • Zhengfang (Alex) Duanmu & Lingyun (Luke) Li [1b Elec] wrote solutions to most labs in txl. • Jiantong (David) Gao & Rui (Ray) Kong [3b Comp] wrote solutions to the vhdl labs in antlr. • Aman Muthrej and Atulan Zaman [3a Comp] wrote solutions to the vhdl labs in Parboiled. • TA’s Jon Eyolfson, Vajih Montaghami, Alireza Mortezaei, Wenzhu Man, and Mohammed Hassan. • TA Wallace Wu developed the vhdl labs. • High school students Brian Engio and Tianyu Guo drew a number of diagrams for this manual, wrote Javadoc comments for the code, and provided helpful comments on the manual. Licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) version 2.5 or greater. http://creativecommons.org/licenses/by-sa/2.5/ca/ http://creativecommons.org/licenses/by-sa/3.0/ Contents 0 Overview 9 Compiler Concepts: call stack, heap 0.1 How the Labs Fit Together . 9 Programming Concepts: version control, push, pull, merge, SSH keys, IDE, 0.2 Learning Progressions . 11 debugger, objects, pointers 0.3 How this project compares to CS241, the text book, etc. 13 0.4 Student work load . 14 0.5 How this course compares to MIT 6.035 .......... 15 0.6 Where do I learn more? .
    [Show full text]
  • Scheme of Teaching and Examination for BE
    Scheme of Teaching and Examination for B.E (CS&E) SEMESTER: III Sl. Subject Course Title Teaching Credits Contact Marks Exam No. Code Department Hours Duration in hrs L T P TOTAL CIE SEE Total 1 MA310 Mathematics III Mathematics 4 0 0 4 4 50 50 100 03 2 CS310 Digital System CSE 4 0 1 5 6 50 50 100 03 Design 3 CS320 Discrete CSE 4 0 0 4 4 50 50 100 03 Mathematical Structures and Combinatorics 4 CS330 Computer CSE 4 0 0 4 4 50 50 100 03 Organization 5 CS340 Data Structures CSE 4 0 1 5 6 50 50 100 03 6 CS350 Object Oriented CSE 4 0 1 5 6 50 50 100 03 Programming with C++ Total Total 27 Total Marks 600 Credits Scheme of Teaching and Examination for B.E (CS&E) SEMESTER: IV Sl. Subject Course Title Teaching Credits Contact Marks Exam No. Code Department Hours Duration in hrs L T P TOTAL CIE SEE Total 1 MA410 Probability, Mathematics 4 0 0 4 4 50 50 100 03 Statistics and Queuing 2 CS410 Operating CSE 4 0 1 5 6 50 50 100 03 Systems 3 CS420 Design and CSE 4 0 1 5 6 50 50 100 03 Analysis of Algorithms 4 CS430 Theory of CSE 4 0 0 4 4 50 50 100 03 Computation 5 CS440 Microprocessors CSE 4 0 1 5 6 50 50 100 03 6 CS450 Data CSE 4 0 0 4 4 50 50 100 03 Communication Total Total 27 Total Marks 600 Credits Scheme of Teaching and Examination for B.E (CS&E) SEMESTER: V Sl.
    [Show full text]
  • Validating LR(1) Parsers
    Validating LR(1) Parsers Jacques-Henri Jourdan1;2, Fran¸coisPottier2, and Xavier Leroy2 1 Ecole´ Normale Sup´erieure 2 INRIA Paris-Rocquencourt Abstract. An LR(1) parser is a finite-state automaton, equipped with a stack, which uses a combination of its current state and one lookahead symbol in order to determine which action to perform next. We present a validator which, when applied to a context-free grammar G and an automaton A, checks that A and G agree. Validating the parser pro- vides the correctness guarantees required by verified compilers and other high-assurance software that involves parsing. The validation process is independent of which technique was used to construct A. The validator is implemented and proved correct using the Coq proof assistant. As an application, we build a formally-verified parser for the C99 language. 1 Introduction Parsing remains an essential component of compilers and other programs that input textual representations of structured data. Its theoretical foundations are well understood today, and mature technology, ranging from parser combinator libraries to sophisticated parser generators, is readily available to help imple- menting parsers. The issue we focus on in this paper is that of parser correctness: how to obtain formal evidence that a parser is correct with respect to its specification? Here, following established practice, we choose to specify parsers via context-free grammars enriched with semantic actions. One application area where the parser correctness issue naturally arises is formally-verified compilers such as the CompCert verified C compiler [14]. In- deed, in the current state of CompCert, the passes that have been formally ver- ified start at abstract syntax trees (AST) for the CompCert C subset of C and extend to ASTs for three assembly languages.
    [Show full text]
  • LATE Ain't Earley: a Faster Parallel Earley Parser
    LATE Ain’T Earley: A Faster Parallel Earley Parser Peter Ahrens John Feser Joseph Hui [email protected] [email protected] [email protected] July 18, 2018 Abstract We present the LATE algorithm, an asynchronous variant of the Earley algorithm for pars- ing context-free grammars. The Earley algorithm is naturally task-based, but is difficult to parallelize because of dependencies between the tasks. We present the LATE algorithm, which uses additional data structures to maintain information about the state of the parse so that work items may be processed in any order. This property allows the LATE algorithm to be sped up using task parallelism. We show that the LATE algorithm can achieve a 120x speedup over the Earley algorithm on a natural language task. 1 Introduction Improvements in the efficiency of parsers for context-free grammars (CFGs) have the potential to speed up applications in software development, computational linguistics, and human-computer interaction. The Earley parser has an asymptotic complexity that scales with the complexity of the CFG, a unique, desirable trait among parsers for arbitrary CFGs. However, while the more commonly used Cocke-Younger-Kasami (CYK) [2, 5, 12] parser has been successfully parallelized [1, 7], the Earley algorithm has seen relatively few attempts at parallelization. Our research objectives were to understand when there exists parallelism in the Earley algorithm, and to explore methods for exploiting this parallelism. We first tried to naively parallelize the Earley algorithm by processing the Earley items in each Earley set in parallel. We found that this approach does not produce any speedup, because the dependencies between Earley items force much of the work to be performed sequentially.
    [Show full text]
  • CS 432 Fall 2020 Top-Down (LL) Parsing
    CS 432 Fall 2020 Mike Lam, Professor Top-Down (LL) Parsing Compilation Current focus "Back end" Source code Tokens Syntax tree Machine code char data[20]; 7f 45 4c 46 01 int main() { 01 01 00 00 00 float x 00 00 00 00 00 = 42.0; ... return 7; } Lexing Parsing Code Generation & Optimization "Front end" Review ● Recognize regular languages with finite automata – Described by regular expressions – Rule-based transitions, no memory required ● Recognize context-free languages with pushdown automata – Described by context-free grammars – Rule-based transitions, MEMORY REQUIRED ● Add a stack! Segue KEY OBSERVATION: Allowing the translator to use memory to track parse state information enables a wider range of automated machine translation. Chomsky Hierarchy of Languages Recursively enumerable Context-sensitive Context-free Most useful Regular for PL https://en.wikipedia.org/wiki/Chomsky_hierarchy Parsing Approaches ● Top-down: begin with start symbol (root of parse tree), and gradually expand non-terminals – Stack contains leaves that still need to be expanded ● Bottom-up: begin with terminals (leaves of parse tree), and gradually connect using non-terminals – Stack contains roots of subtrees that still need to be connected A V = E Top-down a E + E Bottom-up V V b c Top-Down Parsing root = createNode(S) focus = root A → V = E push(null) V → a | b | c token = nextToken() E → E + E loop: | V if (focus is non-terminal): B = chooseRuleAndExpand(focus) for each b in B.reverse(): focus.addChild(createNode(b)) push(b) A focus = pop() else if (token
    [Show full text]
  • Total Parser Combinators
    Total Parser Combinators Nils Anders Danielsson School of Computer Science, University of Nottingham, United Kingdom [email protected] Abstract The library has an interface which is very similar to those of A monadic parser combinator library which guarantees termination classical monadic parser combinator libraries. For instance, con- of parsing, while still allowing many forms of left recursion, is sider the following simple, left recursive, expression grammar: described. The library’s interface is similar to those of many other term :: factor term '+' factor parser combinator libraries, with two important differences: one is factor ::D atom j factor '*' atom that the interface clearly specifies which parts of the constructed atom ::D numberj '(' term ')' parsers may be infinite, and which parts have to be finite, using D j dependent types and a combination of induction and coinduction; We can define a parser which accepts strings from this grammar, and the other is that the parser type is unusually informative. and also computes the values of the resulting expressions, as fol- The library comes with a formal semantics, using which it is lows (the combinators are described in Section 4): proved that the parser combinators are as expressive as possible. mutual The implementation is supported by a machine-checked correct- term factor ness proof. D ] term >> λ n j D 1 ! Categories and Subject Descriptors D.1.1 [Programming Tech- tok '+' >> λ D ! niques]: Applicative (Functional) Programming; E.1 [Data Struc- factor >> λ n D 2 ! tures]; F.3.1 [Logics and Meanings of Programs]: Specifying and return .n n / 1 C 2 Verifying and Reasoning about Programs; F.4.2 [Mathematical factor atom Logic and Formal Languages]: Grammars and Other Rewriting D ] factor >> λ n Systems—Grammar types, Parsing 1 j tok '*' >>D λ ! D ! General Terms Languages, theory, verification atom >> λ n2 return .n n / D ! 1 ∗ 2 Keywords Dependent types, mixed induction and coinduction, atom number parser combinators, productivity, termination D tok '(' >> λ j D ! ] term >> λ n 1.
    [Show full text]
  • CWI Scanprofile/PDF/300
    Centrum voor Wiskunde en lnformatica Centre for Mathematics and Computer Science J. Heering, P. Klint, J.G. Rekers Incremental generation of parsers , Computer Science/Department of Software Technology Report CS-R8822 May Biblk>tlleek Centrum ypor Wisl~unde en lnformatk:a Am~tel>dam The Centre for Mathematics and Computer Science is a research institute of the Stichting Mathematisch Centrum, which was founded on February 11, 1946, as a nonprofit institution aim­ ing at the promotion of mathematics, computer science, and their applications. It is sponsored by the Dutch Government through the Netherlands Organization for the Advancement of Pure Research (Z.W.0.). q\ ' Copyright (t:: Stichting Mathematisch Centrum, Amsterdam 1 Incremental Generation of Parsers J. Heering Department of Software Technology, Centre for Mathematics and Computer Science P.O. Box 4079, 1009 AS Amsterdam, The Netherlands P. Klint Department of Software Technology, Centre for Mathematics and Computer Science P.O. Box 4079, 1009 AS Amsterdam, The Netherlands and Programming Research Group, University of Amsterdam P.O. BOX 41882, 1009 DB Amsterdam, The Netherlands J. Rekers Department of Software Technology, Centre for Mathematics and Computer Science P.O. Box 4079, 1009 AB Amsterdam, The Netherlands A parser checks whether a text is a sentence in a language. Therefore, the parser is provided with the grammar of the language, and it usually generates a structure (parse tree) that represents the text according to that grammar. Most present-day parsers are not directly driven by the grammar but by a 'parse table', which is generated by a parse table generator. A table based parser wolks more efficiently than a grammar based parser does, and provided that the parser is used often enough, the cost of gen­ erating the parse table is outweighed by the gain in parsing efficiency.
    [Show full text]
  • Lecture 10: CYK and Earley Parsers Alvin Cheung Building Parse Trees Maaz Ahmad CYK and Earley Algorithms Talia Ringer More Disambiguation
    Hack Your Language! CSE401 Winter 2016 Introduction to Compiler Construction Ras Bodik Lecture 10: CYK and Earley Parsers Alvin Cheung Building Parse Trees Maaz Ahmad CYK and Earley algorithms Talia Ringer More Disambiguation Ben Tebbs 1 Announcements • HW3 due Sunday • Project proposals due tonight – No late days • Review session this Sunday 6-7pm EEB 115 2 Outline • Last time we saw how to construct AST from parse tree • We will now discuss algorithms for generating parse trees from input strings 3 Today CYK parser builds the parse tree bottom up More Disambiguation Forcing the parser to select the desired parse tree Earley parser solves CYK’s inefficiency 4 CYK parser Parser Motivation • Given a grammar G and an input string s, we need an algorithm to: – Decide whether s is in L(G) – If so, generate a parse tree for s • We will see two algorithms for doing this today – Many others are available – Each with different tradeoffs in time and space 6 CYK Algorithm • Parsing algorithm for context-free grammars • Invented by John Cocke, Daniel Younger, and Tadao Kasami • Basic idea given string s with n tokens: 1. Find production rules that cover 1 token in s 2. Use 1. to find rules that cover 2 tokens in s 3. Use 2. to find rules that cover 3 tokens in s 4. … N. Use N-1. to find rules that cover n tokens in s. If succeeds then s is in L(G), else it is not 7 A graphical way to visualize CYK Initial graph: the input (terminals) Repeat: add non-terminal edges until no more can be added.
    [Show full text]
  • Simple, Efficient, Sound-And-Complete Combinator Parsing for All Context
    Simple, efficient, sound-and-complete combinator parsing for all context-free grammars, using an oracle OCaml 2014 workshop, talk proposal Tom Ridge University of Leicester, UK [email protected] This proposal describes a parsing library that is based on current grammar, the start symbol and the input to the back-end Earley work due to be published as [3]. The talk will focus on the OCaml parser, which parses the input and returns a parsing oracle. The library and its usage. oracle is then used to guide the action phase of the parser. 1. Combinator parsers 4. Real-world performance 3 Parsers for context-free grammars can be implemented directly and In addition to O(n ) theoretical performance, we also optimize our naturally in a functional style known as “combinator parsing”, us- back-end Earley parser. Extensive performance measurements on ing recursion following the structure of the grammar rules. Tradi- small grammars (described in [3]) indicate that the performance is tionally parser combinators have struggled to handle all features of arguably better than the Happy parser generator [1]. context-free grammars, such as left recursion. 5. Online distribution 2. Sound and complete combinator parsers This work has led to the P3 parsing library for OCaml, freely Previous work [2] introduced novel parser combinators that could available online1. The distribution includes extensive examples. be used to parse all context-free grammars. A parser generator built Some of the examples are: using these combinators was proved both sound and complete in the • HOL4 theorem prover. Parsing and disambiguation of arithmetic expressions using This previous approach has all the traditional benefits of combi- precedence annotations.
    [Show full text]
  • Improving Upon Earley's Parsing Algorithm in Prolog
    Improving Upon Earley’s Parsing Algorithm In Prolog Matt Voss Artificial Intelligence Center University of Georgia Athens, GA 30602 May 7, 2004 Abstract This paper presents a modification of the Earley (1970) parsing algorithm in Prolog. The Earley algorithm presented here is based on an implementation in Covington (1994a). The modifications are meant to improve on that algorithm in several key ways. The parser features a predictor that works like a left-corner parser with links, thus decreasing the number of chart entries. It implements subsump- tion checking, and organizes chart entries to take advantage of first argument indexing for quick retrieval. 1 Overview The Earley parsing algorithm is well known for its great efficiency in produc- ing all possible parses of a sentence in relatively little time, without back- tracking, and while handling left recursive rules correctly. It is also known for being considerably slower than most other parsing algorithms1. This pa- per outlines an attempt to overcome many of the pitfalls associated with implementations of Earley parsers. It uses the Earley parser in Covington (1994a) as a starting point. In particular the Earley top-down predictor is exchanged for a predictor that works like a left-corner predictor with links, 1See Covington (1994a) for one comparison of run times for a variety of algorithms. 1 following Leiss (1990). Chart entries store the positions of constituents in the input string, rather than lists containing the constituents themselves, and the arguments of chart entries are arranged to take advantage of first argument indexing. This means a small trade-off between easy-to-read code and efficiency of processing.
    [Show full text]
  • Staged Parser Combinators for Efficient Data Processing
    Staged Parser Combinators for Efficient Data Processing Manohar Jonnalagedda∗ Thierry Coppeyz Sandro Stucki∗ Tiark Rompf y∗ Martin Odersky∗ ∗LAMP zDATA, EPFL {first.last}@epfl.ch yOracle Labs: {first.last}@oracle.com Abstract use the language’s abstraction capabilities to enable compo- Parsers are ubiquitous in computing, and many applications sition. As a result, a parser written with such a library can depend on their performance for decoding data efficiently. look like formal grammar descriptions, and is also readily Parser combinators are an intuitive tool for writing parsers: executable: by construction, it is well-structured, and eas- tight integration with the host language enables grammar ily maintainable. Moreover, since combinators are just func- specifications to be interleaved with processing of parse re- tions in the host language, it is easy to combine them into sults. Unfortunately, parser combinators are typically slow larger, more powerful combinators. due to the high overhead of the host language abstraction However, parser combinators suffer from extremely poor mechanisms that enable composition. performance (see Section 5) inherent to their implementa- We present a technique for eliminating such overhead. We tion. There is a heavy penalty to be paid for the expres- use staging, a form of runtime code generation, to dissoci- sivity that they allow. A grammar description is, despite its ate input parsing from parser composition, and eliminate in- declarative appearance, operationally interleaved with input termediate data structures and computations associated with handling, such that parts of the grammar description are re- parser composition at staging time. A key challenge is to built over and over again while input is processed.
    [Show full text]