LL(*): the Foundation of the ANTLR Parser Generator DRAFT

Total Page:16

File Type:pdf, Size:1020Kb

LL(*): the Foundation of the ANTLR Parser Generator DRAFT LL(*): The Foundation of the ANTLR Parser Generator DRAFT Accepted to PLDI 2011 Terence Parr Kathleen S. Fisher University of San Francisco AT&T Labs Research [email protected] kfi[email protected] Abstract In the \top-down" world, Ford introduced Packrat parsers Despite the power of Parser Expression Grammars (PEGs) and the associated Parser Expression Grammars (PEGs) [5, and GLR, parsing is not a solved problem. Adding nonde- 6]. PEGs preclude only the use of left-recursive grammar terminism (parser speculation) to traditional LL and LR rules. Packrat parsers are backtracking parsers that attempt parsers can lead to unexpected parse-time behavior and in- the alternative productions in the order specified. The first troduces practical issues with error handling, single-step de- production that matches at an input position wins. Pack- bugging, and side-effecting embedded grammar actions. This rat parsers are linear rather than exponential because they paper introduces the LL(*) parsing strategy and an asso- memoize partial results, ensuring input states will never be ciated grammar analysis algorithm that constructs LL(*) parsed by the same production more than once. The Rats! parsing decisions from ANTLR grammars. At parse-time, [7] PEG-based tool vigorously optimizes away memoization decisions gracefully throttle up from conventional fixed k ≥ events to improve speed and reduce the memory footprint. 1 lookahead to arbitrary lookahead and, finally, fail over to A significant advantage of both GLR and PEG parser backtracking depending on the complexity of the parsing de- generators is that they accept any grammar that conforms cision and the input symbols. LL(*) parsing strength reaches to their meta-language (except left-recursive PEGs). Pro- into the context-sensitive languages, in some cases beyond grammers no longer have to wade through reams of conflict what GLR and PEGs can express. By statically removing as messages. Despite this advantage, neither GLR nor PEG much speculation as possible, LL(*) provides the expressiv- parsers are completely satisfactory, for a number of reasons. ity of PEGs while retaining LL's good error handling and First, GLR and PEG parsers do not always do what unrestricted grammar actions. Widespread use of ANTLR was intended. GLR silently accepts ambiguous grammars, (over 70,000 downloads/year) shows that it is effective for a those that match the same input in multiple ways, forcing wide variety of applications. programmers to detect ambiguities dynamically. PEGs have no concept of a grammar conflict because they always choose the “first” interpretation, which can lead to unexpected or 1. Introduction inconvenient behavior. For example, the second production of PEG rule A ! ajab (meaning \A matches either a Parsing is not a solved problem, despite its importance and or ab") will never be used. Input ab never matches the long history of academic study. Because it is tedious and second alternative since the first symbol, a, matches the error-prone to write parsers by hand, researchers have spent first alternative. In a large grammar, such hazards are not decades studying how to generate efficient parsers from high- always obvious and even experienced developers can miss level grammars. Despite this effort, parser generators still them without exhaustive testing. suffer from problems of expressiveness and usability. Second, debugging nondeterministic parsers can be very When parsing theory was originally developed, machine difficult. With bottom-up parsing, the state usually repre- resources were scarce, and so parser efficiency was the sents multiple locations within the grammar, making it dif- paramount concern. In that era, it made sense to force ficult for programmers to predict what will happen next. programmers to contort their grammars to fit the con- Top-down parsers are easier to understand because there is straints of LALR(1) or LL(1) parser generators. In con- a one-to-one mapping from LL grammar elements to parser trast, modern computers are so fast that programmer ef- operations. Further, recursive-descent LL implementations ficiency is now more important. In response to this de- allow programmers to use standard source-level debuggers velopment, researchers have developed more powerful, but to step through parsers and embedded actions, facilitat- more costly, nondeterministic parsing strategies following ing understanding. This advantage is weakened significantly, both the \bottom-up" approach (LR-style parsing) and the however, for backtracking recursive-descent packrat parsers. \top-down" approach (LL-style parsing). Nested backtracking is very difficult to follow! In the \bottom-up" world, Generalized LR (GLR) [16] Third, generating high-quality error messages in nonde- parsers parse in linear to cubic time, depending on how terministic parsers is difficult but very important to com- closely the grammar conforms to classic LR. GLR essentially mercial developers. Providing good syntax error support re- \forks" new subparsers to pursue all possible actions ema- lies on parser context. For example, to recover well from an nating from nondeterministic LR states, terminating any invalid expression, a parser needs to know if it is parsing subparsers that lead to invalid parses. The result is a parse an array index or, say, an assignment. In the first case, the forest with all possible interpretations of the input. Elkhound parser should resynchronize by skipping ahead to a ] token. [10] is a very efficient GLR implementation that achieves In the second case, it should skip to a ; token. Top-down yacc-like parsing speeds when grammars are LALR(1). Pro- parsers have a rule invocation stack and can report things grammers unfamiliar with LALR parsing theory, though, like \invalid expression in array index." Bottom-up parsers, can easily get nonlinear GLR parsers. on the other hand, only know for sure that they are match- speculating parsers, which means it supports source-level de- ing an expression. They are typically less able to deal well bugging, produces high-quality error messages, and allows with erroneous input. Packrat parsers also have ambiguous programmers to embed arbitrary actions. A survey of 89 context since they are always speculating. In fact, they can- ANTLR grammars [1] available from sorceforce.net and not recover from syntax errors because they cannot detect code.google.com reveals that 75% of them had embedded errors until they have seen the entire input. actions, counting conservatively, which reveals that such ac- Finally, nondeterministic parsing strategies cannot easily tions are a useful feature in the ANTLR community. support arbitrary, embedded grammar actions, which are Widespread use shows that LL(*) fits within the pro- useful for manipulating symbol tables, constructing data grammer comfort zone and is effective for a wide vari- structures, etc. Speculating parsers cannot execute side- ety of language applications. ANTLR 3.x has been down- effecting actions like print statements, since the speculated loaded 41,364 (binary jar file) + 62,086 (integrated into action may never really take place. Even side-effect free ANTLRworks) + 31,126 (source code) = 134,576 times ac- actions such as those that compute rule return values can be cording to Google Analytics (unique downloads January 9, awkward in GLR parsers [10]. For example, since the parser 2008 - October 28, 2010). Projects using ANTLR include can match the same rule in multiple ways, it might have Google App Engine (Python), IBM Tivoli Identity Man- to execute multiple competing actions. (Should it merge all ager, BEA/Oracle WebLogic, Yahoo! Query Language, Ap- results somehow or just pick one?) GLR and PEG tools ple XCode IDE, Apple Keynote, Oracle SQL Developer IDE, address this issue by either disallowing actions, disallowing Sun/Oracle JavaFX language, and NetBeans IDE. arbitrary actions, or relying on the programmer to avoid This paper is organized as follows. We first introduce side-effects in actions that could be executed speculatively. ANTLR grammars by example (Section 2). Next we for- mally define predicated grammars and a special subclass 1.1 ANTLR called predicated LL-regular grammars (Section 3). We then describe LL(*) parsers (Section 4), which implement pars- This paper describes version 3.3 of the ANTLR parser gen- ing decisions for predicated LL-regular grammars. Next, we erator and its underlying top-down parsing strategy, called give an algorithm that builds lookahead DFA from ANTLR LL(*), that address these deficiencies. The input to ANTLR grammars (Section 5). Finally, we support our claims regard- is a context-free grammar augmented with syntactic [14] and ing LL(*) efficiency and reduced speculation (Section 6). semantic predicates and embedded actions. Syntactic pred- icates allow arbitrary lookahead, while semantic predicates allow the state constructed up to the point of a predicate to 2. Introduction to LL(*) direct the parse. Syntactic predicates are given as a gram- In this section, we give an intuition for LL(*) parsing by ex- mar fragment that must match the following input. Semantic plaining how it works for two ANTLR grammar fragments predicates are given as arbitrary Boolean-valued code in the constructed to illustrate the algorithm. Consider nontermi- host language of the parser. Actions are written in the host- nal s, which uses the (omitted) nonterminal expr to match language of the parser and have access to the current state. arithmetic expressions. As with PEGs, ANTLR requires programmers to avoid
Recommended publications
  • Parser Tables for Non-LR(1) Grammars with Conflict Resolution Joel E
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Elsevier - Publisher Connector Science of Computer Programming 75 (2010) 943–979 Contents lists available at ScienceDirect Science of Computer Programming journal homepage: www.elsevier.com/locate/scico The IELR(1) algorithm for generating minimal LR(1) parser tables for non-LR(1) grammars with conflict resolution Joel E. Denny ∗, Brian A. Malloy School of Computing, Clemson University, Clemson, SC 29634, USA article info a b s t r a c t Article history: There has been a recent effort in the literature to reconsider grammar-dependent software Received 17 July 2008 development from an engineering point of view. As part of that effort, we examine a Received in revised form 31 March 2009 deficiency in the state of the art of practical LR parser table generation. Specifically, LALR Accepted 12 August 2009 sometimes generates parser tables that do not accept the full language that the grammar Available online 10 September 2009 developer expects, but canonical LR is too inefficient to be practical particularly during grammar development. In response, many researchers have attempted to develop minimal Keywords: LR parser table generation algorithms. In this paper, we demonstrate that a well known Grammarware Canonical LR algorithm described by David Pager and implemented in Menhir, the most robust minimal LALR LR(1) implementation we have discovered, does not always achieve the full power of Minimal LR canonical LR(1) when the given grammar is non-LR(1) coupled with a specification for Yacc resolving conflicts. We also detail an original minimal LR(1) algorithm, IELR(1) (Inadequacy Bison Elimination LR(1)), which we have implemented as an extension of GNU Bison and which does not exhibit this deficiency.
    [Show full text]
  • Backtrack Parsing Context-Free Grammar Context-Free Grammar
    Context-free Grammar Problems with Regular Context-free Grammar Language and Is English a regular language? Bad question! We do not even know what English is! Two eggs and bacon make(s) a big breakfast Backtrack Parsing Can you slide me the salt? He didn't ought to do that But—No! Martin Kay I put the wine you brought in the fridge I put the wine you brought for Sandy in the fridge Should we bring the wine you put in the fridge out Stanford University now? and University of the Saarland You said you thought nobody had the right to claim that they were above the law Martin Kay Context-free Grammar 1 Martin Kay Context-free Grammar 2 Problems with Regular Problems with Regular Language Language You said you thought nobody had the right to claim [You said you thought [nobody had the right [to claim that they were above the law that [they were above the law]]]] Martin Kay Context-free Grammar 3 Martin Kay Context-free Grammar 4 Problems with Regular Context-free Grammar Language Nonterminal symbols ~ grammatical categories Is English mophology a regular language? Bad question! We do not even know what English Terminal Symbols ~ words morphology is! They sell collectables of all sorts Productions ~ (unordered) (rewriting) rules This concerns unredecontaminatability Distinguished Symbol This really is an untiable knot. But—Probably! (Not sure about Swahili, though) Not all that important • Terminals and nonterminals are disjoint • Distinguished symbol Martin Kay Context-free Grammar 5 Martin Kay Context-free Grammar 6 Context-free Grammar Context-free
    [Show full text]
  • Unravel Data Systems Version 4.5
    UNRAVEL DATA SYSTEMS VERSION 4.5 Component name Component version name License names jQuery 1.8.2 MIT License Apache Tomcat 5.5.23 Apache License 2.0 Tachyon Project POM 0.8.2 Apache License 2.0 Apache Directory LDAP API Model 1.0.0-M20 Apache License 2.0 apache/incubator-heron 0.16.5.1 Apache License 2.0 Maven Plugin API 3.0.4 Apache License 2.0 ApacheDS Authentication Interceptor 2.0.0-M15 Apache License 2.0 Apache Directory LDAP API Extras ACI 1.0.0-M20 Apache License 2.0 Apache HttpComponents Core 4.3.3 Apache License 2.0 Spark Project Tags 2.0.0-preview Apache License 2.0 Curator Testing 3.3.0 Apache License 2.0 Apache HttpComponents Core 4.4.5 Apache License 2.0 Apache Commons Daemon 1.0.15 Apache License 2.0 classworlds 2.4 Apache License 2.0 abego TreeLayout Core 1.0.1 BSD 3-clause "New" or "Revised" License jackson-core 2.8.6 Apache License 2.0 Lucene Join 6.6.1 Apache License 2.0 Apache Commons CLI 1.3-cloudera-pre-r1439998 Apache License 2.0 hive-apache 0.5 Apache License 2.0 scala-parser-combinators 1.0.4 BSD 3-clause "New" or "Revised" License com.springsource.javax.xml.bind 2.1.7 Common Development and Distribution License 1.0 SnakeYAML 1.15 Apache License 2.0 JUnit 4.12 Common Public License 1.0 ApacheDS Protocol Kerberos 2.0.0-M12 Apache License 2.0 Apache Groovy 2.4.6 Apache License 2.0 JGraphT - Core 1.2.0 (GNU Lesser General Public License v2.1 or later AND Eclipse Public License 1.0) chill-java 0.5.0 Apache License 2.0 Apache Commons Logging 1.2 Apache License 2.0 OpenCensus 0.12.3 Apache License 2.0 ApacheDS Protocol
    [Show full text]
  • Adaptive LL(*) Parsing: the Power of Dynamic Analysis
    Adaptive LL(*) Parsing: The Power of Dynamic Analysis Terence Parr Sam Harwell Kathleen Fisher University of San Francisco University of Texas at Austin Tufts University [email protected] [email protected] kfi[email protected] Abstract PEGs are unambiguous by definition but have a quirk where Despite the advances made by modern parsing strategies such rule A ! a j ab (meaning “A matches either a or ab”) can never as PEG, LL(*), GLR, and GLL, parsing is not a solved prob- match ab since PEGs choose the first alternative that matches lem. Existing approaches suffer from a number of weaknesses, a prefix of the remaining input. Nested backtracking makes de- including difficulties supporting side-effecting embedded ac- bugging PEGs difficult. tions, slow and/or unpredictable performance, and counter- Second, side-effecting programmer-supplied actions (muta- intuitive matching strategies. This paper introduces the ALL(*) tors) like print statements should be avoided in any strategy that parsing strategy that combines the simplicity, efficiency, and continuously speculates (PEG) or supports multiple interpreta- predictability of conventional top-down LL(k) parsers with the tions of the input (GLL and GLR) because such actions may power of a GLR-like mechanism to make parsing decisions. never really take place [17]. (Though DParser [24] supports The critical innovation is to move grammar analysis to parse- “final” actions when the programmer is certain a reduction is time, which lets ALL(*) handle any non-left-recursive context- part of an unambiguous final parse.) Without side effects, ac- free grammar. ALL(*) is O(n4) in theory but consistently per- tions must buffer data for all interpretations in immutable data forms linearly on grammars used in practice, outperforming structures or provide undo actions.
    [Show full text]
  • On the Relation Between Context-Free Grammars And
    On the Relation between Context-Free Grammars and Parsing Expression Grammars Fabio Mascarenhas Department of Computer Science – UFRJ – Rio de Janeiro – Brazil S´ergio Medeiros School of Science and Technology – UFRN – Natal – Brazil Roberto Ierusalimschy Department of Computer Science – PUC-Rio – Rio de Janeiro – Brazil Abstract Context-Free Grammars (CFGs) and Parsing Expression Grammars (PEGs) have several similarities and a few differences in both their syntax and seman- tics, but they are usually presented through formalisms that hinder a proper comparison. In this paper we present a new formalism for CFGs that highlights the similarities and differences between them. The new formalism borrows from PEGs the use of parsing expressions and the recognition-based semantics. We show how one way of removing non-determinism from this formalism yields a formalism with the semantics of PEGs. We also prove, based on these new formalisms, how LL(1) grammars define the same language whether interpreted as CFGs or as PEGs, and also show how strong-LL(k), right-linear, and LL- regular grammars have simple language-preserving translations from CFGs to PEGs. Once these classes of CFGs can be automatically translated to equivalent PEGs, we can reuse classic top-down grammars in PEG-based tools. Keywords: Context-free grammars, parsing expression grammars, parsing, LL(1), LL(k), LL-regular, right-linear grammars, natural semantics arXiv:1304.3177v2 [cs.FL] 13 Feb 2014 1. Introduction Context-Free Grammars (CFGs) are the formalism of choice for describing the syntax of programming languages. A CFG describes a language as the set Email addresses: [email protected] (Fabio Mascarenhas), [email protected] (S´ergio Medeiros), [email protected] (Roberto Ierusalimschy) Preprint submitted to Science of Computer Programming March 19, 2018 of strings generated from the grammar’s initial symbol by a sequence of rewrit- ing steps.
    [Show full text]
  • The Design & Implementation of an Abstract Semantic Graph For
    Clemson University TigerPrints All Dissertations Dissertations 12-2011 The esiD gn & Implementation of an Abstract Semantic Graph for Statement-Level Dynamic Analysis of C++ Applications Edward Duffy Clemson University, [email protected] Follow this and additional works at: https://tigerprints.clemson.edu/all_dissertations Part of the Computer Sciences Commons Recommended Citation Duffy, Edward, "The eD sign & Implementation of an Abstract Semantic Graph for Statement-Level Dynamic Analysis of C++ Applications" (2011). All Dissertations. 832. https://tigerprints.clemson.edu/all_dissertations/832 This Dissertation is brought to you for free and open access by the Dissertations at TigerPrints. It has been accepted for inclusion in All Dissertations by an authorized administrator of TigerPrints. For more information, please contact [email protected]. THE DESIGN &IMPLEMENTATION OF AN ABSTRACT SEMANTIC GRAPH FOR STATEMENT-LEVEL DYNAMIC ANALYSIS OF C++ APPLICATIONS A Dissertation Presented to the Graduate School of Clemson University In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy Computer Science by Edward B. Duffy December 2011 Accepted by: Dr. Brian A. Malloy, Committee Chair Dr. James B. von Oehsen Dr. Jason P. Hallstrom Dr. Pradip K. Srimani In this thesis, we describe our system, Hylian, for statement-level analysis, both static and dynamic, of a C++ application. We begin by extending the GNU gcc parser to generate parse trees in XML format for each of the compilation units in a C++ application. We then provide verification that the generated parse trees are structurally equivalent to the code in the original C++ application. We use the generated parse trees, together with an augmented version of the gcc test suite, to recover a grammar for the C++ dialect that we parse.
    [Show full text]
  • Parsing Expression Grammars: a Recognition-Based Syntactic Foundation
    Parsing Expression Grammars: A Recognition-Based Syntactic Foundation Bryan Ford Massachusetts Institute of Technology Cambridge, MA [email protected] Abstract 1 Introduction For decades we have been using Chomsky’s generative system of Most language syntax theory and practice is based on generative grammars, particularly context-free grammars (CFGs) and regu- systems, such as regular expressions and context-free grammars, in lar expressions (REs), to express the syntax of programming lan- which a language is defined formally by a set of rules applied re- guages and protocols. The power of generative grammars to ex- cursively to generate strings of the language. A recognition-based press ambiguity is crucial to their original purpose of modelling system, in contrast, defines a language in terms of rules or predi- natural languages, but this very power makes it unnecessarily diffi- cates that decide whether or not a given string is in the language. cult both to express and to parse machine-oriented languages using Simple languages can be expressed easily in either paradigm. For CFGs. Parsing Expression Grammars (PEGs) provide an alterna- example, fs 2 a∗ j s = (aa)ng is a generative definition of a trivial tive, recognition-based formal foundation for describing machine- language over a unary character set, whose strings are “constructed” oriented syntax, which solves the ambiguity problem by not intro- by concatenating pairs of a’s. In contrast, fs 2 a∗ j (jsj mod 2 = 0)g ducing ambiguity in the first place. Where CFGs express nondeter- is a recognition-based definition of the same language, in which a ministic choice between alternatives, PEGs instead use prioritized string of a’s is “accepted” if its length is even.
    [Show full text]
  • Parsing Based on N-Path Tree-Controlled Grammars
    Theoretical and Applied Informatics ISSN 1896–5334 Vol.23 (2011), no. 3-4 pp. 213–228 DOI: 10.2478/v10179-011-0015-7 Parsing Based on n-Path Tree-Controlled Grammars MARTIN Cˇ ERMÁK,JIR͡ KOUTNÝ,ALEXANDER MEDUNA Formal Model Research Group Department of Information Systems Faculty of Information Technology Brno University of Technology Božetechovaˇ 2, 612 66 Brno, Czech Republic icermak, ikoutny, meduna@fit.vutbr.cz Received 15 November 2011, Revised 1 December 2011, Accepted 4 December 2011 Abstract: This paper discusses recently introduced kind of linguistically motivated restriction placed on tree-controlled grammars—context-free grammars with some root-to-leaf paths in their derivation trees restricted by a control language. We deal with restrictions placed on n ¸ 1 paths controlled by a deter- ministic context–free language, and we recall several basic properties of such a rewriting system. Then, we study the possibilities of corresponding parsing methods working in polynomial time and demonstrate that some non-context-free languages can be generated by this regulated rewriting model. Furthermore, we illustrate the syntax analysis of LL grammars with controlled paths. Finally, we briefly discuss how to base parsing methods on bottom-up syntax–analysis. Keywords: regulated rewriting, derivation tree, tree-controlled grammars, path-controlled grammars, parsing, n-path tree-controlled grammars 1. Introduction The investigation of context-free grammars with restricted derivation trees repre- sents an important trend in today’s formal language theory (see [3], [6], [10], [11], [12], [13] and [15]). In essence, these grammars generate their languages just like ordinary context-free grammars do, but their derivation trees have to satisfy some simple pre- scribed conditions.
    [Show full text]
  • Sequence Alignment/Map Format Specification
    Sequence Alignment/Map Format Specification The SAM/BAM Format Specification Working Group 3 Jun 2021 The master version of this document can be found at https://github.com/samtools/hts-specs. This printing is version 53752fa from that repository, last modified on the date shown above. 1 The SAM Format Specification SAM stands for Sequence Alignment/Map format. It is a TAB-delimited text format consisting of a header section, which is optional, and an alignment section. If present, the header must be prior to the alignments. Header lines start with `@', while alignment lines do not. Each alignment line has 11 mandatory fields for essential alignment information such as mapping position, and variable number of optional fields for flexible or aligner specific information. This specification is for version 1.6 of the SAM and BAM formats. Each SAM and BAMfilemay optionally specify the version being used via the @HD VN tag. For full version history see Appendix B. Unless explicitly specified elsewhere, all fields are encoded using 7-bit US-ASCII 1 in using the POSIX / C locale. Regular expressions listed use the POSIX / IEEE Std 1003.1 extended syntax. 1.1 An example Suppose we have the following alignment with bases in lowercase clipped from the alignment. Read r001/1 and r001/2 constitute a read pair; r003 is a chimeric read; r004 represents a split alignment. Coor 12345678901234 5678901234567890123456789012345 ref AGCATGTTAGATAA**GATAGCTGTGCTAGTAGGCAGTCAGCGCCAT +r001/1 TTAGATAAAGGATA*CTG +r002 aaaAGATAA*GGATA +r003 gcctaAGCTAA +r004 ATAGCT..............TCAGC -r003 ttagctTAGGC -r001/2 CAGCGGCAT The corresponding SAM format is:2 1Charset ANSI X3.4-1968 as defined in RFC1345.
    [Show full text]
  • Lexing and Parsing with ANTLR4
    Lab 2 Lexing and Parsing with ANTLR4 Objective • Understand the software architecture of ANTLR4. • Be able to write simple grammars and correct grammar issues in ANTLR4. EXERCISE #1 Lab preparation Ï In the cap-labs directory: git pull will provide you all the necessary files for this lab in TP02. You also have to install ANTLR4. 2.1 User install for ANTLR4 and ANTLR4 Python runtime User installation steps: mkdir ~/lib cd ~/lib wget http://www.antlr.org/download/antlr-4.7-complete.jar pip3 install antlr4-python3-runtime --user Then in your .bashrc: export CLASSPATH=".:$HOME/lib/antlr-4.7-complete.jar:$CLASSPATH" export ANTLR4="java -jar $HOME/lib/antlr-4.7-complete.jar" alias antlr4="java -jar $HOME/lib/antlr-4.7-complete.jar" alias grun='java org.antlr.v4.gui.TestRig' Then source your .bashrc: source ~/.bashrc 2.2 Structure of a .g4 file and compilation Links to a bit of ANTLR4 syntax : • Lexical rules (extended regular expressions): https://github.com/antlr/antlr4/blob/4.7/doc/ lexer-rules.md • Parser rules (grammars) https://github.com/antlr/antlr4/blob/4.7/doc/parser-rules.md The compilation of a given .g4 (for the PYTHON back-end) is done by the following command line: java -jar ~/lib/antlr-4.7-complete.jar -Dlanguage=Python3 filename.g4 or if you modified your .bashrc properly: antlr4 -Dlanguage=Python3 filename.g4 2.3 Simple examples with ANTLR4 EXERCISE #2 Demo files Ï Work your way through the five examples in the directory demo_files: Aurore Alcolei, Laure Gonnord, Valentin Lorentz. 1/4 ENS de Lyon, Département Informatique, M1 CAP Lab #2 – Automne 2017 ex1 with ANTLR4 + Java : A very simple lexical analysis1 for simple arithmetic expressions of the form x+3.
    [Show full text]
  • Parsing Techniques
    Dick Grune, Ceriel J.H. Jacobs Parsing Techniques 2nd edition — Monograph — September 27, 2007 Springer Berlin Heidelberg NewYork Hong Kong London Milan Paris Tokyo Contents Preface to the Second Edition : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : v Preface to the First Edition : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : xi 1 Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1 1.1 Parsing as a Craft. 2 1.2 The Approach Used . 2 1.3 Outline of the Contents . 3 1.4 The Annotated Bibliography . 4 2 Grammars as a Generating Device : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 5 2.1 Languages as Infinite Sets . 5 2.1.1 Language . 5 2.1.2 Grammars . 7 2.1.3 Problems with Infinite Sets . 8 2.1.4 Describing a Language through a Finite Recipe . 12 2.2 Formal Grammars . 14 2.2.1 The Formalism of Formal Grammars . 14 2.2.2 Generating Sentences from a Formal Grammar . 15 2.2.3 The Expressive Power of Formal Grammars . 17 2.3 The Chomsky Hierarchy of Grammars and Languages . 19 2.3.1 Type 1 Grammars . 19 2.3.2 Type 2 Grammars . 23 2.3.3 Type 3 Grammars . 30 2.3.4 Type 4 Grammars . 33 2.3.5 Conclusion . 34 2.4 Actually Generating Sentences from a Grammar. 34 2.4.1 The Phrase-Structure Case . 34 2.4.2 The CS Case . 36 2.4.3 The CF Case . 36 2.5 To Shrink or Not To Shrink . 38 xvi Contents 2.6 Grammars that Produce the Empty Language . 41 2.7 The Limitations of CF and FS Grammars . 42 2.7.1 The uvwxy Theorem . 42 2.7.2 The uvw Theorem .
    [Show full text]
  • Creating Custom DSL Editor for Eclipse Ian Graham Markit Risk Analytics
    Speaking Your Language Creating a custom DSL editor for Eclipse Ian Graham 1 Speaking Your Language | © 2012 Ian Graham Background highlights 2D graphics, Fortran, Pascal, C Unix device drivers/handlers – trackball, display devices Smalltalk Simulation Sensor “data fusion” electronic warfare test-bed for army CASE tools for telecommunications Java Interactive visual tools for QC of seismic processing Seismic processing DSL development tools Defining parameter constraints and doc of each seismic process in XML Automating help generation (plain text and HTML) Smart editor that leverages DSL definition Now at Markit using custom language for financial risk analysis Speaking Your Language | © 2012 Ian Graham Overview Introduction: Speaking Your Language What is a DSL? What is Eclipse? Demo: Java Editor Eclipse architecture Implementing an editor Implementing a simple editor Demo Adding language awareness to your editor Demo Learning from Eclipse examples Resources Speaking Your Language | © 2012 Ian Graham Introduction: Speaking Your Language Goal illustrate power of Eclipse platform Context widening use of Domain Specific Languages Focus editing support for custom languages in Eclipse Speaking Your Language | © 2012 Ian Graham Recipe Editor 5 Text Editor Recipes | © 2006 IBM | Made available under the EPL v1.0 What is a DSL? Domain Specific Language - a simplified language for specific problem domain Specialized programming language E.g. Excel formula, SQL, seismic processing script Specification language
    [Show full text]