DOCSLIB.ORG

Non-Deterministic Recursive Ascent Parsing

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Non-Deterministic Recursive Ascent Parsing Non-deterministic Recursive Ascent Parsing Ren~ Leermakers Philips Research Laboratories, P.O. Box 80.000, 5600 JA Eindhoven, The Netherlands E-mail:[email protected] ABSTRACT this reason. Our presentation, however, contains a sur- prisingly simple correctness proof. In fact, this proof is A purely functional implementation of LR-parsers is this paper's major contribution to parsing theory. One given, together with a simple correctness proof. It is of its lessons is that the CF grammar class is often the presented as a generalization of the recursive descent natural one to proof parsers for, even if these parsers are parser. For non-LR grammars the time-complexity of devoted to some special class of grammars. If the gram- our parser is cubic if the functions that constitute the marlis restricted in some way, a parser for general CF parser are implemented as memo-functions, i.e. func- grammars may have properties that enable smart imple- tions that memorize the results of previous invocations. mentation tricks to enhance efficiency. As we show be- Memo-functions also facilitate a simple way to construct low, the relation between LR-parsers and LR-grammars a very compact representation of the parse forest. For is of this kind. LR(0) grammars, our algorithm is closely related to the Especially in natural language processing, standard recursive ascent parsers recently discovered by Kruse- CF grammars are often too limited in their strong gen- man Aretz [1] and Roberts [2]. Extended CF grammars erative power. The extended CF grammar formalism, (grammars with regular expressions at the right hand allowing rules to have regular expressions at the right side) can be parsed with a simple modification of the hand side, is a useful extension, for that reason. It is not LR-parser for normal CF grammars. difficult to generalize our parser to cope with extended grammars, although the application of LR-parsing to extended CF grammars is well-known to be problematic 1 Introduction [5]. We first present the recursive descent recognizer in In this paper we give a purely functional implementa- a way that allows the desired generalization. Then we tion of LR-parsers, applicable to general CF grammars. obtain the recursive ascent recognizer and its proof. If It will be obtained as a generalization of the well-known the grammar is LR(0) a few implementation tricks lead recursive descent parsing technique. For LR(0) gram- to the recursive ascent recognizer of ref. [1]. Subse- mars, our result implies a deterministic parser that is quently, the time and space complexities of the recog- closely related to the recursive ascent parsers discovered nizer are analysed, and the algorithm for constructing by Kruseman Aretz [1] and Roberts [2]. In the gen- a cubic representation for parse forests is given. The eral non-deterministic case, the parser has cubic time paper ends with a discussion of extended CF grammars. complexity if the parse functions are implemented as memo-functions [3], which are functions that memorize and re-use the results of previous invocations. Memo- 2 Recursive descent functions are easily implemented in most programming languages. The notion of memo-functions is also used Consider CF grammar G, with terminals VT and non- to define an algorithm that constructs a cubic represen- terminals V/v. Let V = VN U VT. A well-known top- tation for the parse forest, i.e. the collection of parse down parsing technique is the recursive descent parser. trees. Recursive descent parsers consist of a number of pro- It has been claimed by Tomita that non-deterministic cedures, usually one for each non-terminal. Here we LR-parsers are useful for natural language processing. present a variant that consists of functions, one for each In [4] he presented a discussion about how to do non- item (dotted rule). We use the unorthodox embracing deterministic LR-parsing, with a device called a graph- operator [.] to map each item to its function: (we use structured stack. With our parser we show that no ex- greek letters for arbitrary elements of V*) plicit stack manipulations are needed; they can be ex- pressed implicitly with the use of appropriate program- [A -~ a.~] : N --. 2 N ming language concepts. where N is the set of integers, or a subset (0...nm~x), Most textbooks on parsing do not include proper with nma= the maximum seutence length. The functions correctness proofs for LR-parsers, mainly because such are to meet the following specification: proofs tend to be rather involved. The theory of LR- parsing should still be considered underdeveloped, for [A --, a. l(0 = {Jl -*" - 63 - with x~...xn the sentence to be parsed. A recursive im- with B E V, and 1/31 the number of symbols in 3 (with plementation for these functions is given by (b • VT, B • H = 0). A recursive ascent recognizer may be obtained v,,) by relating to each state q not only the above [q], but [A --* a.](i) = {i} also a function__ that we take to be the result of applying operator [.] to the state: [a --* a.b-r](i) = {jib = zi+, ^ j E [A ~ ab.-r](i + 1)} [q] : V x N --* 2 I° xN [A ---, a.B-r](i) = {Jl~ • [B ~ .~](i)^j • [A -; ~a.-r](~)} It has the specification We keep to the custom of omitting existential quantifi- [q](B,i) = {(A --* a.3,j)lA --* a.3 e qA cation (here for k,/f) in definitions of this kind. 3 =~* B-rAT ---,* xi+a...xj} The proof is elementary and ba#ed on For i >.nn (n is the sentence length) it follows that [q](i) = [q](B,i) = $, whereas for i _< n the functions 3~(3 = x~+a-r A -r ~* zi+~...~:s)V are recursively implemented by 3B-~$k(3 = B-r ^ B --~ 8 A 8 --;* zi+a...x~^ [q](i) = [q](x,+l, i + 1)u -r --~* 2;k+l...2~j) {(1, j)IB --* .e e ini(q)A (l, j) E-~(B, i)}U If we add a grammar rule S' --* S to G, with S' ([ V {(l,i)lI • q ^ final(l)} then S --** x~...xn is equivalent to n • [S' --* .S](0). [q](B, i) = { (pop(l), J)l The recursive descent recognizer works for any CF (1,j) • [ooto(q, B)](i)^ pop(l) • q}U grammar except for grammars for which ~A~(A ---* {(I,4)1(J, k) • [goto(q, B)I~^ aAcr --** A3). For such left-recursive grammars the rec- pop(J) • ini(q) ^ (1, j) • [q](lhs(S), k)} ognizer does not terminate, as execution of [A --* .a](i) Proof: will lead to a call of itself. The recognition is not a linear First we notice that process in general: the function calls [A --- a.B3"](i) lead to calls [B --* ./i](i) for all values of ~ such that B ---, /8 "** xi+l..-xj is a grammar rule. 3~(3 ~* zi+l't ^ 7 ~" z,+2...zj)v 3B~(3 ~" B-r ^ B ~ c ^ -y --." z,+~...zj)v (~=~^i=j) 3 The ascent recognizer Hence One way to make the recognizer more deterministic is by [q](i) = combining functions corresponding to a number of com- {(A --* a.3,J)l(A --* a.3, j) • r~(z,+~, i+ 1)}u peting items into one function. Let the set of all items {(A --, ,~.3, J)l of G be given by In. Subsets of I6; are called states, and B -.-. eA(A --, a.3,j) • [q](B,i)}u we use q to be an arbitrary state, lWe associate to each {(A --~ a.,i)la --* a. • q} state q a function, re-using the above operator [.], This is equivalent to the earlier version because we may [q] : N ~ 2 I°×N replace the clause B ~ e by B ---, .e • ini(q). Indeed, if state q has item A --* a.fl and if there is a left-most- that meets the specification symbol derivation/3 =~* B-r then all items B --* .A are [q](i) ---- {(A -- a.3,j)l A --. a.3 • q ^ 3 --*" zi+~...xi} included in ini(q). As above, the function reports which parts of the sen- For establishing the correctness of [q--] notice that tence can be derived. But as the function is associated 3 ~* B3" either contains zero steps, in which case to a set q of items, it has to do so for each item in 3 = B'r, or it contains at least one step: q. If we define the initial state q0 = {S' --* .S}, now 3.y(3 =~* B3" A 3' --*" xi+a ...zs) = S --," xl...xn is equivalent to (S' ---* .S,n) • [q0](0). 3~(3 = B-r ^ -r --" xi+l...zj)V Before proceeding, we need a couple of definitions. 3ce.~k(~8 :=~* C-rAG --* B~S A~5 -.*" xi+ l...x~,A -r -'** Let ini(q) be the set of initial items for state q, that xk+l ...x j) are derived from q by the closure operation: Hence [q](B, i) may be written as the union of two sets, ini(q) = { B --* .AIB -. A ^ A --* a.3 • q A 3 =¢ B-r}. [q](B, i) = So USa: The double arrow =¢, denotes a left-most-symbol rewrit- So = {(A --~ a.B3",j)] ing Ba =e~ Cfla, using a non-e rule B ---, Cfl. The A ---. ct.B3" • qA-r ---** xs+l...xj} transition function goto is defined by (B • V) S~ = {(a --. a.3,j)lA --* a.3 • q ^ 3 =~" C-r^ goto(q, B) = {A -* aB.3]A --* a.B3 • (q U ini(q))} C ---* B~ ^ $ --** zi+l...xk ^ 3' --*" zk+l...zi}.
Load more

Recommended publications
  • Derivatives of Parsing Expression Grammars
    Derivatives of Parsing Expression Grammars Aaron Moss Cheriton School of Computer Science University of Waterloo Waterloo, Ontario, Canada [email protected] This paper introduces a new derivative parsing algorithm for recognition of parsing expression gram- mars. Derivative parsing is shown to have a polynomial worst-case time bound, an improvement on the exponential bound of the recursive descent algorithm. This work also introduces asymptotic analysis based on inputs with a constant bound on both grammar nesting depth and number of back- tracking choices; derivative and recursive descent parsing are shown to run in linear time and constant space on this useful class of inputs, with both the theoretical bounds and the reasonability of the in- put class validated empirically. This common-case constant memory usage of derivative parsing is an improvement on the linear space required by the packrat algorithm. 1 Introduction Parsing expression grammars (PEGs) are a parsing formalism introduced by Ford [6]. Any LR(k) lan- guage can be represented as a PEG [7], but there are some non-context-free languages that may also be represented as PEGs (e.g. anbncn [7]). Unlike context-free grammars (CFGs), PEGs are unambiguous, admitting no more than one parse tree for any grammar and input. PEGs are a formalization of recursive descent parsers allowing limited backtracking and infinite lookahead; a string in the language of a PEG can be recognized in exponential time and linear space using a recursive descent algorithm, or linear time and space using the memoized packrat algorithm [6]. PEGs are formally defined and these algo- rithms outlined in Section 3.
    [Show full text]
  • A Typed, Algebraic Approach to Parsing
    A Typed, Algebraic Approach to Parsing Neelakantan R. Krishnaswami Jeremy Yallop University of Cambridge University of Cambridge United Kingdom United Kingdom [email protected] [email protected] Abstract the subject have remained remarkably stable: context-free In this paper, we recall the definition of the context-free ex- languages are specified in BackusśNaur form, and parsers pressions (or µ-regular expressions), an algebraic presenta- for these specifications are implemented using algorithms tion of the context-free languages. Then, we define a core derived from automata theory. This integration of theory and type system for the context-free expressions which gives a practice has yielded many benefits: we have linear-time algo- compositional criterion for identifying those context-free ex- rithms for parsing unambiguous grammars efficiently [Knuth pressions which can be parsed unambiguously by predictive 1965; Lewis and Stearns 1968] and with excellent error re- algorithms in the style of recursive descent or LL(1). porting for bad input strings [Jeffery 2003]. Next, we show how these typed grammar expressions can However, in languages with good support for higher-order be used to derive a parser combinator library which both functions (such as ML, Haskell, and Scala) it is very popular guarantees linear-time parsing with no backtracking and to use parser combinators [Hutton 1992] instead. These li- single-token lookahead, and which respects the natural de- braries typically build backtracking recursive descent parsers notational semantics of context-free expressions. Finally, we by composing higher-order functions. This allows building show how to exploit the type information to write a staged up parsers with the ordinary abstraction features of the version of this library, which produces dramatic increases programming language (variables, data structures and func- in performance, even outperforming code generated by the tions), which is sufficiently beneficial that these libraries are standard parser generator tool ocamlyacc.
    [Show full text]
  • A Declarative Extension of Parsing Expression Grammars for Recognizing Most Programming Languages
    Journal of Information Processing Vol.24 No.2 256–264 (Mar. 2016) [DOI: 10.2197/ipsjjip.24.256] Regular Paper A Declarative Extension of Parsing Expression Grammars for Recognizing Most Programming Languages Tetsuro Matsumura1,†1 Kimio Kuramitsu1,a) Received: May 18, 2015, Accepted: November 5, 2015 Abstract: Parsing Expression Grammars are a popular foundation for describing syntax. Unfortunately, several syn- tax of programming languages are still hard to recognize with pure PEGs. Notorious cases appears: typedef-defined names in C/C++, indentation-based code layout in Python, and HERE document in many scripting languages. To recognize such PEG-hard syntax, we have addressed a declarative extension to PEGs. The “declarative” extension means no programmed semantic actions, which are traditionally used to realize the extended parsing behavior. Nez is our extended PEG language, including symbol tables and conditional parsing. This paper demonstrates that the use of Nez Extensions can realize many practical programming languages, such as C, C#, Ruby, and Python, which involve PEG-hard syntax. Keywords: parsing expression grammars, semantic actions, context-sensitive syntax, and case studies on program- ming languages invalidate the declarative property of PEGs, thus resulting in re- 1. Introduction duced reusability of grammars. As a result, many developers need Parsing Expression Grammars [5], or PEGs, are a popu- to redevelop grammars for their software engineering tools. lar foundation for describing programming language syntax [6], In this paper, we propose a declarative extension of PEGs for [15]. Indeed, the formalism of PEGs has many desirable prop- recognizing context-sensitive syntax. The “declarative” exten- erties, including deterministic behaviors, unlimited look-aheads, sion means no arbitrary semantic actions that are written in a gen- and integrated lexical analysis known as scanner-less parsing.
    [Show full text]
  • Adaptive LL(*) Parsing: the Power of Dynamic Analysis
    Adaptive LL(*) Parsing: The Power of Dynamic Analysis Terence Parr Sam Harwell Kathleen Fisher University of San Francisco University of Texas at Austin Tufts University [email protected] [email protected] kfi[email protected] Abstract PEGs are unambiguous by definition but have a quirk where Despite the advances made by modern parsing strategies such rule A ! a j ab (meaning “A matches either a or ab”) can never as PEG, LL(*), GLR, and GLL, parsing is not a solved prob- match ab since PEGs choose the first alternative that matches lem. Existing approaches suffer from a number of weaknesses, a prefix of the remaining input. Nested backtracking makes de- including difficulties supporting side-effecting embedded ac- bugging PEGs difficult. tions, slow and/or unpredictable performance, and counter- Second, side-effecting programmer-supplied actions (muta- intuitive matching strategies. This paper introduces the ALL(*) tors) like print statements should be avoided in any strategy that parsing strategy that combines the simplicity, efficiency, and continuously speculates (PEG) or supports multiple interpreta- predictability of conventional top-down LL(k) parsers with the tions of the input (GLL and GLR) because such actions may power of a GLR-like mechanism to make parsing decisions. never really take place [17]. (Though DParser [24] supports The critical innovation is to move grammar analysis to parse- “final” actions when the programmer is certain a reduction is time, which lets ALL(*) handle any non-left-recursive context- part of an unambiguous final parse.) Without side effects, ac- free grammar. ALL(*) is O(n4) in theory but consistently per- tions must buffer data for all interpretations in immutable data forms linearly on grammars used in practice, outperforming structures or provide undo actions.
    [Show full text]
  • Efficient Recursive Parsing
    Purdue University Purdue e-Pubs Department of Computer Science Technical Reports Department of Computer Science 1976 Efficient Recursive Parsing Christoph M. Hoffmann Purdue University, [email protected] Report Number: 77-215 Hoffmann, Christoph M., "Efficient Recursive Parsing" (1976). Department of Computer Science Technical Reports. Paper 155. https://docs.lib.purdue.edu/cstech/155 This document has been made available through Purdue e-Pubs, a service of the Purdue University Libraries. Please contact [email protected] for additional information. EFFICIENT RECURSIVE PARSING Christoph M. Hoffmann Computer Science Department Purdue University West Lafayette, Indiana 47907 CSD-TR 215 December 1976 Efficient Recursive Parsing Christoph M. Hoffmann Computer Science Department Purdue University- Abstract Algorithms are developed which construct from a given LL(l) grammar a recursive descent parser with as much, recursion resolved by iteration as is possible without introducing auxiliary memory. Unlike other proposed methods in the literature designed to arrive at parsers of this kind, the algorithms do not require extensions of the notational formalism nor alter the grammar in any way. The algorithms constructing the parsers operate in 0(k«s) steps, where s is the size of the grammar, i.e. the sum of the lengths of all productions, and k is a grammar - dependent constant. A speedup of the algorithm is possible which improves the bound to 0(s) for all LL(l) grammars, and constructs smaller parsers with some auxiliary memory in form of parameters
    [Show full text]
  • Conflict Resolution in a Recursive Descent Compiler Generator
    LL(1) Conflict Resolution in a Recursive Descent Compiler Generator Albrecht Wöß, Markus Löberbauer, Hanspeter Mössenböck Johannes Kepler University Linz, Institute of Practical Computer Science, Altenbergerstr. 69, 4040 Linz, Austria {woess,loeberbauer,moessenboeck}@ssw.uni-linz.ac.at Abstract. Recursive descent parsing is restricted to languages whose grammars are LL(1), i.e., which can be parsed top-down with a single lookahead symbol. Unfortunately, many languages such as Java, C++, or C# are not LL(1). There- fore recursive descent parsing cannot be used or the parser has to make its deci- sions based on semantic information or a multi-symbol lookahead. In this paper we suggest a systematic technique for resolving LL(1) conflicts in recursive descent parsing and show how to integrate it into a compiler gen- erator (Coco/R). The idea is to evaluate user-defined boolean expressions, in order to allow the parser to make its parsing decisions where a one symbol loo- kahead does not suffice. Using our extended compiler generator we implemented a compiler front end for C# that can be used as a framework for implementing a variety of tools. 1 Introduction Recursive descent parsing [16] is a popular top-down parsing technique that is sim- ple, efficient, and convenient for integrating semantic processing. However, it re- quires the grammar of the parsed language to be LL(1), which means that the parser must always be able to select between alternatives with a single symbol lookahead. Unfortunately, many languages such as Java, C++ or C# are not LL(1) so that one either has to resort to bottom-up LALR(1) parsing [5, 1], which is more powerful but less convenient for semantic processing, or the parser has to resolve the LL(1) con- flicts using semantic information or a multi-symbol lookahead.
    [Show full text]
  • A Parsing Machine for Pegs
    A Parsing Machine for PEGs ∗ Sérgio Medeiros Roberto Ierusalimschy [email protected] [email protected] Department of Computer Science PUC-Rio, Rio de Janeiro, Brazil ABSTRACT parsing approaches. The PEG formalism gives a convenient Parsing Expression Grammar (PEG) is a recognition-based syntax for describing top-down parsers for unambiguous lan- foundation for describing syntax that renewed interest in guages. The parsers it describes can parse strings in linear top-down parsing approaches. Generally, the implementa- time, despite backtracking, by using a memoizing algorithm tion of PEGs is based on a recursive-descent parser, or uses called Packrat [1]. Although Packrat has guaranteed worst- a memoization algorithm. case linear time complexity, it also has linear space com- We present a new approach for implementing PEGs, based plexity, with a rather large constant. This makes Packrat on a virtual parsing machine, which is more suitable for impractical for parsing large amounts of data. pattern matching. Each PEG has a corresponding program LPEG [7, 11] is a pattern-matching tool for Lua, a dy- that is executed by the parsing machine, and new programs namically typed scripting language [6, 8]. LPEG uses PEGs are dynamically created and composed. The virtual machine to describe patterns instead of the more popular Perl-like is embedded in a scripting language and used by a pattern- "regular expressions" (regexes). Regexes are a more ad-hoc matching tool. way of describing patterns; writing a complex regex is more We give an operational semantics of PEGs used for pat- a trial and error than a systematic process, and their per- tern matching, then describe our parsing machine and its formance is very unpredictable.
    [Show full text]
  • Advanced Parsing Techniques
    Advanced Parsing Techniques Announcements ● Written Set 1 graded. ● Hard copies available for pickup right now. ● Electronic submissions: feedback returned later today. Where We Are Where We Are Parsing so Far ● We've explored five deterministic parsing algorithms: ● LL(1) ● LR(0) ● SLR(1) ● LALR(1) ● LR(1) ● These algorithms all have their limitations. ● Can we parse arbitrary context-free grammars? Why Parse Arbitrary Grammars? ● They're easier to write. ● Can leave operator precedence and associativity out of the grammar. ● No worries about shift/reduce or FIRST/FOLLOW conflicts. ● If ambiguous, can filter out invalid trees at the end. ● Generate candidate parse trees, then eliminate them when not needed. ● Practical concern for some languages. ● We need to have C and C++ compilers! Questions for Today ● How do you go about parsing ambiguous grammars efficiently? ● How do you produce all possible parse trees? ● What else can we do with a general parser? The Earley Parser Motivation: The Limits of LR ● LR parsers use shift and reduce actions to reduce the input to the start symbol. ● LR parsers cannot deterministically handle shift/reduce or reduce/reduce conflicts. ● However, they can nondeterministically handle these conflicts by guessing which option to choose. ● What if we try all options and see if any of them work? The Earley Parser ● Maintain a collection of Earley items, which are LR(0) items annotated with a start position. ● The item A → α·ω @n means we are working on recognizing A → αω, have seen α, and the start position of the item was the nth token. ● Using techniques similar to LR parsing, try to scan across the input creating these items.
    [Show full text]
  • Verifiable Parse Table Composition for Deterministic Parsing*
    Verifiable Parse Table Composition for Deterministic Parsing? August Schwerdfeger and Eric Van Wyk Department of Computer Science and Engineering University of Minnesota, Minneapolis, MN {schwerdf,evw}@cs.umn.edu Abstract. One obstacle to the implementation of modular extensions to programming languages lies in the problem of parsing extended lan- guages. Specifically, the parse tables at the heart of traditional LALR(1) parsers are so monolithic and tightly constructed that, in the general case, it is impossible to extend them without regenerating them from the source grammar. Current extensible frameworks employ a variety of solutions, ranging from a full regeneration to using pluggable binary mod- ules for each different extension. But recompilation is time-consuming, while the pluggable modules in many cases cannot support the addition of more than one extension, or use backtracking or non-deterministic parsing techniques. We present here a middle-ground approach that allows an extension, if it meets certain restrictions, to be compiled into a parse table fragment. The host language parse table and fragments from multiple extensions can then always be efficiently composed to produce a conflict-free parse table for the extended language. This allows for the distribution of de- terministic parsers for extensible languages in a pre-compiled format, eliminating the need for the “source code” grammar to be distributed. In practice, we have found these restrictions to be reasonable and admit many useful language extensions. 1 Introduction In parsing programming languages, the usual practice is to generate a single parser for the language to be parsed. A well known and often-used approach is LR parsing [1] which relies on a process, sometimes referred to as grammar compilation, to generate a monolithic parse table representing the grammar being parsed.
    [Show full text]
  • Syntactic Analysis, Or Parsing, Is the Second Phase of Compilation: the Token File Is Converted to an Abstract Syntax Tree
    Syntactic Analysis Syntactic analysis, or parsing, is the second phase of compilation: The token file is converted to an abstract syntax tree. Analysis Synthesis Compiler Passes of input program of output program (front -end) (back -end) character stream Intermediate Lexical Analysis Code Generation token intermediate stream form Syntactic Analysis Optimization abstract intermediate syntax tree form Semantic Analysis Code Generation annotated target AST language 1 Syntactic Analysis / Parsing • Goal: Convert token stream to abstract syntax tree • Abstract syntax tree (AST): – Captures the structural features of the program – Primary data structure for remainder of analysis • Three Part Plan – Study how context-free grammars specify syntax – Study algorithms for parsing / building ASTs – Study the miniJava Implementation Context-free Grammars • Compromise between – REs, which can’t nest or specify recursive structure – General grammars, too powerful, undecidable • Context-free grammars are a sweet spot – Powerful enough to describe nesting, recursion – Easy to parse; but also allow restrictions for speed • Not perfect – Cannot capture semantics, as in, “variable must be declared,” requiring later semantic pass – Can be ambiguous • EBNF, Extended Backus Naur Form, is popular notation 2 CFG Terminology • Terminals -- alphabet of language defined by CFG • Nonterminals -- symbols defined in terms of terminals and nonterminals • Productions -- rules for how a nonterminal (lhs) is defined in terms of a (possibly empty) sequence of terminals
    [Show full text]
  • Lecture 3: Recursive Descent Limitations, Precedence Climbing
    Lecture 3: Recursive descent limitations, precedence climbing David Hovemeyer September 9, 2020 601.428/628 Compilers and Interpreters Today I Limitations of recursive descent I Precedence climbing I Abstract syntax trees I Supporting parenthesized expressions Before we begin... Assume a context-free struct Node *Parser::parse_A() { grammar has the struct Node *next_tok = lexer_peek(m_lexer); following productions on if (!next_tok) { the nonterminal A: error("Unexpected end of input"); } A → b C A → d E struct Node *a = node_build0(NODE_A); int tag = node_get_tag(next_tok); (A, C, E are if (tag == TOK_b) { nonterminals; b, d are node_add_kid(a, expect(TOK_b)); node_add_kid(a, parse_C()); terminals) } else if (tag == TOK_d) { What is the problem node_add_kid(a, expect(TOK_d)); node_add_kid(a, parse_E()); with the parse function } shown on the right? return a; } Limitations of recursive descent Recall: a better infix expression grammar Grammar (start symbol is A): A → i = A T → T*F A → E T → T/F E → E + T T → F E → E-T F → i E → T F → n Precedence levels: Nonterminal Precedence Meaning Operators Associativity A lowest Assignment = right E Expression + - left T Term * / left F highest Factor No Parsing infix expressions Can we write a recursive descent parser for infix expressions using this grammar? Parsing infix expressions Can we write a recursive descent parser for infix expressions using this grammar? No Left recursion Left-associative operators want to have left-recursive productions, but recursive descent parsers can’t handle left recursion
    [Show full text]
  • Basics of Compiler Design
    Basics of Compiler Design Anniversary edition Torben Ægidius Mogensen DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF COPENHAGEN Published through lulu.com. c Torben Ægidius Mogensen 2000 – 2010 [email protected] Department of Computer Science University of Copenhagen Universitetsparken 1 DK-2100 Copenhagen DENMARK Book homepage: http://www.diku.dk/∼torbenm/Basics First published 2000 This edition: August 20, 2010 ISBN 978-87-993154-0-6 Contents 1 Introduction 1 1.1 What is a compiler? . 1 1.2 The phases of a compiler . 2 1.3 Interpreters . 3 1.4 Why learn about compilers? . 4 1.5 The structure of this book . 5 1.6 To the lecturer . 6 1.7 Acknowledgements . 7 1.8 Permission to use . 7 2 Lexical Analysis 9 2.1 Introduction . 9 2.2 Regular expressions . 10 2.2.1 Shorthands . 13 2.2.2 Examples . 14 2.3 Nondeterministic finite automata . 15 2.4 Converting a regular expression to an NFA . 18 2.4.1 Optimisations . 20 2.5 Deterministic finite automata . 22 2.6 Converting an NFA to a DFA . 23 2.6.1 Solving set equations . 23 2.6.2 The subset construction . 26 2.7 Size versus speed . 29 2.8 Minimisation of DFAs . 30 2.8.1 Example . 32 2.8.2 Dead states . 34 2.9 Lexers and lexer generators . 35 2.9.1 Lexer generators . 41 2.10 Properties of regular languages . 42 2.10.1 Relative expressive power . 42 2.10.2 Limits to expressive power . 44 i ii CONTENTS 2.10.3 Closure properties . 45 2.11 Further reading .
    [Show full text]