DOCSLIB.ORG

Recursive-Descent Parsing Continued

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Recursive-Descent Parsing Continued Recursive-Descent Parsing continued CS F331 Programming Languages CSCE A331 Programming Language Concepts Lecture Slides Friday, February 14, 2020 Glenn G. Chappell Department of Computer Science University of Alaska Fairbanks [email protected] © 2017–2020 Glenn G. Chappell Review 2020-02-14 CS F331 / CSCE A331 Spring 2020 2 Review Overview of Lexing & Parsing Parsing Lexer Parser Character Lexeme AST or Stream Stream Error return (*dp + 2.6); //x return (*dp + 2.6); //x returnStmt key op id op num binOp: + lit punct punct unOp: * numLit: 2.6 Two phases: id: dp § Lexical analysis (lexing) § Syntax analysis (parsing) The output of a parser is typically an abstract syntax tree (AST). Specifications of these will vary. 2020-02-14 CS F331 / CSCE A331 Spring 2020 3 Review The Basics of Syntax Analysis — Categories of Parsers Parsing methods can be divided into two broad categories. Top-Down Parsers § Go through derivation top to bottom, expanding nonterminals. § Important subcategory: LL parsers (read input Left-to-right, produce Leftmost derivation). § Often hand-coded—but not always. § Method we look at: Predictive Recursive Descent. Bottom-Up Parsers § Go through the derivation bottom to top, reducing substrings to nonterminals. § Important subcategory: LR parsers (read input Left-to-right, produce Rightmost derivation). § Almost always automatically generated. § Method we look at: Shift-Reduce. 2020-02-14 CS F331 / CSCE A331 Spring 2020 4 Review The Basics of Syntax Analysis — Categories of Grammars LL(k) grammars: those usable by LL parsers that k is a number. make decisions based on the next k lexemes. Similarly: LR(k) grammars: those usable by LR parsers that make decisions based on the next k lexemes. All Grammars CFGs LR(1) Grammars Regular Grammars LL(1) Grammars 2020-02-14 CS F331 / CSCE A331 Spring 2020 5 Review Recursive-Descent Parsing — Intro thru Example #2 [1/2] Recursive Descent is a top-down parsing method. When we avoid backtracking: predictive. Predictive Recursive- Descent is an LL parsing method. There is one parsing function for each nonterminal. This parses all strings that the nonterminal can be expanded into. Parsing-function code is a translation of the right-hand side of the production for the nonterminal. § A terminal in the right-hand side becomes a check that the input string contains the proper lexeme. § A nonterminal in the right-hand side becomes a call to its parsing function. § If that returns false, our function must return false—no backtracking. § Brackets [ … ] in the right-hand side become a conditional. § Braces { … } in the right-hand side become See rdparser2.lua a loop. & userdparser2.lua. 2020-02-14 CS F331 / CSCE A331 Spring 2020 6 Review Recursive-Descent Parsing — Intro thru Example #2 [2/2] A naively written parser may call some inputs correct when it cannot parse the entire input. Example: ((x))) Syntactically Extra correct stuff Two Solutions § Introduce a new end of input lexeme (standard symbol: $). Revise the grammar to include it. § After parsing, check whether the end of the input was reached. Our parsers use the second solution. See rdparser2.lua & userdparser2.lua. 2020-02-14 CS F331 / CSCE A331 Spring 2020 7 Review Recursive-Descent Parsing — Example #3: Expressions [1/3] We wish to parse arithmetic expressions in their usual form, with variables, numeric literals, binary +, -, *, and / operators, and parentheses. When given a syntactically correct expression, our parser should return an abstract syntax tree (AST). All operators will be binary and left-associative: “a + b + c” means “(a + b) + c”. Precedence will be as usual: “a + b * c” means “a + (b * c)”. These may be overridden using parentheses: “(a + b) * c”. Due to the limitations of our in-class lexer, the expression “k-4” will need to be written as “k - 4”. 2020-02-14 CS F331 / CSCE A331 Spring 2020 8 Review Recursive-Descent Parsing — Example #3: Expressions [2/3] We started with Grammar 3, which has start symbol expr. This grammar encodes our associativity and precedence rules, and it allows us to use parentheses to override them. Grammar 3 expr → term | expr ( “+” | “-” ) term term → factor | term ( “*” | “/” ) factor factor → ID | NUMLIT | “(” expr “)” But Grammar 3 is not an LL(1) grammar, so we cannot use it. We see the problem if we try to write a Recursive-Descent parser. 2020-02-14 CS F331 / CSCE A331 Spring 2020 9 Review Recursive-Descent Parsing — Example #3: Expressions [3/3] Grammar 3 function parse_expr() expr → term if parse_term() then | expr ( “+” | “-” ) term … -- Construct AST term → factor return true | term ( “*” | “/” ) factor elseif parse_expr() then factor → ID … | NUMLIT | “(” expr “)” The above code has problems. § First, we are not supposed to be doing backtracking; if the call to parse_term returns false, then we must return false. § But even if we solve that, there is another problem. If parse_expr is called with input that does not begin with a valid term, then we get infinite recursion. 2020-02-14 CS F331 / CSCE A331 Spring 2020 10 Review Recursive-Descent Parsing — LL(1) Grammars An LL(1) grammar is a CFG that can be handled by an LL parsing method—such as Predictive Recursive Descent—without lookahead. The name comes from the fact that these parsers handle their input in a Left-to-right order, and they go through the steps required to generate a Leftmost derivation. When there is a choice to be made, an LL parser without lookahead must be able to make that choice based on the current lexeme. If this cannot be done, then the grammar is not LL(1). 2020-02-14 CS F331 / CSCE A331 Spring 2020 11 Review Recursive-Descent Parsing — Transforming Grammars Sometimes, we can transform a non-LL(1) grammar into an LL(1) grammar that generates the same language. For example, here is Grammar C, which is not LL(1), along with an LL(1) grammar that generates the same language. Grammar C Grammar Ca xx → yy | zz xx → yy | zz | “” yy → “” | “a” yy → “a” zz → “” | “b” zz → “b” 2020-02-14 CS F331 / CSCE A331 Spring 2020 12 Review Recursive-Descent Parsing — Back to Ex. #3 (Left-Associativity) Grammar 3b, below, is what we function parse_expr() want. It works with a Predictive if not parse_term() then Recursive-Descent parser, and return false we can use it to parse left- end associative binary operators. while matchString("+") or matchString("-") do Grammar 3b if not parse_term() then return false expr → term { ( “+” | “-” ) term } end term → factor { ( “*” | “/” ) factor } end factor → ID return true | NUMLIT end | “(” expr “)” Now, what about generating an AST? 2020-02-14 CS F331 / CSCE A331 Spring 2020 13 Recursive-Descent Parsing continued 2020-02-14 CS F331 / CSCE A331 Spring 2020 14 Recursive-Descent Parsing Back to Example #3: Expressions — ASTs [1/4] We want write a parser that returns an abstract syntax tree (AST). First we review some rooted-tree terminology. A node with one or more children—that is, a node A that is not a leaf—is an internal node. B C The tree at right has two internal nodes, labeled A and C. D E The subtrees of an internal node are those rooted A at each of its children. In the tree at right, the two subtrees of the A B C node are circled. D E 2020-02-14 CS F331 / CSCE A331 Spring 2020 15 Recursive-Descent Parsing Back to Example #3: Expressions — ASTs [2/4] Recall that a parse tree, or concrete syntax tree, includes one leaf node for each lexeme in the input, and also one node for each nonterminal in the derivation. Below is the parse tree for a + 2, based on Grammar 3b. expr This is not new! term + term It is the same kind of tree we drew before. factor factor a 2 2020-02-14 CS F331 / CSCE A331 Spring 2020 16 Recursive-Descent Parsing Back to Example #3: Expressions — ASTs [3/4] In an abstract syntax tree (AST), each internal node represents an operation. Its subtrees are the ASTs for the entities the operation is applied to. On the left is our parse tree for a + 2. On the right is an AST for the same expression. Abstract Parse Tree expr + Syntax Tree This is new. term + term a 2 factor factor a 2 Above, operation is broadly defined. For example, an AST node might represent the operation “execute all these statements, in order”. Its subtrees would be ASTs for the statements. 2020-02-14 CS F331 / CSCE A331 Spring 2020 17 Recursive-Descent Parsing Back to Example #3: Expressions — ASTs [4/4] The subtrees of an internal node in an AST are ASTs themselves. So we can build an AST from smaller ASTs in the same way we build an expression from smaller expressions. AST for AST for + a + 2 * (a + 2) * b a 2 + b a 2 Notes § ASTs omit portions of syntax that only serve to guide the parser— like semicolons, parentheses, commas, extra nonterminals. § There is no universal specification for an AST. Instead the ASTs used in a software project are specified so as to best meet the needs of that project. 2020-02-14 CS F331 / CSCE A331 Spring 2020 18 Recursive-Descent Parsing Back to Example #3: Expressions — Representing ASTs [1/7] + * a 2 + b a 2 We need to represent ASTs like these in Lua. § Represent a single node by the string form of its lexeme. § If there is more than one node in an AST, then represent it as an array whose first item represents the root, while each remaining item represents one of the subtrees of the root, in order. The first AST, in Lua: { "+", "a", "2" } TRY IT: What is the Lua representation of the second AST? 2020-02-14 CS F331 / CSCE A331 Spring 2020 19 Recursive-Descent Parsing Back to Example #3: Expressions — Representing ASTs [2/7] + * a 2 + b a 2 We need to represent ASTs like these in Lua.
Load more

Recommended publications
  • Derivatives of Parsing Expression Grammars
    Derivatives of Parsing Expression Grammars Aaron Moss Cheriton School of Computer Science University of Waterloo Waterloo, Ontario, Canada [email protected] This paper introduces a new derivative parsing algorithm for recognition of parsing expression gram- mars. Derivative parsing is shown to have a polynomial worst-case time bound, an improvement on the exponential bound of the recursive descent algorithm. This work also introduces asymptotic analysis based on inputs with a constant bound on both grammar nesting depth and number of back- tracking choices; derivative and recursive descent parsing are shown to run in linear time and constant space on this useful class of inputs, with both the theoretical bounds and the reasonability of the in- put class validated empirically. This common-case constant memory usage of derivative parsing is an improvement on the linear space required by the packrat algorithm. 1 Introduction Parsing expression grammars (PEGs) are a parsing formalism introduced by Ford [6]. Any LR(k) lan- guage can be represented as a PEG [7], but there are some non-context-free languages that may also be represented as PEGs (e.g. anbncn [7]). Unlike context-free grammars (CFGs), PEGs are unambiguous, admitting no more than one parse tree for any grammar and input. PEGs are a formalization of recursive descent parsers allowing limited backtracking and infinite lookahead; a string in the language of a PEG can be recognized in exponential time and linear space using a recursive descent algorithm, or linear time and space using the memoized packrat algorithm [6]. PEGs are formally defined and these algo- rithms outlined in Section 3.
    [Show full text]
  • A Typed, Algebraic Approach to Parsing
    A Typed, Algebraic Approach to Parsing Neelakantan R. Krishnaswami Jeremy Yallop University of Cambridge University of Cambridge United Kingdom United Kingdom [email protected] [email protected] Abstract the subject have remained remarkably stable: context-free In this paper, we recall the definition of the context-free ex- languages are specified in BackusśNaur form, and parsers pressions (or µ-regular expressions), an algebraic presenta- for these specifications are implemented using algorithms tion of the context-free languages. Then, we define a core derived from automata theory. This integration of theory and type system for the context-free expressions which gives a practice has yielded many benefits: we have linear-time algo- compositional criterion for identifying those context-free ex- rithms for parsing unambiguous grammars efficiently [Knuth pressions which can be parsed unambiguously by predictive 1965; Lewis and Stearns 1968] and with excellent error re- algorithms in the style of recursive descent or LL(1). porting for bad input strings [Jeffery 2003]. Next, we show how these typed grammar expressions can However, in languages with good support for higher-order be used to derive a parser combinator library which both functions (such as ML, Haskell, and Scala) it is very popular guarantees linear-time parsing with no backtracking and to use parser combinators [Hutton 1992] instead. These li- single-token lookahead, and which respects the natural de- braries typically build backtracking recursive descent parsers notational semantics of context-free expressions. Finally, we by composing higher-order functions. This allows building show how to exploit the type information to write a staged up parsers with the ordinary abstraction features of the version of this library, which produces dramatic increases programming language (variables, data structures and func- in performance, even outperforming code generated by the tions), which is sufficiently beneficial that these libraries are standard parser generator tool ocamlyacc.
    [Show full text]
  • Adaptive LL(*) Parsing: the Power of Dynamic Analysis
    Adaptive LL(*) Parsing: The Power of Dynamic Analysis Terence Parr Sam Harwell Kathleen Fisher University of San Francisco University of Texas at Austin Tufts University [email protected] [email protected] kfi[email protected] Abstract PEGs are unambiguous by definition but have a quirk where Despite the advances made by modern parsing strategies such rule A ! a j ab (meaning “A matches either a or ab”) can never as PEG, LL(*), GLR, and GLL, parsing is not a solved prob- match ab since PEGs choose the first alternative that matches lem. Existing approaches suffer from a number of weaknesses, a prefix of the remaining input. Nested backtracking makes de- including difficulties supporting side-effecting embedded ac- bugging PEGs difficult. tions, slow and/or unpredictable performance, and counter- Second, side-effecting programmer-supplied actions (muta- intuitive matching strategies. This paper introduces the ALL(*) tors) like print statements should be avoided in any strategy that parsing strategy that combines the simplicity, efficiency, and continuously speculates (PEG) or supports multiple interpreta- predictability of conventional top-down LL(k) parsers with the tions of the input (GLL and GLR) because such actions may power of a GLR-like mechanism to make parsing decisions. never really take place [17]. (Though DParser [24] supports The critical innovation is to move grammar analysis to parse- “final” actions when the programmer is certain a reduction is time, which lets ALL(*) handle any non-left-recursive context- part of an unambiguous final parse.) Without side effects, ac- free grammar. ALL(*) is O(n4) in theory but consistently per- tions must buffer data for all interpretations in immutable data forms linearly on grammars used in practice, outperforming structures or provide undo actions.
    [Show full text]
  • Abstract Syntax Trees & Top-Down Parsing
    Abstract Syntax Trees & Top-Down Parsing Review of Parsing • Given a language L(G), a parser consumes a sequence of tokens s and produces a parse tree • Issues: – How do we recognize that s ∈ L(G) ? – A parse tree of s describes how s ∈ L(G) – Ambiguity: more than one parse tree (possible interpretation) for some string s – Error: no parse tree for some string s – How do we construct the parse tree? Compiler Design 1 (2011) 2 Abstract Syntax Trees • So far, a parser traces the derivation of a sequence of tokens • The rest of the compiler needs a structural representation of the program • Abstract syntax trees – Like parse trees but ignore some details – Abbreviated as AST Compiler Design 1 (2011) 3 Abstract Syntax Trees (Cont.) • Consider the grammar E → int | ( E ) | E + E • And the string 5 + (2 + 3) • After lexical analysis (a list of tokens) int5 ‘+’ ‘(‘ int2 ‘+’ int3 ‘)’ • During parsing we build a parse tree … Compiler Design 1 (2011) 4 Example of Parse Tree E • Traces the operation of the parser E + E • Captures the nesting structure • But too much info int5 ( E ) – Parentheses – Single-successor nodes + E E int 2 int3 Compiler Design 1 (2011) 5 Example of Abstract Syntax Tree PLUS PLUS 5 2 3 • Also captures the nesting structure • But abstracts from the concrete syntax a more compact and easier to use • An important data structure in a compiler Compiler Design 1 (2011) 6 Semantic Actions • This is what we’ll use to construct ASTs • Each grammar symbol may have attributes – An attribute is a property of a programming language construct
    [Show full text]
  • Efficient Recursive Parsing
    Purdue University Purdue e-Pubs Department of Computer Science Technical Reports Department of Computer Science 1976 Efficient Recursive Parsing Christoph M. Hoffmann Purdue University, [email protected] Report Number: 77-215 Hoffmann, Christoph M., "Efficient Recursive Parsing" (1976). Department of Computer Science Technical Reports. Paper 155. https://docs.lib.purdue.edu/cstech/155 This document has been made available through Purdue e-Pubs, a service of the Purdue University Libraries. Please contact [email protected] for additional information. EFFICIENT RECURSIVE PARSING Christoph M. Hoffmann Computer Science Department Purdue University West Lafayette, Indiana 47907 CSD-TR 215 December 1976 Efficient Recursive Parsing Christoph M. Hoffmann Computer Science Department Purdue University- Abstract Algorithms are developed which construct from a given LL(l) grammar a recursive descent parser with as much, recursion resolved by iteration as is possible without introducing auxiliary memory. Unlike other proposed methods in the literature designed to arrive at parsers of this kind, the algorithms do not require extensions of the notational formalism nor alter the grammar in any way. The algorithms constructing the parsers operate in 0(k«s) steps, where s is the size of the grammar, i.e. the sum of the lengths of all productions, and k is a grammar - dependent constant. A speedup of the algorithm is possible which improves the bound to 0(s) for all LL(l) grammars, and constructs smaller parsers with some auxiliary memory in form of parameters
    [Show full text]
  • Conflict Resolution in a Recursive Descent Compiler Generator
    LL(1) Conflict Resolution in a Recursive Descent Compiler Generator Albrecht Wöß, Markus Löberbauer, Hanspeter Mössenböck Johannes Kepler University Linz, Institute of Practical Computer Science, Altenbergerstr. 69, 4040 Linz, Austria {woess,loeberbauer,moessenboeck}@ssw.uni-linz.ac.at Abstract. Recursive descent parsing is restricted to languages whose grammars are LL(1), i.e., which can be parsed top-down with a single lookahead symbol. Unfortunately, many languages such as Java, C++, or C# are not LL(1). There- fore recursive descent parsing cannot be used or the parser has to make its deci- sions based on semantic information or a multi-symbol lookahead. In this paper we suggest a systematic technique for resolving LL(1) conflicts in recursive descent parsing and show how to integrate it into a compiler gen- erator (Coco/R). The idea is to evaluate user-defined boolean expressions, in order to allow the parser to make its parsing decisions where a one symbol loo- kahead does not suffice. Using our extended compiler generator we implemented a compiler front end for C# that can be used as a framework for implementing a variety of tools. 1 Introduction Recursive descent parsing [16] is a popular top-down parsing technique that is sim- ple, efficient, and convenient for integrating semantic processing. However, it re- quires the grammar of the parsed language to be LL(1), which means that the parser must always be able to select between alternatives with a single symbol lookahead. Unfortunately, many languages such as Java, C++ or C# are not LL(1) so that one either has to resort to bottom-up LALR(1) parsing [5, 1], which is more powerful but less convenient for semantic processing, or the parser has to resolve the LL(1) con- flicts using semantic information or a multi-symbol lookahead.
    [Show full text]
  • A Parsing Machine for Pegs
    A Parsing Machine for PEGs ∗ Sérgio Medeiros Roberto Ierusalimschy [email protected] [email protected] Department of Computer Science PUC-Rio, Rio de Janeiro, Brazil ABSTRACT parsing approaches. The PEG formalism gives a convenient Parsing Expression Grammar (PEG) is a recognition-based syntax for describing top-down parsers for unambiguous lan- foundation for describing syntax that renewed interest in guages. The parsers it describes can parse strings in linear top-down parsing approaches. Generally, the implementa- time, despite backtracking, by using a memoizing algorithm tion of PEGs is based on a recursive-descent parser, or uses called Packrat [1]. Although Packrat has guaranteed worst- a memoization algorithm. case linear time complexity, it also has linear space com- We present a new approach for implementing PEGs, based plexity, with a rather large constant. This makes Packrat on a virtual parsing machine, which is more suitable for impractical for parsing large amounts of data. pattern matching. Each PEG has a corresponding program LPEG [7, 11] is a pattern-matching tool for Lua, a dy- that is executed by the parsing machine, and new programs namically typed scripting language [6, 8]. LPEG uses PEGs are dynamically created and composed. The virtual machine to describe patterns instead of the more popular Perl-like is embedded in a scripting language and used by a pattern- "regular expressions" (regexes). Regexes are a more ad-hoc matching tool. way of describing patterns; writing a complex regex is more We give an operational semantics of PEGs used for pat- a trial and error than a systematic process, and their per- tern matching, then describe our parsing machine and its formance is very unpredictable.
    [Show full text]
  • AST Indexing: a Near-Constant Time Solution to the Get-Descendants-By-Type Problem Samuel Livingston Kelly Dickinson College
    Dickinson College Dickinson Scholar Student Honors Theses By Year Student Honors Theses 5-18-2014 AST Indexing: A Near-Constant Time Solution to the Get-Descendants-by-Type Problem Samuel Livingston Kelly Dickinson College Follow this and additional works at: http://scholar.dickinson.edu/student_honors Part of the Software Engineering Commons Recommended Citation Kelly, Samuel Livingston, "AST Indexing: A Near-Constant Time Solution to the Get-Descendants-by-Type Problem" (2014). Dickinson College Honors Theses. Paper 147. This Honors Thesis is brought to you for free and open access by Dickinson Scholar. It has been accepted for inclusion by an authorized administrator. For more information, please contact [email protected]. AST Indexing: A Near-Constant Time Solution to the Get-Descendants-by-Type Problem by Sam Kelly Submitted in partial fulfillment of Honors Requirements for the Computer Science Major Dickinson College, 2014 Associate Professor Tim Wahls, Supervisor Associate Professor John MacCormick, Reader Part-Time Instructor Rebecca Wells, Reader April 22, 2014 The Department of Mathematics and Computer Science at Dickinson College hereby accepts this senior honors thesis by Sam Kelly, and awards departmental honors in Computer Science. Tim Wahls (Advisor) Date John MacCormick (Committee Member) Date Rebecca Wells (Committee Member) Date Richard Forrester (Department Chair) Date Department of Mathematics and Computer Science Dickinson College April 2014 ii Abstract AST Indexing: A Near-Constant Time Solution to the Get- Descendants-by-Type Problem by Sam Kelly In this paper we present two novel abstract syntax tree (AST) indexing algorithms that solve the get-descendants-by-type problem in near constant time.
    [Show full text]
  • Syntactic Analysis, Or Parsing, Is the Second Phase of Compilation: the Token File Is Converted to an Abstract Syntax Tree
    Syntactic Analysis Syntactic analysis, or parsing, is the second phase of compilation: The token file is converted to an abstract syntax tree. Analysis Synthesis Compiler Passes of input program of output program (front -end) (back -end) character stream Intermediate Lexical Analysis Code Generation token intermediate stream form Syntactic Analysis Optimization abstract intermediate syntax tree form Semantic Analysis Code Generation annotated target AST language 1 Syntactic Analysis / Parsing • Goal: Convert token stream to abstract syntax tree • Abstract syntax tree (AST): – Captures the structural features of the program – Primary data structure for remainder of analysis • Three Part Plan – Study how context-free grammars specify syntax – Study algorithms for parsing / building ASTs – Study the miniJava Implementation Context-free Grammars • Compromise between – REs, which can’t nest or specify recursive structure – General grammars, too powerful, undecidable • Context-free grammars are a sweet spot – Powerful enough to describe nesting, recursion – Easy to parse; but also allow restrictions for speed • Not perfect – Cannot capture semantics, as in, “variable must be declared,” requiring later semantic pass – Can be ambiguous • EBNF, Extended Backus Naur Form, is popular notation 2 CFG Terminology • Terminals -- alphabet of language defined by CFG • Nonterminals -- symbols defined in terms of terminals and nonterminals • Productions -- rules for how a nonterminal (lhs) is defined in terms of a (possibly empty) sequence of terminals
    [Show full text]
  • Lecture 3: Recursive Descent Limitations, Precedence Climbing
    Lecture 3: Recursive descent limitations, precedence climbing David Hovemeyer September 9, 2020 601.428/628 Compilers and Interpreters Today I Limitations of recursive descent I Precedence climbing I Abstract syntax trees I Supporting parenthesized expressions Before we begin... Assume a context-free struct Node *Parser::parse_A() { grammar has the struct Node *next_tok = lexer_peek(m_lexer); following productions on if (!next_tok) { the nonterminal A: error("Unexpected end of input"); } A → b C A → d E struct Node *a = node_build0(NODE_A); int tag = node_get_tag(next_tok); (A, C, E are if (tag == TOK_b) { nonterminals; b, d are node_add_kid(a, expect(TOK_b)); node_add_kid(a, parse_C()); terminals) } else if (tag == TOK_d) { What is the problem node_add_kid(a, expect(TOK_d)); node_add_kid(a, parse_E()); with the parse function } shown on the right? return a; } Limitations of recursive descent Recall: a better infix expression grammar Grammar (start symbol is A): A → i = A T → T*F A → E T → T/F E → E + T T → F E → E-T F → i E → T F → n Precedence levels: Nonterminal Precedence Meaning Operators Associativity A lowest Assignment = right E Expression + - left T Term * / left F highest Factor No Parsing infix expressions Can we write a recursive descent parser for infix expressions using this grammar? Parsing infix expressions Can we write a recursive descent parser for infix expressions using this grammar? No Left recursion Left-associative operators want to have left-recursive productions, but recursive descent parsers can’t handle left recursion
    [Show full text]
  • Understanding Source Code Evolution Using Abstract Syntax Tree Matching
    Understanding Source Code Evolution Using Abstract Syntax Tree Matching Iulian Neamtiu Jeffrey S. Foster Michael Hicks [email protected] [email protected] [email protected] Department of Computer Science University of Maryland at College Park ABSTRACT so understanding how the type structure of real programs Mining software repositories at the source code level can pro- changes over time can be invaluable for weighing the merits vide a greater understanding of how software evolves. We of DSU implementation choices. Second, we are interested in present a tool for quickly comparing the source code of dif- a kind of “release digest” for explaining changes in a software ferent versions of a C program. The approach is based on release: what functions or variables have changed, where the partial abstract syntax tree matching, and can track sim- hot spots are, whether or not the changes affect certain com- ple changes to global variables, types and functions. These ponents, etc. Typical release notes can be too high level for changes can characterize aspects of software evolution use- developers, and output from diff can be too low level. ful for answering higher level questions. In particular, we To answer these and other software evolution questions, consider how they could be used to inform the design of a we have developed a tool that can quickly tabulate and sum- dynamic software updating system. We report results based marize simple changes to successive versions of C programs on measurements of various versions of popular open source by partially matching their abstract syntax trees. The tool programs, including BIND, OpenSSH, Apache, Vsftpd and identifies the changes, additions, and deletions of global vari- the Linux kernel.
    [Show full text]
  • Parser Generation Bottom-Up Parsing LR Parsing Constructing LR Parser
    CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Parser Generation Bottom-Up Parsing • Main Problem: given a grammar G, how to build a top-down parser or a • Construct parse tree “bottom-up” --- from leaves to the root bottom-up parser for it ? • Bottom-up parsing always constructs right-most derivation • parser : a program that, given a sentence, reconstructs a derivation for that • Important parsing algorithms: shift-reduce, LR parsing sentence ---- if done sucessfully, it “recognize” the sentence • LR parser components: input, stack (strings of grammar symbols and states), • all parsers read their input left-to-right, but construct parse tree differently. driver routine, parsing tables. • bottom-up parsers --- construct the tree from leaves to root input: a1 a2 a3 a4 ...... an $ shift-reduce, LR, SLR, LALR, operator precedence stack s m LR Parsing output • top-down parsers --- construct the tree from root to leaves Xm .. recursive descent, predictive parsing, LL(1) s1 X1 s0 parsing Copyright 1994 - 2015 Zhong Shao, Yale University Parser Generation: Page 1 of 27 Copyright 1994 - 2015 Zhong Shao, Yale University Parser Generation: Page 2 of 27 CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS LR Parsing Constructing LR Parser How to construct the parsing table and ? • A sequence of new state symbols s0,s1,s2,..., sm ----- each state ACTION GOTO sumarize the information contained in the stack below it. • basic idea: first construct DFA to recognize handles, then use DFA to construct the parsing tables ! different parsing table yield different LR parsers SLR(1), • Parsing configurations: (stack, remaining input) written as LR(1), or LALR(1) (s0X1s1X2s2...Xmsm , aiai+1ai+2...an$) • augmented grammar for context-free grammar G = G(T,N,P,S) is defined as G’ next “move” is determined by sm and ai = G’(T, N { S’}, P { S’ -> S}, S’) ------ adding non-terminal S’ and the • Parsing tables: ACTION[s,a] and GOTO[s,X] production S’ -> S, and S’ is the new start symbol.
    [Show full text]