Table Driven Prediction for Recursive Descent LL(K) Parsers
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Parsing Based on N-Path Tree-Controlled Grammars
Theoretical and Applied Informatics ISSN 1896–5334 Vol.23 (2011), no. 3-4 pp. 213–228 DOI: 10.2478/v10179-011-0015-7 Parsing Based on n-Path Tree-Controlled Grammars MARTIN Cˇ ERMÁK,JIR͡ KOUTNÝ,ALEXANDER MEDUNA Formal Model Research Group Department of Information Systems Faculty of Information Technology Brno University of Technology Božetechovaˇ 2, 612 66 Brno, Czech Republic icermak, ikoutny, meduna@fit.vutbr.cz Received 15 November 2011, Revised 1 December 2011, Accepted 4 December 2011 Abstract: This paper discusses recently introduced kind of linguistically motivated restriction placed on tree-controlled grammars—context-free grammars with some root-to-leaf paths in their derivation trees restricted by a control language. We deal with restrictions placed on n ¸ 1 paths controlled by a deter- ministic context–free language, and we recall several basic properties of such a rewriting system. Then, we study the possibilities of corresponding parsing methods working in polynomial time and demonstrate that some non-context-free languages can be generated by this regulated rewriting model. Furthermore, we illustrate the syntax analysis of LL grammars with controlled paths. Finally, we briefly discuss how to base parsing methods on bottom-up syntax–analysis. Keywords: regulated rewriting, derivation tree, tree-controlled grammars, path-controlled grammars, parsing, n-path tree-controlled grammars 1. Introduction The investigation of context-free grammars with restricted derivation trees repre- sents an important trend in today’s formal language theory (see [3], [6], [10], [11], [12], [13] and [15]). In essence, these grammars generate their languages just like ordinary context-free grammars do, but their derivation trees have to satisfy some simple pre- scribed conditions. -
High-Performance Context-Free Parser for Polymorphic Malware Detection
High-Performance Context-Free Parser for Polymorphic Malware Detection Young H. Cho and William H. Mangione-Smith The University of California, Los Angeles, CA 91311 {young, billms}@ee.ucla.edu http://cares.icsl.ucla.edu Abstract. Due to increasing economic damage from computer network intrusions, many routers have built-in firewalls that can classify packets based on header information. Such classification engine can be effective in stopping attacks that target protocol specific vulnerabilities. However, they are not able to detect worms that are encapsulated in the packet payload. One method used to detect such application-level attack is deep packet inspection. Unlike the most firewalls, a system with a deep packet inspection engine can search for one or more specific patterns in all parts of the packets. Although deep packet inspection increases the packet filtering effectiveness and accuracy, most of the current implementations do not extend beyond recognizing a set of predefined regular expressions. In this paper, we present an advanced inspection engine architecture that is capable of recognizing language structures described by context-free grammars. We begin by modifying a known regular expression engine to function as the lexical analyzer. Then we build an efficient multi-threaded parsing co-processor that processes the tokens from the lexical analyzer according to the grammar. 1 Introduction One effective method for detecting network attacks is called deep packet inspec- tion. Unlike the traditional firewall methods, deep packet inspection is designed to search and detect the entire packet for known attack signatures. However, due to high processing requirement, implementing such a detector using a general purpose processor is costly for 1+ gigabit network. -
LLLR Parsing: a Combination of LL and LR Parsing
LLLR Parsing: a Combination of LL and LR Parsing Boštjan Slivnik University of Ljubljana, Faculty of Computer and Information Science, Ljubljana, Slovenia [email protected] Abstract A new parsing method called LLLR parsing is defined and a method for producing LLLR parsers is described. An LLLR parser uses an LL parser as its backbone and parses as much of its input string using LL parsing as possible. To resolve LL conflicts it triggers small embedded LR parsers. An embedded LR parser starts parsing the remaining input and once the LL conflict is resolved, the LR parser produces the left parse of the substring it has just parsed and passes the control back to the backbone LL parser. The LLLR(k) parser can be constructed for any LR(k) grammar. It produces the left parse of the input string without any backtracking and, if used for a syntax-directed translation, it evaluates semantic actions using the top-down strategy just like the canonical LL(k) parser. An LLLR(k) parser is appropriate for grammars where the LL(k) conflicting nonterminals either appear relatively close to the bottom of the derivation trees or produce short substrings. In such cases an LLLR parser can perform a significantly better error recovery than an LR parser since the most part of the input string is parsed with the backbone LL parser. LLLR parsing is similar to LL(∗) parsing except that it (a) uses LR(k) parsers instead of finite automata to resolve the LL(k) conflicts and (b) does not perform any backtracking. -
Compiler Design
CCOOMMPPIILLEERR DDEESSIIGGNN -- PPAARRSSEERR http://www.tutorialspoint.com/compiler_design/compiler_design_parser.htm Copyright © tutorialspoint.com In the previous chapter, we understood the basic concepts involved in parsing. In this chapter, we will learn the various types of parser construction methods available. Parsing can be defined as top-down or bottom-up based on how the parse-tree is constructed. Top-Down Parsing We have learnt in the last chapter that the top-down parsing technique parses the input, and starts constructing a parse tree from the root node gradually moving down to the leaf nodes. The types of top-down parsing are depicted below: Recursive Descent Parsing Recursive descent is a top-down parsing technique that constructs the parse tree from the top and the input is read from left to right. It uses procedures for every terminal and non-terminal entity. This parsing technique recursively parses the input to make a parse tree, which may or may not require back-tracking. But the grammar associated with it ifnotleftfactored cannot avoid back- tracking. A form of recursive-descent parsing that does not require any back-tracking is known as predictive parsing. This parsing technique is regarded recursive as it uses context-free grammar which is recursive in nature. Back-tracking Top- down parsers start from the root node startsymbol and match the input string against the production rules to replace them ifmatched. To understand this, take the following example of CFG: S → rXd | rZd X → oa | ea Z → ai For an input string: read, a top-down parser, will behave like this: It will start with S from the production rules and will match its yield to the left-most letter of the input, i.e. -
Type Checking by Context-Sensitive Languages
TYPE CHECKING BY CONTEXT-SENSITIVE LANGUAGES Lukáš Rychnovský Doctoral Degree Programme (1), FIT BUT E-mail: rychnov@fit.vutbr.cz Supervised by: Dušan Kolárˇ E-mail: kolar@fit.vutbr.cz ABSTRACT This article presents some ideas from parsing Context-Sensitive languages. Introduces Scattered- Context grammars and languages and describes usage of such grammars to parse CS languages. The main goal of this article is to present results from type checking using CS parsing. 1 INTRODUCTION This work results from [Kol–04] where relationship between regulated pushdown automata (RPDA) and Turing machines is discussed. We are particulary interested in algorithms which may be used to obtain RPDAs from equivalent Turing machines. Such interest arises purely from practical reasons because RPDAs provide easier implementation techniques for prob- lems where Turing machines are naturally used. As a representant of such problems we study context-sensitive extensions of programming language grammars which are usually context- free. By introducing context-sensitive syntax analysis into the source code parsing process a whole class of new problems may be solved at this stage of a compiler. Namely issues with correct variable definitions, type checking etc. 2 SCATTERED CONTEXT GRAMMARS Definition 2.1 A scattered context grammar (SCG) G is a quadruple (V,T,P,S), where V is a finite set of symbols, T ⊂ V,S ∈ V\T, and P is a set of production rules of the form ∗ (A1,...,An) → (w1,...,wn),n ≥ 1,∀Ai : Ai ∈ V\T,∀wi : wi ∈ V Definition 2.2 Let G = (V,T,P,S) be a SCG. Let (A1,...,An) → (w1,...,wn) ∈ P. -
The Use of Predicates in LL(K) and LR(K) Parser Generators
Purdue University Purdue e-Pubs ECE Technical Reports Electrical and Computer Engineering 7-1-1993 The seU of Predicates In LL(k) And LR(k) Parser Generators (Technical Summary) T. J. Parr Purdue University School of Electrical Engineering R. W. Quong Purdue University School of Electrical Engineering H. G. Dietz Purdue University School of Electrical Engineering Follow this and additional works at: http://docs.lib.purdue.edu/ecetr Parr, T. J.; Quong, R. W.; and Dietz, H. G., "The sU e of Predicates In LL(k) And LR(k) Parser Generators (Technical Summary) " (1993). ECE Technical Reports. Paper 234. http://docs.lib.purdue.edu/ecetr/234 This document has been made available through Purdue e-Pubs, a service of the Purdue University Libraries. Please contact [email protected] for additional information. TR-EE 93-25 JULY 1993 The Use of Predicates In LL(k) And LR(k) Parser ene era tors? (Technical Summary) T. J. Purr, R. W. Quong, and H. G. Dietz School of Electrical Engineering Purdue University West Lafayette, IN 47907 (317) 494-1739 [email protected] Abstract Although existing LR(1) or U(1) parser generators suffice for many language recognition problems, writing a straightforward grammar to translate a complicated language, such as C++ or even C, remains a non-trivial task. We have often found that adding translation actions to the grammar is harder than writing the grammar itself. Part of the problem is that many languages are context-sensitive. Simple, natural descriptions of these languages escape current language tool technology because they were not designed to handle semantic information. -
Compiler Construction
Compiler construction Martin Steffen February 20, 2017 Contents 1 Abstract 1 1.1 Parsing . .1 1.1.1 First and follow sets . .1 1.1.2 Top-down parsing . 15 1.1.3 Bottom-up parsing . 43 2 Reference 78 1 Abstract Abstract This is the handout version of the slides. It contains basically the same content, only in a way which allows more compact printing. Sometimes, the overlays, which make sense in a presentation, are not fully rendered here. Besides the material of the slides, the handout versions may also contain additional remarks and background information which may or may not be helpful in getting the bigger picture. 1.1 Parsing 31. 1. 2017 1.1.1 First and follow sets Overview • First and Follow set: general concepts for grammars – textbook looks at one parsing technique (top-down) [Louden, 1997, Chap. 4] before studying First/Follow sets – we: take First/Follow sets before any parsing technique • two transformation techniques for grammars, both preserving the accepted language 1. removal for left-recursion 2. left factoring First and Follow sets • general concept for grammars • certain types of analyses (e.g. parsing): – info needed about possible “forms” of derivable words, 1. First-set of A which terminal symbols can appear at the start of strings derived from a given nonterminal A 1 2. Follow-set of A Which terminals can follow A in some sentential form. 3. Remarks • sentential form: word derived from grammar’s starting symbol • later: different algos for First and Follow sets, for all non-terminals of a given grammar • mostly straightforward • one complication: nullable symbols (non-terminals) • Note: those sets depend on grammar, not the language First sets Definition 1 (First set). -
Part 1: Introduction
Part 1: Introduction By: Morteza Zakeri PhD Student Iran University of Science and Technology Winter 2020 Agenda • What is ANTLR? • History • Motivation • What is New in ANTLR v4? • ANTLR Components: How it Works? • Getting Started with ANTLR v4 February 2020 Introduction to ANTLR – Morteza Zakeri 2 What is ANTLR? • ANTLR (pronounced Antler), or Another Tool For Language Recognition, is a parser generator that uses LL(*) for parsing. • ANTLR takes as input a grammar that specifies a language and generates as output source code for a recognizer for that language. • Supported generating code in Java, C#, JavaScript, Python2 and Python3. • ANTLR is recursive descent parser Generator! (See Appendix) February 2020 Introduction to ANTLR – Morteza Zakeri 3 Runtime Libraries and Code Generation Targets • There is no language specific code generators • There is only one tool, written in Java, which is able to generate Lexer and Parser code for all targets, through command line options. • The available targets are the following (2020): • Java, C#, C++, Swift, Python (2 and 3), Go, PHP, and JavaScript. • Read more: • https://github.com/antlr/antlr4/blob/master/doc/targets.md 11 February 2020 Introduction to ANTLR – Morteza Zakeri Runtime Libraries and Code Generation Targets • $ java -jar antlr4-4.8.jar -Dlanguage=CSharp MyGrammar.g4 • https://github.com/antlr/antlr4/tree/master/runtime/CSharp • https://github.com/tunnelvisionlabs/antlr4cs • $ java -jar antlr4-4.8.jar -Dlanguage=Cpp MyGrammar.g4 • https://github.com/antlr/antlr4/blob/master/doc/cpp-target.md • $ java -jar antlr4-4.8.jar -Dlanguage=Python3 MyGrammar.g4 • https://github.com/antlr/antlr4/blob/master/doc/python- target.md 11 February 2020 Introduction to ANTLR – Morteza Zakeri History • Initial release: • February 1992; 24 years ago. -
Compiler Construction
Compiler construction PDF generated using the open source mwlib toolkit. See http://code.pediapress.com/ for more information. PDF generated at: Sat, 10 Dec 2011 02:23:02 UTC Contents Articles Introduction 1 Compiler construction 1 Compiler 2 Interpreter 10 History of compiler writing 14 Lexical analysis 22 Lexical analysis 22 Regular expression 26 Regular expression examples 37 Finite-state machine 41 Preprocessor 51 Syntactic analysis 54 Parsing 54 Lookahead 58 Symbol table 61 Abstract syntax 63 Abstract syntax tree 64 Context-free grammar 65 Terminal and nonterminal symbols 77 Left recursion 79 Backus–Naur Form 83 Extended Backus–Naur Form 86 TBNF 91 Top-down parsing 91 Recursive descent parser 93 Tail recursive parser 98 Parsing expression grammar 100 LL parser 106 LR parser 114 Parsing table 123 Simple LR parser 125 Canonical LR parser 127 GLR parser 129 LALR parser 130 Recursive ascent parser 133 Parser combinator 140 Bottom-up parsing 143 Chomsky normal form 148 CYK algorithm 150 Simple precedence grammar 153 Simple precedence parser 154 Operator-precedence grammar 156 Operator-precedence parser 159 Shunting-yard algorithm 163 Chart parser 173 Earley parser 174 The lexer hack 178 Scannerless parsing 180 Semantic analysis 182 Attribute grammar 182 L-attributed grammar 184 LR-attributed grammar 185 S-attributed grammar 185 ECLR-attributed grammar 186 Intermediate language 186 Control flow graph 188 Basic block 190 Call graph 192 Data-flow analysis 195 Use-define chain 201 Live variable analysis 204 Reaching definition 206 Three address -
Left-Recursive Grammars
Parsing • A grammar describes the strings of tokens that are syntactically legal in a PL • A recogniser simply accepts or rejects strings. • A generator produces sentences in the language described by the grammar • A parser construct a derivation or parse tree for a sentence (if possible) • Two common types of parsers: – bottom-up or data driven – top-down or hypothesis driven • A recursive descent parser is a way to implement a top- down parser that is particularly simple. Top down vs. bottom up parsing • The parsing problem is to connect the root node S S with the tree leaves, the input • Top-down parsers: starts constructing the parse tree at the top (root) of the parse tree and move down towards the leaves. Easy to implement by hand, but work with restricted grammars. examples: - Predictive parsers (e.g., LL(k)) A = 1 + 3 * 4 / 5 • Bottom-up parsers: build the nodes on the bottom of the parse tree first. Suitable for automatic parser generation, handle a larger class of grammars. examples: – shift-reduce parser (or LR(k) parsers) • Both are general techniques that can be made to work for all languages (but not all grammars!). Top down vs. bottom up parsing • Both are general techniques that can be made to work for all languages (but not all grammars!). • Recall that a given language can be described by several grammars. • Both of these grammars describe the same language E -> E + Num E -> Num + E E -> Num E -> Num • The first one, with it’s left recursion, causes problems for top down parsers. -
ANTLR Reference PDF Manual
ANTLR Reference Manual Home | Download | News | About ANTLR | Support Latest version is 2.7.3. Download now! » » Home » Download ANTLR Reference Manual » News »Using ANTLR » Documentation ANTLR » FAQ » Articles Reference Manual » Grammars Credits » File Sharing » Code API Project Lead and Supreme Dictator Terence Parr » Tech Support University of San Franciso »About ANTLR Support from » What is ANTLR jGuru.com » Why use ANTLR Your View of the Java Universe » Showcase Help with initial coding » Testimonials John Lilly, Empathy Software » Getting Started C++ code generator by » Software License Peter Wells and Ric Klaren » ANTLR WebLogs C# code generation by »StringTemplate Micheal Jordan, Kunle Odutola and Anthony Oguntimehin. »TML Infrastructure support from Perforce: »PCCTS The world's best source code control system Substantial intellectual effort donated by Loring Craymer Monty Zukowski Jim Coker Scott Stanchfield John Mitchell Chapman Flack (UNICODE, streams) Source changes for Eclipse and NetBeans by Marco van Meegen and Brian Smith ANTLR Version 2.7.3 March 22, 2004 What's ANTLR http://www.antlr.org/doc/index.html (1 of 6)31.03.2004 17:11:46 ANTLR Reference Manual ANTLR, ANother Tool for Language Recognition, (formerly PCCTS) is a language tool that provides a framework for constructing recognizers, compilers, and translators from grammatical descriptions containing Java, C++, or C# actions [You can use PCCTS 1.xx to generate C-based parsers]. Computer language translation has become a common task. While compilers and tools for traditional computer languages (such as C or Java) are still being built, their number is dwarfed by the thousands of mini-languages for which recognizers and translators are being developed. -
Town-Down LL Parsing
Lecture Notes on Top-Down Predictive LL Parsing 15-411: Compiler Design Frank Pfenning∗ Lecture 8 1 Introduction In this lecture we discuss a parsing algorithm that traverses the input string from left to right giving a left-most derivation and makes a decision on ¯ ¯ which grammar production to use based on the first character/token of the input string. This parsing algorithm is called LL(1). If that were ambigu- ous, the grammar would have to be rewritten to fall into this class, which is not always possible. Most hand-written parsers are recursive descent parsers, which follow the LL(1) principles. Alternative presentations of the material in this lecture can be found in a paper by Shieber et al. [SSP95]. The textbook [App98, Chapter 3] covers parsing. For more detailed background on parsing, also see [WM95]. 2 LL(1) Parsing We have seen in the previous section, that the general idea of recursive de- scent parsing without restrictions forces us to non-deterministically choose between several productions which might be applied and potentially back- track if parsing gets stuck after a choice, or even loop (if the grammar is left-recursive). Backtracking is not only potentially very inefficient, but it makes it difficult to produce good error messages in case the string is not grammatically well-formed. Say we try three different ways to parse a given input and all fail. How could we say which of these is the source ∗With edits by Andre´ Platzer LECTURE NOTES L8.2 Top-Down Predictive LL Parsing of the error? How could we report an error location? This is compounded because nested choices multiply the number of possibilities.