DOCSLIB.ORG

Maximal-Munch” Tokenization in Linear Time Tom Reps [TOPLAS 1998]

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Maximal-Munch” Tokenization in Linear Time Tom Reps [TOPLAS 1998] Fall 2016-2017 Compiler Principles Lecture 1: Lexical Analysis Roman Manevich Ben-Gurion University of the Negev Agenda • Understand role of lexical analysis in a compiler • Regular languages reminder • Lexical analysis algorithms • Scanner generation 2 Javascript example • Can you some identify basic units in this code? var currOption = 0; // Choose content to display in lower pane. function choose ( id ) { var menu = ["about-me", "publications", "teaching", "software", "activities"]; for (i = 0; i < menu.length; i++) { currOption = menu[i]; var elt = document.getElementById(currOption); if (currOption == id && elt.style.display == "none") { elt.style.display = "block"; } else { elt.style.display = "none"; } } } 3 Javascript example • Can you some identify basic units in this code? keyword ? ? ? ? ? var currOption = 0; // Choose content to display in lower pane. ? function choose ( id ) { var menu = ["about-me", "publications", "teaching", "software", "activities"]; ? for (i = 0; i < menu.length; i++) { currOption = menu[i]; var elt = document.getElementById(currOption); if (currOption == id && elt.style.display == "none") { elt.style.display = "block"; } else { elt.style.display = "none"; } } } 4 Javascript example • Can you some identify basic units in this code? keyword identifier operator numeric literal punctuation comment var currOption = 0; // Choose content to display in lower pane. string literal function choose ( id ) { var menu = ["about-me", "publications", "teaching", "software", "activities"]; whitespace for (i = 0; i < menu.length; i++) { currOption = menu[i]; var elt = document.getElementById(currOption); if (currOption == id && elt.style.display == "none") { elt.style.display = "block"; } else { elt.style.display = "none"; } } } 5 Role of lexical analysis • First part of compiler front-end Lexical Syntax AST Symbol Inter. Code High-level Analysis Analysis Table Rep. Generation Executable Language etc. (IR) Parsing Code (scheme) • Convert stream of characters into stream of tokens – Split text into most basic meaningful strings • Simplify input for syntax analysis 6 From scanning to parsing program text 59 + (1257 * xPosition) Lexical Analyzer Lexical error valid token stream num + ( num * id ) Grammar: E id E num Parser E E + E E E * E syntax valid E ( E ) error + Abstract Syntax Tree num * num x 7 Scanner output wherewhere isis thethe whitewhite space?space? var currOption = 0; // Choose content to display in lower pane. Stream of Tokens function choose ( id ) { LINE: ID(value) var menu = ["about-me", "publications“, "teaching", "software", "activities"]; 1: VAR 1: ID(currOption) for (i = 0; i < menu.length; i++) { 1: EQ currOption = menu[i]; 1: INT_LITERAL(0) var elt = document.getElementById(currOption); 1: SEMI if (currOption == id && elt.style.display == "none") { elt.style.display = "block"; 3: FUNCTION } 3: ID(choose) else { 3: LP elt.style.display = "none"; 3: ID(id) } 3: EP } 3: LCB } ... 8 Tokens 9 What is a token? • Lexeme – substring of original text constituting an identifiable unit – Identifiers, values, reserved words, … • Record type storing: – Kind – Value (when applicable) – Start-position/end-position – Any information that is useful for the parser • Different for different languages 10 Example tokens Type Examples Identifier x, y, z, foo, bar NUM 42 FLOATNUM -3.141592654 STRING “so long, and thanks for all the fish” LPAREN ( RPAREN ) IF if … 11 C++ example 1 • Splitting text into tokens can be tricky • How should the code below be split? vector<vector<int>> myVector >> >, > or operator two tokens ? 12 C++ example 2 • Splitting text into tokens can be tricky • How should the code below be split? vector<vector<int> > myVector >, > two tokens 13 Separating tokens Type Examples Comments /* ignore code */ // ignore until end of line White spaces \t \n • Lexemes that are recognized but get consumed rather than transmitted to parser – if i f i/*comment*/f 14 Preprocessor directives in C Type Examples Include directives #include<foo.h> Macros #define THE_ANSWER 42 15 First step of designing a scanner • Define each type of lexeme – Reserved words: var, if, for, while – Operators: < = ++ – Identifiers: myFunction – Literals: 123 “hello” – Annotations: @SuppressWarnings • How can we define lexemes of unbounded length? 16 First step of designing a scanner • Define each type of lexeme – Reserved words: var, if, for, while – Operators: < = ++ – Identifiers: myFunction – Literals: 123 “hello” – Annotations: @SuppressWarnings • How can we define lexemes of unbounded length? – Regular expressions 17 Agenda • Understand role of lexical analysis in a compiler – Convert text to stream of tokens • Regular languages reminder • Lexical analysis algorithms • Scanner generation 18 Regular languages reminder 19 Basic definitions and facts • Formal languages – Alphabet = finite set of letters – Word = sequence of letter – Language = set of words • Regular languages defined equivalently by – Regular expressions – Finite-state automata 20 Regular expressions • Empty string: Є • Letter: a1, …, ak Alphabet • Concatenation: R1 R2 • Union: R1 | R2 • Kleene-star: R* – Shorthand: R+ stands for R R* • scope: (R) • Example: (0* 1*) | (1* 0*) – What is this language? 21 Exercise 1 - Question • Language of Java identifiers – Identifiers start with either an underscore ‘_’ or a letter – Continue with either underscore, letter, or digit 22 Exercise 1 - Answer • Language of Java identifiers – Identifiers start with either an underscore ‘_’ or a letter – Continue with either underscore, letter, or digit – (_|a|b|…|z|A|…|Z)(_|a|b|…|z|A|…|Z|0|…|9)* 23 Exercise 1 – Better answer • Language of Java identifiers – Identifiers start with either an underscore ‘_’ or a letter – Continue with either underscore, letter, or digit – (_|a|b|…|z|A|…|Z)(_|a|b|…|z|A|…|Z|0|…|9)* – Using shorthand macros First = _|a|b|…|z|A|…|Z Next = First|0|…|9 R = First Next* 24 Exercise 2 - Question • Language of rational numbers in decimal representation (no leading, ending zeros) – Positive examples: • 0 • 123.757 • .933333 • 0.7 – Negative examples: • 007 • 0.30 25 Exercise 2 - Answer • Language of rational numbers in decimal representation (no leading, ending zeros) – Digit = 1|2|…|9 Digit0 = 0|Digit Num = Digit Digit0* Frac = Digit0* Digit Pos = Num | .Frac | 0.Frac| Num.Frac PosOrNeg = (Є|-)Pos R = 0 | PosOrNeg 26 Exercise 3 - Question • Equal number of opening and closing parenthesis: [n]n = [], [[]], [[[]]], … 27 Exercise 3 - Answer • Equal number of opening and closing parenthesis: [n]n = [], [[]], [[[]]], … • Not regular • Context-free • Grammar: S ::= [] | [S] 28 Finite automata 29 Finite automata: known results • Types of finite automata: – Deterministic (DFA) – Non-deterministic (NFA) – Non-deterministic + epsilon transitions • Theorem: translation of regular expressions to NFA+epsilon (linear time) • Theorem: translation of NFA+epsilon to DFA – Worst-case exponential time • Theorem [Myhill-Nerode]: DFA can be minimized 30 Finite automata • An automaton M = Q, , , q0, F is defined by states and transitions transition accepting b state c a start b start state 31 Exercise - Question • What is the language defined by the automaton below? b c a start b 32 Exercise - Answer • What is the language defined by the automaton below? – a b* c – Generally: all paths leading to accepting states b c a start b 33 Non-deterministic automata • Allow multiple transitions from given state labeled by same letter b c a start c a b 34 NFA+Є automata • Є transitions can “fire” without reading the input b a c start Є 35 A little about me • Joined Ben-Gurion University in 2012 • Research interests – Inductive programming and synthesis – Static analysis and verification – Language-supported parallelism 36 I am here for • Teaching you theory and practice of popular compiler algorithms – Hopefully make you think about solving problems by examples from the compilers world – Answering questions about material • Contacting me – e-mail: [email protected] – Office hours: see course web-page • Announcements • Forums (per assignment) 37 Tentative syllabus Front Intermediate Code Optimizations End Representation Generation Operational Dataflow Register Scanning Semantics Analysis Allocation Top-down Loop Energy Lowering Parsing (LL) Optimizations Optimization Bottom-up Instruction Parsing (LR) Selection mid-term exam 38 Reg-exp vs. automata A high-level • Regular expressions are declarative language – Offer compact way to define a regular language by humans – Don’t offer direct way to check whether a given word is in the language A machine • Automata are operative language – Define an algorithm for deciding whether a given word is in a regular language – Not a natural notation for humans 39 From Regular expressions to automata 40 From reg. exp. to NFA+Є automata • Theorem: there is an algorithm to build an NFA+Є automaton for any regular expression • Proof: by induction on the structure of the regular expression start 41 Inductive constructions R = start R1 R1 | R2 a start R = a start R2 R R* R1 R2 R1 R2 start start 42 Running time of NFA+Є • Construction requires O(k) states for a reg-exp of length k • Running an NFA+Є with k states on string of length n takes O(n·k2) time 2 Each state in a configuration of – Can we reduce the k factor? O(k) states may have O(k) outgoing edges, so processing an input letter may take O(k2) time 43 From NFA+Є to DFA • Construction requires O(k) states for a reg-exp of length k • Running an NFA+Є with k states on string of length
Load more

Recommended publications
  • Modern Compiler Implementation in Java. Second Edition
    Team-Fly Modern Compiler Implementation in Java, Second Edition by Andrew W. ISBN:052182060x Appel and Jens Palsberg Cambridge University Press © 2002 (501 pages) This textbook describes all phases of a compiler, and thorough coverage of current techniques in code generation and register allocation, and the compilation of functional and object-oriented languages. Table of Contents Modern Compiler Implementation in Java, Second Edition Preface Part One - Fundamentals of Compilation Ch apt - Introduction er 1 Ch apt - Lexical Analysis er 2 Ch apt - Parsing er 3 Ch apt - Abstract Syntax er 4 Ch apt - Semantic Analysis er 5 Ch apt - Activation Records er 6 Ch apt - Translation to Intermediate Code er 7 Ch apt - Basic Blocks and Traces er 8 Ch apt - Instruction Selection er 9 Ch apt - Liveness Analysis er 10 Ch apt - Register Allocation er 11 Ch apt - Putting It All Together er 12 Part Two - Advanced Topics Ch apt - Garbage Collection er 13 Ch apt - Object-Oriented Languages er 14 Ch apt - Functional Programming Languages er 15 Ch apt - Polymorphic Types er 16 Ch apt - Dataflow Analysis er 17 Ch apt - Loop Optimizations er 18 Ch apt - Static Single-Assignment Form er 19 Ch apt - Pipelining and Scheduling er 20 Ch apt - The Memory Hierarchy er 21 Ap pe ndi - MiniJava Language Reference Manual x A Bibliography Index List of Figures List of Tables List of Examples Team-Fly Team-Fly Back Cover This textbook describes all phases of a compiler: lexical analysis, parsing, abstract syntax, semantic actions, intermediate representations, instruction selection via tree matching, dataflow analysis, graph-coloring register allocation, and runtime systems.
    [Show full text]
  • CS 444: Compiler Construction
    CS 444: Compiler Construction Gabriel Wong Contents Introduction 3 Front-end Analysis . .3 Formal Languages . .3 Lexical Analysis / Scanning 3 Regular Expressions . .3 Regex to DFA . .4 Definition of NFA and DFA . .4 Computing -closure . .4 NFA to DFA Conversion . .5 Scanning . .6 Constructing Scanning DFA . .6 Parsing 7 Context-Free Grammar . .7 Top-down / Bottom-up Parsing . .8 LL(1) Parsing . .9 Augmented Grammar . 10 LL(1) Algorithm . 10 LR(0) Parsing . 11 LR(1) Parsing . 15 SLR(1) Parser . 15 Comparison Between LR Parsers . 16 LR(1) NFA . 16 LALR(1) Parser . 17 Abstract Syntax Tree . 19 Semantic Analysis 19 Implementing Environments . 20 Namespaces . 20 Java Name Resolution . 21 1. Create class environment . 21 2. Resolve type names . 21 3. Check class hierachy . 21 4. Disambiguate ambiguous names . 23 5. Resolve expressions (variables, fields) . 23 6. Type checking . 24 7. Resolve methods and instance fields . 27 Static Analysis 28 Java Reachability Analysis (JLS 14.20) . 28 Java Definite Assignment Analysis (JLS 16) . 29 1 CONTENTS CONTENTS Live Variable Analysis . 30 Code Generation 31 IA-32 Assembly . 32 Assembler Directives . 32 Strategy for Code Generation . 33 Data Layout . 33 Constants . 34 Local Variables . 34 Method Calls . 35 Object Layout . 36 Vtables . 37 Dispatching Interface Methods . 37 Subtype Testing . 38 Arrays................................................. 38 2 Regular Expressions LEXICAL ANALYSIS / SCANNING Notes taken from lectures by Ondřej Lhoták in Winter 2016. Introduction A compiler translates from a source language (eg. Java) to a target machine language (eg. x86). Scanning, parsing (A1) and context sensitive analysis (A234) are part of the compiler front end.
    [Show full text]
  • Context-Aware Scanning and Determinism-Preserving Grammar Composition, in Theory and Practice
    Context-Aware Scanning and Determinism-Preserving Grammar Composition, in Theory and Practice A THESIS SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL OF THE UNIVERSITY OF MINNESOTA BY August Schwerdfeger IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY July, 2010 c August Schwerdfeger 2010 ALL RIGHTS RESERVED Acknowledgments I would like to thank all my colleagues in the MELT group for all their input and assis- tance through the course of the project, and especially for their help in providing ready- made tests and applications for my software. Thanks specifically to Derek Bodin and Ted Kaminski for their efforts in integrating the Copper parser generator into MELT’s attribute grammar tools; to Lijesh Krishnan for developing the ableJ system of Java ex- tensions and assisting me extensively with its further development; and to Yogesh Mali for implementing and testing the Promela grammar with Copper. I also thank my advisor, Eric Van Wyk, for his continuous guidance throughout my time as a student, and also for going well above and beyond the call of duty in helping me to edit and prepare this thesis for presentation. I thank the members of my thesis committee, Mats Heimdahl, Gopalan Nadathur, and Wayne Richter, for their time and efforts in serving and reviewing. I especially wish to thank Prof. Nadathur for his many detailed questions and helpful suggestions for improving this thesis. Work on this thesis has been partially funded by National Science Foundation Grants 0347860 and 0905581. Support has also been received from funds provided by the Institute of Technology (soon to be renamed the College of Science and Engi- neering) and the Department of Computer Science and Engineering at the University of Minnesota.
    [Show full text]
  • Rewriting Strategies for Instruction Selection
    Rewriting Strategies for Instruction Selection Martin Bravenboer Eelco Visser www.stratego-language.org Technical Report UU-CS-2002-021 Institute of Information and Computing Sciences Utrecht University April 2002 This technical report is a preprint of: M. Bravenboer and E. Visser. Rewriting Strategies for Instruction Selection. To appear in S. Tison (editor) Rewriting Techniques and Applications (RTA'02). Lecture Notes in Computer Science. Springer-Verlag, Copenhagen, Denmark, June 2002. ( c Springer-Verlag) Copyright c 2002 Martin Bravenboer and Eelco Visser Address: Institute of Information and Computing Sciences Universiteit Utrecht P.O.Box 80089 3508 TB Utrecht email: [email protected] http://www.cs.uu.nl/~visser/ Rewriting Strategies for Instruction Selection Martin Bravenboer Eelco Visser Institute of Information and Computing Sciences, Universiteit Utrecht, P.O. Box 80089, 3508 TB Utrecht, The Netherlands. http://www.cs.uu.nl/ visser, [email protected], [email protected] ∼ Abstract. Instruction selection (mapping IR trees to machine instruc- tions) can be expressed by means of rewrite rules. Typically, such sets of rewrite rules are highly ambiguous. Therefore, standard rewriting en- gines based on fixed, exhaustive strategies are not appropriate for the execution of instruction selection. Code generator generators use spe- cial purpose implementations employing dynamic programming. In this paper we show how rewriting strategies for instruction selection can be encoded concisely in Stratego, a language for program transformation based on the paradigm of programmable rewriting strategies. This em- bedding obviates the need for a language dedicated to code generation, and makes it easy to combine code generation with other optimizations. 1 Introduction Code generation is the phase in the compilation of high-level programs in which an intermediate representation (IR) of a program is translated to a list of machine instructions.
    [Show full text]
  • Lecture 9 Code Generation Instruction Selection Instruction Selection
    Lecture 9 Code Generation Instruction Selection Instruction Selection • Mapping IR trees to assembly instructions • Tree patterns • Retain virtual registers (temps) • Algorithms for instruction selection • Maximal munch • Dynamic programming • Tree grammars • Instruction selection for the Minijava compiler • Generic assembly instructions (Asm.instr) Instructions and Tree Patterns + * ADD ri ← rj + rk MUL ri ← rj * rk LOAD ri ← M[rj + c] FETCH FETCH FETCH FETCH MEM MEM MEM MEM CONST + + CONST CONST STORE M[ri + c] ← rj MOVE MOVE MOVE MOVE MEM MEM MEM MEM CONST + + CONST CONST Tiling with Instruction Patterns a[i] := x where i = TEMPi a = M[fp + ca] = ca($fp) x = M[fp + cx] = cx($fp) MOVE MEM FETCH + MEM FETCH * + FP CONST(c ) MEM TEMPi CONST(4) x + FP CONST(ca) Tiling with Instruction Patterns MOVE a[i] := x MOVEM M[ri ] ← M[rj] MEM FETCH MEM MOVE MEM FETCH + MEM FETCH * + FP CONST(c ) MEM TEMPi CONST(4) x + FP CONST(ca) Optimal, Optimum Tilings A tiling of an IR tree with instruction patterns is optimum if the set of patterns has the least “cost” by some measure, e.g.: 1. the smallest number of patterns (producing the shortest instruction sequence, thus smallest code size) 2. total number of cycles to execute, thus fastest code A tiling is optimal if no two adjacent tiles can be combined into a single tile of lower cost. optimum ⇒ optimal Maximal Munch 1. Choose the “largest” tile that matches at the root of the IR tree (this is the munch) 2. Recursively apply maximal much at each subtree of this munch.
    [Show full text]
  • One Parser to Rule Them All
    One Parser to Rule Them All Ali Afroozeh Anastasia Izmaylova Centrum Wiskunde & Informatica, Amsterdam, The Netherlands {ali.afroozeh, anastasia.izmaylova}@cwi.nl Abstract 1. Introduction Despite the long history of research in parsing, constructing Parsing is a well-researched topic in computer science, and parsers for real programming languages remains a difficult it is common to hear from fellow researchers in the field and painful task. In the last decades, different parser gen- of programming languages that parsing is a solved prob- erators emerged to allow the construction of parsers from a lem. This statement mostly originates from the success of BNF-like specification. However, still today, many parsers Yacc [18] and its underlying theory that has been developed are handwritten, or are only partly generated, and include in the 70s. Since Knuth’s seminal paper on LR parsing [25], various hacks to deal with different peculiarities in program- and DeRemer’s work on practical LR parsing (LALR) [6], ming languages. The main problem is that current declara- there is a linear parsing technique that covers most syntactic tive syntax definition techniques are based on pure context- constructs in programming languages. Yacc, and its various free grammars, while many constructs found in program- ports to other languages, enabled the generation of efficient ming languages require context information. parsers from a BNF specification. Still, research papers and In this paper we propose a parsing framework that em- tools on parsing in the last four decades show an ongoing braces context information in its core. Our framework is effort to develop new parsing techniques.
    [Show full text]
  • Instruction Selection
    Lecture Notes on Instruction Selection 15-411: Compiler Design Frank Pfenning∗ Lecture 2 1 Introduction In this lecture we discuss the process of instruction selection, which typ- cially turns some form of intermediate code into a pseudo-assembly lan- guage in which we assume to have infinitely many registers called “temps”. We next apply register allocation to the result to assign machine registers and stack slots to the temps before emitting the actual assembly code. Ad- ditional material regarding instruction selection can be found in the text- book [App98, Chapter 9]. 2 A Simple Source Language We use a very simple source language where a program is just a sequence of assignments terminated by a return statement. The right-hand side of each assignment is a simple arithmetic expression. Later in the course we describe how the input text is parsed and translated into some intermediate form. Here we assume we have arrived at an intermediate representation where expressions are still in the form of trees and we have to generate in- structions in pseudo-assembly. We call this form IR Trees (for “Intermediate Representation Trees”). We describe the possible IR trees in a kind of pseudo-grammar, which should not be read as a description of the concrete syntax, but the recursive structure of the data. ∗Edited by Andre´ Platzer LECTURE NOTES L2.2 Instruction Selection Programs ~s ::= s1; : : : ; sn sequence of statements Statements s ::= t = e assignment j return e return, always last Expressions e ::= c integer constant j t temp (variable) j e1 ⊕ e2 binary operation Binops ⊕ ::= + j − j ∗ j = j ::: 3 Abstract Assembly Code Target For our very simple source, we use an equally simple target.
    [Show full text]
  • Compilers & Programming Systems
    CS502: Compilers & Programming Systems Low IR and Assembly code generation Zhiyuan Li Department of Computer Science Purdue University, USA • We first discuss the generation of low-level intermediate code of a program. – Inside the compiler such a code has its low-level representation, called low-level internal representation, or low IR. • We then discuss the generation of assembly code by walking through the low-level IR Motivation of machine independent representation • From the MIPS code example shown previously, we see that the assembly code is heavily dependent on the specific instruction format of a particular processor • We want to perform optimizations on machine independent representation such that we do not implement a new set of optimizations for each particular processor Differences between Low IR and Assembly Code •In low IR – Unlimited user-level registers are assumed. Hence, • symbolic registers, instead of hardware registers are named as operands. – Operators on different data type are overloaded. • For example, “+” represents both integer add and float add. – Instructions selected for special cases are abstracted away. • For example, a simple load instruction represents loading of a value, instead of two instructions (loading the high half and then the low half) as on RISC processors such as MIPS and SPARC. • Another example is that we simply write R4 * 2 when it could be done by a potentially less expensive left shift instruction. Intermediate Code • A well-known form of intermediate code is called three- address code (3AC) – In the “quadruples” form, 3AC for a function is represented by a list quadruples (dest, op1, op2, op), with “op” being the operation – An alternative form is a list of “triples” (op1, op2, op) and the index or the pointer to a specific triple implies the location of the intermediate result • When we plug such locations (indices or pointers) to the operand fields of an operation, we in effect obtain a forest, i.e.
    [Show full text]
  • 7. Code Generation
    7. Code Generation Oscar Nierstrasz Thanks to Jens Palsberg and Tony Hosking for their kind permission to reuse and adapt the CS132 and CS502 lecture notes. http://www.cs.ucla.edu/~palsberg/ http://www.cs.purdue.edu/homes/hosking/ Roadmap > Runtime storage organization > Procedure call conventions > Instruction selection > Register allocation > Example: generating Java bytecode See Modern compiler implementation in Java (Second edition), chapters 6 & 9. !2 Roadmap > Runtime storage organization > Procedure call conventions > Instruction selection > Register allocation > Example: generating Java bytecode !3 Typical run-time storage organization Heap grows “up”, stack grows “down”. • Allows both stack and heap maximal freedom. • Code and static data may be separate or intermingled. !4 In a 32-bit architecture, memory addresses range from from 0 to 4GB, broken into pages, of which only the low and the high pages are actually allocated. The low pages hold the compiled program code, static data, and heap data. The heap “grows” upwards as needed. High memory addresses refer to the run-time stack. They “grow” downward with each procedure call and “shrink” upward with each return. Certain memory pages (e.g., holding compiled code) may possibly be protected against modification. The Procedure Abstraction > The procedure abstraction supports separate compilation —build large programs —keep compile times reasonable —independent procedures > The linkage convention (calling convention): —a social contract — procedures inherit a valid run-time environment and restore one for their parents —platform dependent — code generated at compile time !5 Procedures as abstractions function foo() function bar(int a) { { int a, b; int x; ... ... bar(a); bar(x); ... ... } } bar() must preserve foo()’s state while executing.
    [Show full text]
  • Javacc Tutorial
    Chapter 1 Introduction to JavaCC 1.1 JavaCC and Parser Generation JavaCC is a parser generator and a lexical analyzer generator. Parsers and lexical analysers are software components for dealing with input of character sequences. Compilers and interpreters incorporate lexical analysers and parsers to decipher files containing programs, however lexical analysers and parsers can be used in a wide variety of other applications, as I hope the examples in this bookwill illustrate. So what are lexical analysers and parsers? Lexical analysers can breaka sequence of characters into a subsequences called tokens and it also classifies the tokens. Consider a short program in the C programming language. int main() { return 0 ; } The lexical analyser of a C compiler would breakthis into the following sequence of tokens “int”, “ ”, “main”, “(”, “)”, “”,“{”, “\n”, “\t”, “return” “”,“0”,“”,“;”,“\n”, “}”, “\n”, “” . The lexical analyser also identifies the kind of each token; in our example the sequence of 2 token kinds might be KWINT, SPACE, ID, OPAR, CPAR, SPACE, OBRACE, SPACE, SPACE, KWRETURN, SPACE, OCTALCONST, SPACE, SEMICOLON, SPACE, CBRACE, SPACE, EOF . The token of kind EOF represents the end of the original file. The sequence of tokens is then passed on to the parser. In the case of C, the parser does not need all the tokens; in our example, those clasified as SPACE are not passed on to the parser. The parser then analyses the sequence of tokens to determine the structure of the program. Often in compilers, the parser outputs a tree representing the structure of the program. This tree then serves as an input to components of the compiler responsible for analysis and code generation.
    [Show full text]
  • Instruction Selection Compiler Backend Intermediate Representations
    Instruction Selection 15-745 Optimizing Compilers Spring 2007 1 School of Computer Science School of Computer Science Compiler Backend Intermediate Representations instruction IR IR IR Assem Source Front Middle Back Target selector Code End End End Code Front end - produces an intermediate representation (IR) register TempMap allocator Middle end - transforms the IR into an equivalent IR that runs more efficiently Back end - transforms the IR into native code instruction Assem scheduler IR encodes the compiler’s knowledge of the program Middle end usually consists of several passes 2 3 School of Computer Science School of Computer Science Page ‹#› Types of Intermediate Intermediate Representations Representations Decisions in IR design affect the speed and efficiency of Structural the compiler – Graphically oriented Examples: Some important IR properties – Heavily used in source-to-source translators Trees, DAGs – Ease of generation – Tend to be large – Ease of manipulation Linear – Procedure size – Pseudo-code for an abstract machine Examples: – Freedom of expression – Level of abstraction varies 3 address code Stack machine code – Level of abstraction – Simple, compact data structures The importance of different properties varies between – Easier to rearrange compilers Hybrid Example: – Selecting an appropriate IR for a compiler is critical – Combination of graphs and linear code Control-flow graph 4 5 School of Computer Science School of Computer Science Level of Abstraction Level of Abstraction The level of detail exposed in an IR
    [Show full text]
  • Back-End Code Generation Itree Stmts and Exprs Side-Effects
    CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Back-End Code Generation Itree Stmts and Exprs • Given a list of itree fragments, how to generate the corresponding assembly • itree statements stm and itree expressions exp code ? datatype stm = SEQ of stm * stm | LABEL of label datatype frag | JUMP of exp = PROC of {name : Tree.label, function name | CJUMP of test * label * label body : Tree.stm, function body itree | MOVE of exp * exp frame : Frame.frame} static frame layout | EXP of exp | DATA of string and exp = BINOP of binop * exp * exp | CVTOP of cvtop * exp * size * size • Main challenges: certain aspects of itree statements and expressions do not | MEM of exp * size correspond exactly with machine languages: | TEMP of temp | NAME of label # of temp. registers on real machines are limited | CONST of int | CONSTF of real real machine’s conditional-JUMP statement takes only one label | ESEQ of stm * exp | CALL of exp * exp list high-level constructs ESEQ and CALL ---- side-effects Copyright 1994 - 2017 Zhong Shao, Yale University Machine Code Generation: Page 1 of 18 Copyright 1994 - 2017 Zhong Shao, Yale University Machine Code Generation: Page 2 of 18 CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Side-Effects Summary: IR -> Machine Code • Side-effects means updating the contents of a memory cell or a temporary • Step #1 : Transform the itree code into a list of canonical trees register. What are itree expressions that might cause side effects ? a. eliminate SEQ and ESEQ nodes ESEQ and CALL nodes
    [Show full text]