Lexical Analysis Example: Source Code

CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Lexical Analysis Example: Source Code

A Sample Toy Program: • Read source program and produce a list of tokens (“linear” analysis) (* define valid mutually recursive procedures *) let token source lexical parser function do_nothing1(a: int, b: string)= program analyzer do_nothing2(a+1) get next token function do_nothing2(d: int) = do_nothing1(d, “str”)

• The lexical structure is specified using regular expressions in do_nothing1(0, “str2”) • Other secondary tasks: end (1) get rid of white spaces (e.g., \t,\n,\sp) and comments

(2) line numbering What do we really care here ?

CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS The Lexical Structure Tokens Output after the Lexical Analysis ----- token + associated value • Tokens are the atomic unit of a language, and are usually specific strings or

LET 51 FUNCTION 56 ID(do_nothing1) 65 instances of classes of strings. LPAREN 76 ID(a) 77 COLON 78 Tokens Sample Values Informal Description ID(int) 80 COMMA 83 ID(b) 85 COLON 86 ID(string) 88 RPAREN 94 LET let keyword LET EQ 95 ID(do_nothing2) 99 END end keyword END LPAREN 110 ID(a) 111 PLUS 112 PLUS + INT(1) 113 RPAREN 114 FUNCTION 117 LPAREN ( ID(do_nothing2) 126 LPAREN 137 ID(d) 138 COLON 139 ID(int) 141 COLON : RPAREN 144 EQ 146 STRING “str” ID(do_nothing1) 150 LPAREN 161 RPAREN ) ID(d) 162 COMMA 163 STRING(str) 165 INT 49, 48 integer constants RPAREN 170 IN 173 ID do_nothing1, a, letter followed by letters, digits, and under- ID(do_nothing1) 177 LPAREN 188 int, string scores INT(0) 189 COMMA 190 STRING(str2) 192 EQ = RPAREN 198 END 200 EOF 203 EOF end of file

Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 3 of 40 Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 4 of 40 CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Lexical Analysis, How? Regular Expressions • regular expressions are concise, linguistic characterization of regular • First, write down the lexical specification (how each token is defined?) languages (regular sets)

* using regular expression to specify the lexical structure: identifier = letter (letter | digit | underscore)

identifier = letter (letter | digit | underscore)* “or” “ 0 or more” letter = a | ... | z | A | ... | Z digit = 0 | 1 | ... | 9 • each regular expression define a regular language --- a set of strings over some alphabet, such as ASCII characters; each member of this set is called a • Second, based on the above lexical specification, build the lexical analyzer (to sentence, or a word recognize tokens) by hand, Regular Expression Spec ==> NFA ==> DFA ==>Transition Table ==> Lexical Analyzer • we use regular expressions to define each category of tokens

• Or just by using lex --- the lexical analyzer generator For example, the above identifier specifies a set of strings that are a sequence of letters, digits, and underscores, starting with a letter. Regular Expression Spec (in lex format) ==> feed to lex ==> Lexical Analyzer

CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Regular Expressions and Regular Example

Languages Regular Expression Explanation • Given an alphabet the regular expressions over  and their corresponding a* 0 or more a’s regular languages are a+ 1 or more a’s * a) denotes  ,the empty string, denotes the language {  }. (a|b) all strings of a’s and b’s (including ) (aa|ab|ba|bb)* all strings of a’s and b’s of even length b) for each a in , a denotes { a } --- a language with one string. [a-zA-Z] shorthand for “a|b|...|z|A|...|Z”

c) if R denotes LR and S denotes LS then R | S denotes the language [0-9] shorthand for “0|1|2|...|9” L L , i.e, { x | x L or x L }. R S R S 0([0-9])*0 numbers that start and end with 0 * ? d) if R denotes LR and S denotes LS then RS denotes the language (ab|aab|b) (a|aa|e) LRLS , that is, { xy | x LRand y LS }. ? all strings that contain foo as substring * * * e) if R denotes LR then R denotes the language LR where L is the i i n n union of all L (i=0,...,and L is just {x1x2...xi | x1L, ..., xiL}. • the following is not a regular expression: a b (n > 0)

f) if R denotes LR then (R) denotes the same language LR.

Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 7 of 40 Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 8 of 40 CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Lexical Specification Transition Diagrams • Flowchart with states and edges; each edge is labelled with characters; certain •Using regular expressions to specify tokens subset of states are marked as “final states”

keyword = begin | end | if | then | else • Transition from state to state proceeds along edges according to the next input * identifier = letter (letter | digit | underscore) character integer = digit+ relop = < | <= | = | <> | > | >= letter digit underscore letter = a | b | ... | z | A | B | ... | Z start letter delimiter final state digit = 0 | 1 | 2 | ... | 9 0 1 2 • Ambiguity : is “begin” a keyword or an identifier ?

• Next step: to construct a token recognizer for languages given by regular • Every string that ends up at a final state is accepted expressions --- by using finite automata ! • If get “stuck”, there is no transition for a given character, it is an error given a string x, the token recognizer says “yes” if x is a sentence of the specified language and says “no” otherwise • Transition diagrams can be easily translated to programs using case statements (in C).

CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Transition Diagrams (cont’d) Finite Automata

The token recognizer (for identifiers) based on transition diagrams: • Finite Automata are similar to transition diagrams; they have states and labelled edges; there are one unique start state and one or more than one final state0: c = getchar(); states if (isalpha(c)) goto state1; error(); • Nondeterministic Finite Automata (NFA) : ... state1: c = getchar(); a)  can label edges (these edges are called -transitions) if (isalpha(c) || isdigit(c) || b) some character can label 2 or more edges out of the same state isunderscore(c)) goto state1; if (c == ‘,’ || ... || c == ‘)’) goto state2; • Deterministic Finite Automata (DFA) : error(); a) no edges are labelled with  ... b) each charcter can label at most one edge out of the same state state2: ungetc(c,stdin); /* retract current char */ return(ID, ... the current identifier ...); • NFA and DFA accepts string x if there exists a path from the start state to a final state labeled with characters in x Next: 1. finite automata are generalized transition diagrams ! 2. how to build finite automata from regular expressions? NFA: multiple paths DFA: one unique path

Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 11 of 40 Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 12 of 40 CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Example: NFA Example: DFA

start a final state start b b final state state state a b 0 1 2 3 0 1 2 3 start a b b start b a a a b An NFA accepts (a|b)*abb A DFA accepts (a|b)*abb

There are many possible moves --- to accept a string, we only need one There is only one possible sequence of moves --- either lead to a final state and sequence of moves that lead to a final state. accept or the input string is rejected

input string: aabb a a b b input string: aabb One sucessful sequence: 0 0 1 2 3 The sucessful sequence: a a b b Another unsuccessful sequence: a a b b 0 1 1 2 3 0 0 0 0 0

CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Transition Table NFA with -transitions • Finite Automata can also be represented using transition tables 1. NFA can have -transitions --- edges labelled with 

For NFA, each entry is a set of states: For DFA, each entry is a unique state: a 1 2  a STATE ab STATE ab 0 0 {0,1} {0} 010 b  1-{2} 112 3 4 b 2-{3} 213 3-- 310 accepts the regular language denoted by (aa*|bb*)

Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 15 of 40 Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 16 of 40 CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Regular Expressions -> NFA RE -> NFA (cont’d) • How to construct NFA (with -transitions) from a regular expression ? 2. “Inductive” Construction N1 : NFA for R1 • Algorithm : apply the following construction rules , use unique names for all N2 : NFA for R2 • R1 | R2 the states. (inportant invariant: always one final state !)  N1  i f   1. Basic Construction the new and N2 unique final state initial state  final state •  for N1 and N2 i f for N1 and N2

merge : final state of N1 •a  a and initial state of N i f • R1 R2 2

initial state N1 N2 final state for N1 for N2

CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS RE -> NFA (cont’d) Example : RE -> NFA Converting the regular expression : (a|b)*abb 2. “Inductive” Construction (cont’d) a a (in a|b)===> 2 3

initial state b * for N final state • R 1 N1 : NFA for R1 b (in a|b)===> 1 for N1 4 5

  i N1 f a 2 3     a|b ====> 1 6 b  4 5 

Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 19 of 40 Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 20 of 40 CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Example : RE -> NFA (cont’d) Example : RE -> NFA (cont’d) Converting the regular expression : (a|b)*abb Converting the regular expression : (a|b)*abb

(a|b)* ====>  (a|b)*abb ====> 

a a 2 3  2 3        0 1 6 7 0 1 6 7 b b  4 5   4 5   

abb ====> (several steps are omitted) a a b b b b X 8 9 10 8 9 10

CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS NFA -> DFA NFA -> DFA (cont’d) • NFA are non-deterministic; need DFA in order to write a deterministic prorgam • each DFA-state is a set of NFA-states ! • suppose the start state of the NFA is s, then the start state for its DFA is - • There exists an algorithm (“subset construction”) to convert any NFA to a DFA CLOSURE(s) ; the final states of the DFA are those that include a NFA-final-state that accepts the same language • Algorithm : converting an NFA N into a DFA D ----

• States in DFA are sets of states from NFA; DFA simulates “in parallel” all Dstates = {e-CLOSURE(s0),s0 is N’s start state} possible moves of NFA on given input. Dstates are initially “unmarked” while there is an unmarked D-state X do { • Definition: for each state s in NFA, mark X for each a in S do { -CLOSURE(s) = { s }  { t | s can reach t via -transitions } T = {states reached from any si in X via a}  Y = e-CLOSURE(T) if Y not in Dstates then add Y to Dstates “unmarked” • Definition: for each set of states S in NFA, add transition from X to Y, labelled with a } -CLOSURE(S) = i -CLOSURE(s) for all si in S }

Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 23 of 40 Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 24 of 40 CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Example : NFA -> DFA Example : NFA -> DFA (cont’d) • converting NFA for (a|b)*abb to a DFA ------B’s b-transitions: T = {5, 9}; a new state D = -CLOSURE({5, 9}) = {1, 2, 4, 5, 6, 7, 9} The start state A = -CLOSURE(0) = {0, 1, 2, 4, 7}; Dstates={A} add a transition from B to D labelled with b Dstates = {A, B, C, D} 1st iteration: A is unmarked; mark A now; a-transitions: T = {3, 8} then we pick C, and mark C a new state B= -CLOSURE(3) -CLOSURE(8) C’s a-transitions: T = {3, 8}; its -CLOSURE is B. = {3, 6, 1, 2, 4, 7} 8) = {1, 2, 3, 4, 6, 7, 8} add a transition from C to B labelled with a add a transition from A to B labelled with a C’s b-transitions: T = {5}; its -CLOSURE is C itself. add a transition from C to C labelled with b b-transitions: T = {5} a new state C = -CLOSURE(5) = {1, 2, 4, 5, 6, 7} next we pick D, and mark D add a transition from A to C labelled with b D’s a-transitions: T = {3, 8}; its -CLOSURE is B. Dstates = {A, B, C} add a transition from D to B labelled with a D’s b-transitions: T = {5, 10}; 2nd iteration: B, C are unmarked; we pick B and mark B first; a new state E = -CLOSURE({5, 10}) = {1, 2, 4, 5, 6, 7, 10} B = {1, 2, 3, 4, 6, 7, 8} Dstates = {A, B, C, D, E}; E is a final state since it has 10; B’s a-transitions: T = {3, 8}; T’s -CLOSURE is B itself. add a transition from B to B labelled with a next we pick E, and mark E

CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Example : NFA -> DFA (cont’d) Other Algorithms E’s a-transitions: T = {3, 8}; its -CLOSURE is B. add a transition from E to B labelled with a • How to minimize a DFA ? (see Dragon Book 3.9, pp141) E’s b-transitions: T = {5}; its -CLOSURE is C itself. add a transition from E to C labelled with b

all states in Dstates are marked, the DFA is constructed ! • How to convert RE to DFA directly ? (see Dragon Book 3.9, pp135) a a a • How to prove two Regular Expressions are equivalent ? (see Dragon Book a b b pp150, Exercise 3.22) A B D E a

b C b

Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 27 of 40 Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 28 of 40 CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Lex Lex Specification • Lex is a program generator ------it takes lexical specification as input, and produces a lexical processor written in C. DIGITS [0-9] lex definition ......

Lex %% Specification Lex lex.yy.c expression action foo.l integer printf(“INT”); translation rules ...... lex.yy.c C Compiler a.out %% ..... char getc() { ...... user’s C functions (optional) input text a.out sequence of tokens }

• expression is a regular expression ; action is a piece of C program; • Implementation of Lex: • for details, read the Lesk&Schmidt paper Lex Spec -> NFA -> DFA -> Transition Tables + Actions -> yylex()

CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS ML-Lex ML-Lex Specification • ML-Lex is like Lex ------it takes lexical specification as input, and produces type pos = int user’s ML a lexical processor written in Standard ML. val lineNum = ... declarations val lexresult = ...... Lex %% Specification ML-Lex foo.lex.sml %s COMMENT STRING; foo.lex SPACE=[ \t\n\012]; ml-lex definitions DIGITS=[0-9]; ..... foo.lex.sml ML Compiler module Mlex %% expression => (action); integer => (print(“INT”)); translation rules input text Mlex sequence of tokens ...... => (...lineNum...); can call the above ML declarations

• expression is a regular expression ; action is a piece of ML program; when the • Implementation of ML-Lex is similar to implementation of Lex input matches the expression, the action is executed, the text matched is placed in the variable yytext.

Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 31 of 40 Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 32 of 40 CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS What does ML-Lex generate? ML-Lex Definitions • Things you can write inside the “ml-lex definitions” section (2nd part): foo.lex ML-Lex foo.lex.sml %s COMMENT STRING define new start states sample foo.lex.sml: everything in part 1 of foo.lex %reject REJECT() to reject a match structure Mlex = %count count the line number struct %structure {identifier} the resulting structure name structure UserDeclarations = struct ... end (the default is Mlex) ...... fun makeLexer yyinput = .... (hint: you probably don’t need use %reject, %count,or %structure end for assignment 2.) Definition of named regular expressions : To use the generated lexical processor: val lexer = identifier = regular expression Mlex.makeLexer(fn _ => input (openIn “toy”)); val nextToken = lexer() SPACE=[ \t\n\012] input filename IDCHAR=[_a-zA-Z0-9] each call returns one token !

CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS ML-Lex Translation Rules ML-Lex Translation Rules (cont’d) • Each translation rule (3rd part) are in the form • Valid ML-Lex regular expressions: (cont’d)

regular expression => (action); escape sequences: (can be used inside or outside square brackets) \b backspace • Valid ML-Lex regular expressions: (see ML-Lex-manual pp 4-6) \n newline \t tab a character stands for itself except for the reserved chars: \ddd any ascii char (ddd is 3 digit decimal) ? * + | ( ) ^ $ / ; . = < > [ { ” \ to use these chars, use backslash! for example, \\\” represents . any char except newline (equivalent to [^\n]) the string \” “x” match string x exactly even if it contains reserved chars x? an optional x using square brackets to enclose a set of characters x* 0 or more x’s ( \ - ^ are reserved) x+ 1 or more x’s x|y x or y [abc] char a, or b, or c ^x if at the beginning, match at the beginning of a line only [^abc] all chars except a, b, c {x} substitute definition x (defined in the lex definition section) [a-z] all chars from a to z (x) same as regular expression x [\n\t\b] new line, tab, or backspace x{n} repeating x for n times [-abc] char - or a or b or c x{m-n} repeating x from m to n times

Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 35 of 40 Copyright 1994 - 2017 Zhong Shao, Yale University Lexical Analysis : Page 36 of 40 CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS ML-Lex Translation Rules (cont’d) Ambiguity what are valid actions ? • what if more than one translation rules matches ? • Actions are basically ML code (with the following extensions) A. longest match is preferred B. among rules which matched the same number of • All actions in a lex file must return values of the same type characters, the rule given first is preferred %% •Use yytext to refer to the current string %% 1 while => (Tokens.WHILE(...)); [a-z]+ => (print yytext); 2 [a-zA-Z][a-zA-Z0-9_]* => (Tokens.ID(yytext,...)); [0-9]{3} => (print (Char.ord(sub(yytext,0)))); 3 “<” => (Tokens.LESS(...)); 4 “<=” => (Tokens.LE(yypos,...)); • Can refer to anything defined in the ML-Declaration section (1st part)

• YYBEGIN start-state ----- enter into another start state input “while” matches rule 1 according B above • lex() and continue() to reinvoking the lexing function input “<=” matches rule 4 according A above • yypos --- refer to the current position

CS421 COMPILERS AND INTERPRETERS CS421 COMPILERS AND INTERPRETERS Start States (or Start Conditions) Implementation of Lex •start states permit multiple lexical analyzers to run together. • construct NFA for sum of Lex translation rules (regexp/action);

• each translation rule can be prefixed with • convert NFA to DFA, then minimize the DFA

• the lexer is initially in a predefined start stae called INITIAL • to recognize the input, simulate DFA to termination; find the last DFA state that includes NFA final state, execute associated action (this pickes longest match). • define new start states (in ml-lex-definitions): %s COMMENT STRING If the last DFA state has >1 NFA final states, pick one for rule that appears first • to switch to another start states (in action): YYBEGIN COMMENT • how to represent DFA, the transition table: • example: multi-line comments in C 2D array indexed by state and input-character too big ! %% %s COMMENT each state has a linked list of (char, next-state) pairs too slow! %% hybrid scheme is the best ”/*” => (YYBEGIN COMMENT; continue()); ”*/” => (YYBEGIN INITIAL; continue()); .|”\n” => (continue()); ......