This Lecture • Parser generators (“compiler compilers”) COMP3012/G53CMP: Lecture 4 • The parser generator Happy Syntactic Analysis: Parser Generators • A TXL parser written using Happy Henrik Nilsson • A TXL interpreter written using Happy University of Nottingham, UK COMP3012/G53CMP: Lecture 4 – p.1/32 COMP3012/G53CMP: Lecture 4 – p.2/32 Parser Generators (1) Parser Generators (2) • Constructing parsers by hand can be very An LR(0) DFA recognizing viable prefixes for tedious and time consuming. S → aABe A → bcA | c B → d • This is true in particular for LR(k) and LALR I 1 I I 5 I parsers: constructing the corresponding DFAs 0 S → a · ABe 8 a A S → aA · Be B is extremely laborious. S →·aABe A →·bcA S → aAB · e B →·d A →·c • E.g., this simple grammar (from the prev. lect.) I b c d I e 2 I3 I6 9 A → b · cA A → c · B → d · S → aABe · S → aABe c I4 A → bcA | c c I A → bc · A 7 b A B → d A →·bcA A → bcA · A →·c gives rise to a 10 state LR(0) DFA! COMP3012/G53CMP: Lecture 4 – p.3/32 COMP3012/G53CMP: Lecture 4 – p.4/32 Parser Generators (3) Parser Generators (4) • Parser construction is in many ways a very Consider an LR shift-reduce parser: mechanical process. Why not write a • Some of the actions when parsing abccde: program to do the hard work for us? State Stack (γ) Input (w) Move • A Parser Generator (or “compiler compiler”) . takes a grammar as input and outputs a I5 aA de Shift parser (a program) for that grammar. I6 aAd e Reduce by B → d I8 aAB e Shift • The input grammar is augmented with I9 aABe ǫ Reduce by S → aABe “semantic actions”: code fragments that get S ǫ Done invoked when a derivation step is performed. • A reduction corresponds to a derivation step • The semantic actions typically construct an in the grammar (an LR parser performs a AST or interpret the program being parsed. rightmost derivation in reverse). COMP3012/G53CMP: Lecture 4 – p.5/32 COMP3012/G53CMP: Lecture 4 – p.6/32 Parser Generators (5) Parser Generators (6) • At a reduction, the terminals and non- Some examples of parser generators: terminals of the RHS of the production (the • Yacc (“Yet Another Compiler Compiler”): A handle) are on the parse stack, associated classic UNIX LALR parser generator for C. with semantic information or semantic http://dinosaur.compilertools.net/ values; e.g., the corresponding AST fragments, expression values. • Bison: GNU project parser generator, a free Yacc replacement, for C and C++. • Think of the RHS symbols as variables bound to the the semantic value resulting • Happy: a parser generator for Haskell, similar from a successful derivation for that symbol. to Yacc and Bison. http://www.haskell.org/happy/ • Construction of AST, evaluation of expressions, etc. proceeds in bottom-up order. • Cup: LALR parser generator for Java. COMP3012/G53CMP: Lecture 4 – p.7/32 COMP3012/G53CMP: Lecture 4 – p.8/32 Parser Generators (7) Happy Parser for TXL (1) • ANTLR: LL(k) (recursive descent) parser We are going to develop a TXL parser using generator and other translator-oriented tools Happy. The TXL CFG: for Java, C#, C++. http://www.antlr.org/ TXLProgram → Exp • Many more compiler tools for Java here: Exp → AddExp http://catalog.compilertools.net/ java.html AddExp → MulExp | AddExp + MulExp • And a general catalogue of compiler tools: http://catalog.compilertools.net/ | AddExp - MulExp Note: Left-recursive! (To impart associativity.) LR parsers have no problems with left- or right- recursion (except right recursion uses more stack). COMP3012/G53CMP: Lecture 4 – p.9/32 COMP3012/G53CMP: Lecture 4 – p.10/32 Happy Parser for TXL (2) Happy Parser for TXL (3) Haskell datatype for tokens: The TXL CFG continued: data Token = T_Int Int MulExp → PrimExp | T_Id Id | MulExp * PrimExp | T_Plus | T_Minus | MulExp / PrimExp | T_Times PrimExp IntegerLiteral → | T_Divide | Identifier | T_LeftPar | ( Exp ) | T_RightPar | let Identifier = Exp in Exp | T_Equal | T_Let | T_In COMP3012/G53CMP: Lecture 4 – p.11/32 COMP3012/G53CMP: Lecture 4 – p.12/32 Happy Parser for TXL (4) Happy Parser for TXL (5) A simple Happy parser specification: Haskell datatypes for AST: { Module Header } data BinOp = Plus | Minus | Times | Divide %name ParserFunctionName %tokentype { TokenTypeName } data Exp = LitInt Int | Var Id %token | BinOpApp BinOp Exp Exp Specification of Terminal Symbols | Let Id Exp Exp %% Grammar productions with semantic actions { Further Haskell Code } COMP3012/G53CMP: Lecture 4 – p.13/32 COMP3012/G53CMP: Lecture 4 – p.14/32 Happy Parser for TXL (6) Happy Parser for TXL (7) The terminal symbol specification specifies • The code fragment between curly braces is a terminals to be used in productions and relates Haskell pattern that is matched against the them to Haskell constructors for the tokens: actual tokens returned by the parsing function. %token • If this pattern contains the special variable int { T_Int $$ } $$, then the corresponding part of the ident { T_Id $$ } matched token becomes the semantic value. ’+’ { T_Plus } Examples: T_Int $$, T_Id $$ ’-’ T_Minus { } • Otherwise the entire token becomes the ... semantic value. Examples: T_Plus, T_In ’=’ { T_Equal } • The semantic values of different terminal let { T_Let } symbols may thus have different types. in { T_In } COMP3012/G53CMP: Lecture 4 – p.15/32 COMP3012/G53CMP: Lecture 4 – p.16/32 Happy Parser for TXL (5) Happy Parser for TXL (6) The grammar productions are written in BNF, It is also possible to add type annotations: with an additional semantic action defining the semantic value for each production: add_exp :: { Exp } add_exp add_exp : mul_exp {$1} : mul_exp {$1} | add_exp ’+’ mul_exp {BinOpApp Plus $1 $3} | add_exp ’+’ mul_exp {BinOpApp Plus $1 $3} | add_exp ’-’ mul_exp {BinOpApp Minus $1 $3} | add_exp ’-’ mul_exp {BinOpApp Minus $1 $3} mul_exp Most useful when semantic values are of : prim_exp {$1} different types. | mul_exp ’*’ prim_exp {BinOpApp Times $1 $3} See HappyTXL.y for the complete example. | mul_exp ’/’ prim_exp {BinOpApp Divide $1 $3} COMP3012/G53CMP: Lecture 4 – p.17/32 COMP3012/G53CMP: Lecture 4 – p.18/32 Shift/Red. and Red./Red. Conflicts (1) Shift/Red. and Red./Red. Conflicts (2) Context-free grammars are often initially ambiguous. Consider the grammar fragment: In LR-parsing, ambiguous grammars lead to Cmd → ... shift/reduce and reduce/reduce conflicts: | if Exp then Cmd • shift/reduce: some states have mixed | if Exp then Cmd else Cmd complete and incomplete items: According to this grammar, a program fragment A → a · if e1 then if e2 then c1 else c2 A → a · b can be parsed in two ways, with very different meanings (the “dangling else” problem): Should parser shift or reduce? if e1 then (if e2 then c1) else c2 if e1 then (if e2 then c1 else c2) COMP3012/G53CMP: Lecture 4 – p.19/32 COMP3012/G53CMP: Lecture 4 – p.20/32 Shift/Red. and Red./Red. Conflicts (3) Shift/Red. and Red./Red. Conflicts (4) • reduce/reduce: some states have more than • Shift/reduce conflicts often resolved by opting one complete item: for shifting: - Typically the default option (e.g. Yacc, A → a· Bison, Happy) B → a· - Usually gives the desired result; e.g., resolves the dangling else problem in a natural way. Reduce, but by which production? • Reduce/reduce conflicts are worse as no reason to pick one production over another: grammar has to be manually disambiguated. COMP3012/G53CMP: Lecture 4 – p.21/32 COMP3012/G53CMP: Lecture 4 – p.22/32 Precedence and Associativity A TXL Interpreter (1) Happy (like e.g. Yacc and Bison) allows operator The semantic actions do not have to construct an precedence and associativity to be explicitly AST. An alternative is to interpret the code being specified to disambiguate a grammar: parsed. Basic idea: %left ’+’ ’-’ exp :: { Int } %left ’*’ ’/’ exp exp : exp ’+’ exp { BinOpApp Plus $1 $3 } : exp ’+’ exp {$1 + $3} | exp ’-’ exp { BinOpApp Minus $1 $3 } | exp ’-’ exp {$1 - $3} | exp ’*’ exp { BinOpApp Times $1 $3 } ... | exp ’/’ exp { BinOpApp Divide $1 $3 } But TXL has a let-construct ... ... What about TXL VARIABLES??? E.g.: See HappyTXL2.y for further details. let x = 3 in x + x, semantic value of x? COMP3012/G53CMP: Lecture 4 – p.23/32 COMP3012/G53CMP: Lecture 4 – p.24/32 A TXL Interpreter (2) A TXL Interpreter (3) One way: • The semantic action for a variable looks up • Each semantic action returns a function of type the variable value in the environment: Env -> Int | ident { \env -> env $1 } where (for example) • The semantic action for let extends the Type Env = Id -> Int argument environment and evaluates the body in the extended environment: • The semantic action for evaluating a | let ident ’=’ exp in exp composite expression passes on the {\env -> let v = $4 env environment. E.g. semantic action for +: in $6 (\i -> if i == $2 | exp ’+’ exp then v { \env -> $1 env + $3 env } else env i)} COMP3012/G53CMP: Lecture 4 – p.25/32 COMP3012/G53CMP: Lecture 4 – p.26/32 A TXL Interpreter (4) A TXL Interpreter (5) Another way: Attribute Grammars • The semantic action for literal just returns the value of the literal, ignoring the environment: • Each non-terminal associated with a set of | int { \_ -> $1 } semantic attributes (values). • • A program gets evaluated by applying the overall Set of equations for each production defines result function to the empty environment: the attributes. (\_ -> error "undefined variable") • Attributes can be: - Inherited: defined in terms of attributes of See HappyTXLInterpreter.y for further parents or siblings in derivation tree. details. - Synthesized: defined in terms of attributes of children in derivation tree. COMP3012/G53CMP: Lecture 4 – p.27/32 COMP3012/G53CMP: Lecture 4 – p.28/32 A TXL Interpreter (6) A TXL Interpreter (7) • Semantic equation for literals: • We need two attributes: | int { $$ = $1 } %attribute value { Int } %attribute env { Env } • Semantic equation for identifiers: The first one is the default attribute: | ident { $$ = $$.env $1 } assumed unless other attribute name • Semantic equations for let: explicitly stated, and gives overall result.