Parser Generators

Aurochs and ANTLR and Yaccs,,y Oh My

Tuesday, December 1, 2009 Reading: Appel 3.3

CS235 Languages and Automata

Department of Computer Science Wellesley College

Compiler Structure Abstract Source Syntax Program Lexer Tokens Tree (AST) Type (a.k.a. Scanner, Parser (character Tokenizer) Checker stream) Intermediate Representation Front End (CS235) Semantic Analysis Middle Stages Intermediate Representation Global Analysis (CS251/ Information CS301) Optimizer (Symbol and Attribute Tables) Intermediate Representation

Back End Code (used by all phases Generator of the compiler) (CS301) Machine code or byte code

Parser Generators 34-2

1 Motivation

We now know how to construct LL(k) and LR(k) parsers by hand. But constructing parser tables from grammars and implementing parsers from parser tables is tedious and error prone. Isn’t there a better way? Idea: Hey, aren’t computers good at automating these sorts of things?

Parser Generators 34-3

Parser Generators

A parser generator is a program whose inputs are: 1. a grammar that specifies a tree structure for linear sequences of tokens (typically created by a separate lexer program). 2. a specification of semantic actions to perform when building the parse tree. Its output is an automatically constructed program that parses a token sequence into a tree, performing the specified semantic actions along the way In the remainder of this lecture, we will explore and compare a two parser generators: Yacc and Aurochs.

Parser Generators 34-4

2 ml-yacc: An LALR(1) Parser Generator Slip.yacc Slip.yacc.desc .yacc file LR parsing table ml-yacc (token datatype, info for debugging parsing rules, shift-reduce and precedence & reduce-reduce conflicts associativity info) Slip.yacc.sig Slip.yacc.sml token datatype Slip.lex parsing program .lex file ml-lex (token rules & token-handling code) Slip.lex.sml

ant token stream ant program scanner program parser program abstract (as character (SML) (SML) syntax tree stream) A Slip program Slip tokens Slip AST

Parser Generators 34-5

Format of a .yacc File

Header section with SML code %% Specification of terminals & variables and various declarations ((gpincluding precedence and associativity) %% Grammar productions with semantic actions

Parser Generators 34-6

3 IntexpUnambiguousAST.yacc (Part 1)

(* no SML code in this case *) %% (* separates initial section from middle section of .yacc file *) Name used to prefix various names created by ml-yacc %name Intexp %pos int Declares the type of positions for terminals. Each terminal has a left and right position. Here, position is just an int (a character position in a string or file) %term INT of int Specifies the terminals of the language. ml-yacc automatically | ADD | SUB | MUL | DIV | EXPT constructs a Tokens module based on this specification. Tokens specified without "of" will have a constructor of | LPAREN (* "(" *) two args: (1) its left position and (2) its right position. | RPAREN (* ")" *) Tokens specified with "of" will have a constructor of three | EOF (* End of file*) args: (1) the component datum (whose type follows "of"); (2) its left position; and (3) its the right position. %nonterm Start of AST.exp Specifies the non-terminals of the language and the kind of value ||p Exp of AST.ex p that the parser will generate for them (an AST in this case). | Term of AST.exp The "start" non-terminal should not appear on the right-hand | Factor of AST.exp sides of any productions. | Unit of AST.exp %start Start Start symbol of the grammar

(* Middle section continued on next slide *)

Parser Generators 34-7

IntexpUnambiguousAST.yacc (Part 2)

(* Middle section continued from previous slide *)

Lists the keywords of the language. The error handler of the ml-yacc generated %keyword parser treats keywords specially. In this example, there aren’t any keywords.

%eop EOF Indicates tokens that end the input

%noshift EOF Tells the parser never to shift the specfied tokens

%nodefault Suppresses generation of default reduction

%verbose Generates a description of the LALR parser table in filename.yacc.desc. This is helpful for debugging shift/reduce and reduce/reduce conflicts %value INT(0) Specifies default values for value-bearing terminals. Terminals with default values may be created by an ml-yacc-generated parser as part of error-correction.

Note; in this middle section, you can also specify associativity and precedence of operators. See IntexpPrecedence.yacc (later slide) for an example.

(* Grammar productions and semantic actions on next slide *)

Parser Generators 34-8

4 AST module for integer expressions structure AST = struct

datatype exp = Int of int | BinApp of binop * exp * exp

and binop = Add | Sub | Mul | Div | Expt

(* Various AST functions omitted from this slide *) end

Parser Generators 34-9

IntexpUnambiguousAST.yacc (Part 3)

%% (* separates middle section from final section *) (* Grammar productions and associated semantic actions (in parens) go here *)

Start: Exp (Exp) Within parens, nonterminals stand for the value generated by the parser for the nonterminal, as specified in the %nonterm declaration (an AST expression in this case). Exp : Term (Term) (* The following rules specify left associativity *) | Exp ADD Term (AST.BinApp(AST.Add,Exp,Term)) | Exp SUB Term (AST.BinApp(AST.Sub,Exp,Term))

Term : Factor (Factor) (* The following rules specify left associativity *) | Term MUL Factor (AST.BinApp(AST.Mul,Term,Factor)) | Term DIV Factor (AST.BinApp(AST.Div,Term,Factor))

Factor : Unit (Unit) (* The following rule specifies right associativity *) | Unit EXPT Factor (AST.BinApp(AST.Expt,Unit,Factor))

Unit : INT (AST.Int(INT)) Within parens, terminals stand for the component datum | LPAREN Exp RPAREN (Exp) specified after “of” in the %term declaration. E.g., the datum associated with INT is an integer.

Parser Generators 34-10

5 Interfacing with the Lexer

(*contents of Intexp.lex *) open Tokens type pos = int type lexresult= (svalue,pos) token fun eof () = Tokens.EOF(0 , 0) fun integer(str,lexPos) = case Int.fromString(str) of NONE => raise Fail("Shouldn't happen: sequence of digits not recognized as integer -- " ^ str) | SOME(n) => INT(n,lexPos,lexPos)

(* For scanner initialization (not needed here) *) fun init() = () %% %header (functor IntexpLexFun(structure Tokens: Intexp_TOKENS)); alhlpha=[a-zA-Z]; digit=[0-9]; whitespace=[\ \t\n]; symbol=[+*^()\[]; any= [^]; %% {digit}+ => (integer(yytext,yypos)); "+" => (ADD(yypos,yypos)); (* more lexer rules go here *) Parser Generators 34-11

Parser interface code, Part 1 structure Intexp = struct (* Module for interfacing with parser and lexer *)

(* Most of the following is "boilerplate" code that needs to be slightly edited on a per parser basis. *)

structure IntexpLrVals = IntexpLrValsFun(structure Token = LrParser.Token) structure IntexpLex = IntexpLexFun(structure Tokens = IntexpLrVals.Tokens); structure IntexpParser = Join(structure LrParser = LrParser structure ParserData = IntexpLrVals.ParserData structure Lex = IntexpLex)

val invoke = fn lexstream => (* The invoke function invokes the parser given a lexer *) let val print_error = fn (str,pos,_) => TextIO.output(TextIO.stdOut, "***Intexp Parser Error at character position " ^ (Int.toString pos) ^ "***\n" ^ str^ "\n") in IntexpParser.parse(0,lexstream,print_error,()) end

fun newLexer fcn = (* newLexer creates a lexer from a given input-reading function *) let val lexer = IntexpParser.makeLexer fcn val _ = IntexpLex.UserDeclarations.init() in lexer end

(* continued on next slide *) ml-yacc 35-12

6 Parser interface code, Part 2

(* continued from previous slide *) fun stringToLexer str = (* creates a lexer from a string *) let val done = ref false in newLexer (fn n => if (!done) then "“ else (done := true; str)) end fun fileToLexer filename = (* creates a lexer from a file *) let val inStream = TextIO.openIn(filename) in newLexer (fn n => TextIO.inputAll(inStream)) end fun lexerToParser (lexer) = (* creates a parser from a lexer *) let val dummyEOF = IntexpLrVals.Tokens.EOF(0,0) val (result,lexer) = invoke lexer val (nextToken,lexer) = IntexpParser.Stream.get lexer in if IntexpParser.sameToken(nextToken,dummyEOF) then result else (TextIO.output(TextIO.stdOut, "*** INTEXP PARSER WARNING -- unconsumed input ***\n"); result) end val parseString = lexerToParser o stringToLexer (* parses a string *) val parseFile = lexerToParser o fileToLexer (* parses a file *) end (* struct *)

Parser Generators 34-13

Putting it all together: the load file

(* This is the contents of load-intexp-unambiguous-ast.sml *) CM.make("$/basis.cm"); (* loads SML basis library *) CM.make("$/ml-yacc-lib.cm"); (* loads SML YACC library *) use "AST.sml"; ((ypgp* datatype for integer expression abstract sy ntax trees *) use "IntexpUnambiguousAST.yacc.sig"; (* defines Intexp_TOKENS and other datatypes *) use "IntexpUnambiguousAST.yacc.sml"; (* defines shift-reduce parser *) use "Intexp.lex.sml"; (* load lexer *after* parser since it uses tokens defined by parser *) use "Intexp.sml"; (* load top-level parser interface *) Control.Print .printLength := 1000; (* set printing parameters so that *) Control.Print.printDepth := 1000; (* we’ll see all details of parse trees *) Control.Print.stringDepth := 1000; (* and strings *) open Intexp; (* open the parsing module so that we can use parseString and parseFile without qualification. *)

Parser Generators 34-14

7 Taking it for a spin

[fturbak@cardinal intexp-parsers] ml-lex Intexp.lex

Number of states = 14 Number of distinct rows = 3 Linux Approx. memory size of trans. tabl shell e = 387 bytes [fturbak@cardinal intexp-parsers] ml-yacc IntexpUnambiguousAST.yacc [fturbak@cardinal intexp-parsers]

- use "load-intexp-unambiguous-ast.sml"; (* … lots of printed output omitted here … *)

- parseString "1+2*3^4^5*6+7"; val it = BinApp SML (Add, intepreter BinApp (Add,Int 1, BinApp (Mul, BinApp (Mul,Int 2,BinApp (Expt,Int 3,BinApp (Expt,Int 4,Int 5))), Int 6)),Int 7) : IntexpParser.result Parser Generators 34-15

IntexpUnambiguousAST.yacc.desc state 0: … lots of states omitted …

Start : . Exp state 18:

INT shift 6 Unit : LPAREN Exp RPAREN . (reduce by rule 10) LPAREN shift 5 ADD reduce by rule 10 Start goto 19 SUB reduce by rule 10 Exp goto 4 MUL reduce by rule 10 Term goto 3 DIV reduce by rule 10 Factor goto 2 EXPT reduce by rule 10 Unit goto 1 RPAREN reduce by rule 10 EOF reduce by rule 10 . error

. error state 1:

Factor : Unit . (reduce by rule 7) state 19: Factor : Unit . EXPT Factor

ADD reduce by rule 7 EOF accept SUB reduce by rule 7 MUL reduce by rule 7 DIV reduce by rule 7 . error EXPT shift 7 RPAREN reduce by rule 7 72 of 104 action table entries left after compaction EOF reduce by rule 7 21 goto table entries

. error

Parser Generators 34-16

8 IntexpUnambiguousCalc.yacc We can calculate the result of an integer expression rather than generating an AST.

(* An integer exponentiation function *) %% fun expt (base,0) = 1 | expt(base,power) = Start: Exp (Exp) base*(expt(base,power-1)) Exp : Term (Term) (* The following rules specify left associativity *) %% | Exp ADD Term (Exp + Term) | Exp SUB Term (Exp - Term) (* Only changed parts shown *) Term : Factor (Factor) %nonterm (* The following rules specify left associativity *) Start of int | Term MUL Factor (Term * Factor) | Exp of int | Term DIV Factor (Term div Factor) | Term of int ((g* div is integer division *) | Factor of int | Unit of int Factor : Unit (Unit) (* The following rule specifies right associativity *) ) | Unit EXPT Factor (expt(Unit,Factor))

Unit : INT (INT) | LPAREN Exp RPAREN (Exp)

Parser Generators 34-17

Testing IntexpUnambiguousCalc

[fturbak@cardinal intexp-parsers] ml-yacc IntexpUnambiguousCalc.yacc Linux [fturbak@cardinal intexp-parsers] shell

- use "load-intexp-unambiguous-calc.sml"; (* … lots of printed output omitted here … *) SML - parseString "1+2*3"; intepreter val it = 7 : IntexpParser. result

- parseString "1-2-3"; val it = ~4 : IntexpParser.result

Parser Generators 34-18

9 IntexpAmbiguousAST.yacc We can use an ambiguous grammar. (In this case, it leads to shift/reduce conflicts.)

%% (* Only changed parts shown *)

%nonterm Start of AST.exp | Exp of AST.exp )

Start: Exp (Exp)

Parser Generators 34-19

IntexpAmbiguousAST.yacc.desc

25 shift/reduce conflicts error: state 8: shift/reduce conflict (shift EXPT, reduce by rule 6) error: state 8: shift/reduce conflict (shift DIV, reduce by rule 6) error: state 8: shift/reduce conflict (shift MUL, reduce by rule 6) error: state 8: shift/reduce conflict (shift SUB, reduce by rule 6) error: state 8: shift/reduce conflict (shift ADD, reduce by rule 6) error: state 9: shift/reduce conflict (shift EXPT, reduce by rule 5) error: state 9: shift/reduce conflict (shift DIV, reduce by rule 5) error: state 9: shift/reduce conflict (shift MUL, reduce by rule 5) error: state 9: shift/reduce conflict (shift SUB, reduce by rule 5) error: state 9: shift/reduce conflict (shift ADD, reduce by rule 5) error: state 10: shift/reduce conflict (shift EXPT, reduce by rule 4) error: state 10: shift/reduce conflict (shift DIV, reduce by rule 4) error: state 10: shift/reduce conflict (shift MUL, reduce by rule 4) error: state 10: shift/reduce conflict (shift SUB, reduce by rule 4) error: state 10: shift/reduce conflict (shift ADD, reduce by rule 4) error: state 11: shift/reduce conflict (shift EXPT, reduce by rule 3) error: state 11: shift/reduce conflict (shift DIV, reduce by rule 3) error: state 11: shift/reduce conflict (shift MUL, reduce by rule 3) error: state 11: shift/reduce conflict (shift SUB, reduce by rule 3) error: state 11: shift/reduce conflict (shift ADD, reduce by rule 3) error: state 12: shift/reduce conflict (shift EXPT, reduce by rule 2) error: state 12: shift/reduce conflict (shift DIV, reduce by rule 2) error: state 12: shift/reduce conflict (shift MUL, reduce by rule 2) error: state 12: shift/reduce conflict (shift SUB, reduce by rule 2) error: state 12: shift/reduce conflict (shift ADD, reduce by rule 2)

… state descriptions omitted …

Parser Generators 34-20

10 IntexpPrecedenceAST.yacc We can resolve conflicts with precedence/associativity declarations (can also handle dangling else problem this way).

%% (* Only changed parts shown *)

%nonterm Start of AST.exp | Exp of AST.exp )

(* Specify associativity and precedence from low to high *) %left ADD SUB %left MUL DIV %right EXPT

Start: Exp (Exp)

Stepping Back

Whew! That’s a lot of work! Is it all worth it? There’s a tremendous amount of initial setuppg and glue code to get Lex and Yacc working together. But once that’s done, we can focus on specifying and debugging high-level declarations of the token structure (in Lex) and grammar (in Yacc) rather than on SML programming. In Yacc, generally spend most of the time debugging shift/reduce and reduce/reduce conflicts – must be very familiar with the LALR(1) table descriptions in the .yacc.desc file.

Parser Generators 34-22

11 Other Parser Generators

 There are a plethora of other LALR parser generators for many languages, including:  GNU Bison (Yacc for C, C++)  jb (Bison for Java )  CUP (Yacc for Java)  There are also LL(k) parser generators, particularly ANTLR, which generates parsers and other tree-manipulation programs for Java, C, C++, C#, Python, Ruby, and JavaScript. There is also an sml-antlr.  There are a relatively new (starting in 2002) collection of so- called packrat parsers for Parsing Expression Grammars (PEGs) for a wide range of target languages. We will study one of these: Aurochs.

Parser Generators 34-23

Parsing Expression Grammars (PEGs) Key problem in building top-down (recursive-descent) parsers: deciding which production to use on tokens. Three strategies:  Predictive parser: determine which production to use based on k tokens of lookahead. Often mangle grammars to make this work.  Backtracking parser: when have several choices, try them in order. If one fails try the next. Very general and high-level, but can take exponential time to parse input.  PEG Parser: have a prioritized choice operator in which unconditionally use first successful match.  Can directly express common disambiguation rules like longest-match, followed-by, and not-followed-by.  Can express lexing rules and parsing rules in same grammar! This eliminates glue code between lexer and parser.  Can implement PEGs via a packrat parser that memoizes parsing results and guarantees parsing in linear time (but possibly exponential space)

Parser Generators 34-24

12 Aurochs: a PEG Parser

We will explore PEG parsing using the Aurochs parser generator, developed by Berke Durak and available from http://aurochs.fr. The input to Aurochs is (1) a file to be parsed and (2) a .peg file sppfygEecifying PEG declarations f or lexing and ppgarsing rules. The output of Aurochs is an XML parse tree representation. (It is also possible to generate parse trees in particular languages, but for simplicity we’ll stick with the XML trees.)

Parser Generators 34-25

Aurochs Example [lynux@localhost slipmm-parsers]$ aurochs -parse simple.slip SlipmmParens.peg Aurochs 1.0.93 - Parsing file simple.slip using Nog interpreter - Grammar loaded from file SlipmmParens.peg - RESULT Contents of the file simple.slip begin x := (3+4); print ((x-1)*(x+2)); end - No targets Parser Generators 34-26

13 Aurochs Grammar Expressions

Expression Meaning 'c' The single ASCII character 'c' "hello" The string “hello” BOF, EOF Beginning and end of file tokens sigma Any character epsilon The empty string [a-zA-Z0-9] grep-like character sets [^\n\t ] grep-like complements of character sets e1 e2 Concatenation: parse e1, then e2 e* Repetition: parses as many occurrences of e as possible, including zero. e+ Nonzero repetition: parses at least one occurrences of e. e? Option: parses zero or one occurrence of e. &e Conjunction: Parses e, but without consuming it. ~e Negation: Attempts to parse e. If parsing e fails, then ~e succeeds without consuming any input. If parsing e succeeds, then ~e fails (also without consuming input). e1 | e2 Ordered disjunction (prioritized choice): Attempt to parse e1 at the current position. If this succeeds, it will never try to parse e2 at the current position. If it fails, it will never try to parse e1 again at that position, and will try to parse e2. If e2 fails, the whole disjunction fails. (e) Parentheses used for explicit grouping name Parse according to e from production name ::= e Parser Generators 34-27

Some Simple Expressions

Expression Meaning ‘-’? [1-9][0-9]* Any nonempty sequence of digits not beginning with 0, preceded by an optional minus sign &[aeiou] [a-z]+ Any nonempty sequence of lowercase letters beginning with a vowel. ~0 [0-9]+ Any nonempty sequence of digits not beginning with 0 ~(“begin”|”end”|”let”) Alphanumeric names beginning with lowercase letters, &[a-z] [a-zA-Z0-9]+ excluding the keywords begin, end, and let. exp ::= (exp “+” exp) Assume id matches identifiers and num matches numbers. | id “=“ exp | id | num Then this can parse (x=3 + x) and (x=y=3 + (1 + z=y)) exp ::= (exp “+” exp) Due to prioritized choice, this will never parse x=3. | id | id “=“ exp | num if exp then stm else stm Solves dangling else problem by specifying that else goes | if exp then stm with the closest if.

Parser Generators 34-28

14 Numerical Expressions in Aurochs

(* Tokens *) float ::= '-'? decimal+ ('.' decimal+)? ('e' ('-'|'+')? decimal+)?; integer ::= '-'? decimal+; decimal ::= [0-9]; sp ::= (' '|'\n'|'\r'|'\t')*; (* any number of whitespaces *)

ADD ::= sp ''+ sp; SUB ::= sp ''- sp; MUL ::= sp ' *' sp; DIV ::= sp '/ ' sp; LPAREN ::= sp '(' sp; RPAREN ::= sp ')' sp;

(* Expressions *) start ::= exp sp EOF;

exp ::= term ADD exp (* right associative *) | term SUB exp (* right associative *) | term;

term ::= factor MUL term (* right associative *) | factor DIV term (* right associative *) | SUB factor (* unary negation *) | factor;

factor ::= number | LPAREN exp RPAREN;

number ::= float|integer;

Parser Generators 34-29

XML Parse Trees for Numexps in Aurochs

(* Tokens the same as before, so omitted *)

(* Expressions *) start ::= exp sp EOF;

exp ::= term ADD exp (* right associative *) | _{term SUB exp} (* right associative *) | term;

term ::= factor MUL term (* right associative *) |

factor DIV term

(* right associative *) | SUB factor | factor;

factor ::= number | LPAREN exp RPAREN;

number ::= value:(float|integer) ;

Parser Generators 34-30

15 Testing Numexp Parser

[lynux@localhost calculator]$ aurochs -quiet -parse exp.txt numexp.peg _{_{Contents of the file exp.txt} (1-2-3)*-(-4+5.6+7.8e-9)} Parser Generators 34-31

Transforming Left Recursion

(* Expressions *)

start ::= exp sp EOF;

expp(p()p) ::= term (op:(ADD|SUB) exp)* ;

term ::= SUB factor | factor (op:(MUL|DIV) term)* ;

factor ::= number | LPAREN exp RPAREN;

number ::= value:(float|integer) ;

Parser Generators 34-32

16 Left Recursion Transformation Exampl

[lynux@localhost calculator]$ aurochs -quiet -parse sub.txt numexp-leftrec.peg Contents of the file sub.txt (1-2-3-4)

Parser Generators 34-33

Moving Spaces from Tokens to Expressions

(* Tokens *) float ::= '-'? decimal+ ('.' decimal+)? ('e' ('-'|'+')? decimal+)?; integer ::= '-'? decimal+; decimal ::= [0-9]; sp ::= (' '|'\n'|'\r'|'\t')*; (* any number of whitespaces *)

ADD ::= ''+ ; SUB ::= ''- ; MUL ::= '*'; DIV ::= '//;'; LPAREN ::= '('; RPAREN ::= ')';

(* Expressions *) start ::= sp exp sp EOF;

exp ::= sp term sp ADD sp exp sp (* right associative *) | _{sp term sp SUB sp exp sp} (* right associative *) | term;

term ::= sp factor sp MUL sp term sp (* right associative *) |

sp factor sp DIV sp term sp

(* right associative *) | sp SUB sp factor sp | sp factor sp;

factor ::= sp number sp | sp LPAREN sp exp sp RPAREN sp;

number ::= sp value:(float|integer) sp ;

Parser Generators 34-34

17 Modifiers: Surrounding, Outfixing, and Friends

 The following modifiers take an expression x as parameter and transform each production a ::= b as follows:  appending x do ... done transforms them into a ::= b x  prepending x do ... done transforms them into a ::= x b  surrounding x do ... done transforms them into a ::= x b x

 The following modifiers transform each concatenation a b c:  outfixing x do ... done transforms them into x a x b x c x  suffixing x do ... done transforms them into a x b x c x  prefixing x do ... done transforms them into x a x b x c Note that these modifiers also act within implicit concatenations produced by the operators '*' and '+'.

These rules found at http://aurochs.fr Parser Generators 34-35

A Shorthand for all those Expression Spaces

start ::= sp exp sp EOF;

outfixing sp do surrounding sp do exp ::= term ADD exp (* right associative *) | _{term SUB exp} (* right associative *) | term;

term ::= factor MUL term (* right associative *) |

factor DIV term

(* right associative *) | SUB factor | factor;

factor ::= number ||p LPAREN exp RPAREN;

number ::= value:(float|integer) ; done; done;

Parser Generators 34-36

18 Aurochs Productions for Slip-- Tokens

whitespace ::= [ \n\r\t] | BOF; (* BOF = beginning of file token *)

(* sp is at least one ignorable unit (space or comment) *) sp ::= whitespace*;

PRINT ::= "print"; BEGIN ::= "begin"; END ::= "end"; kdkeyword ::= eyword> val:lue: (PRINT | BEGIN | END) ;eyword>;

ADD ::= '+'; SUB ::= '-'; MUL ::= '*'; DIV ::= '/'; op ::= value: (ADD | SUB | MUL | DIV) ;

LPAREN ::= "("; RPAREN ::= ")"; SEMI ::= ";"; GETS ::= ":="; punct ::= value: (LPAREN | RPAREN | SEMI | GETS) ;

alpha ::= [a-zA-Z]; alphaNumUnd ::= [a-zA-Z0-9_]; digit ::= [0-9]; reserved ::= "print" | "begin" | "end"; ident ::= value: (~(reserved sp) alpha alphaNumUnd*) ; integer ::= value: digit+ ;

token ::= sp (keyword|punct|op|ident|integer) sp; tokenList ::= token* ;

start ::= tokenList EOF;

Parser Generators 34-37

Slip-- Tokenizing example

[lynux@localhost slipmm-parsers]$ aurochs -quiet -parse simple.slip SlipmmTokens.peg Contents of the file simple.slip begin x := (3+4); print ((x-1)*(x+2)); end Parser Generators 34-38

19 Slip-- Tokens with Positions

[lynux@localhost slipmm-parsers]$ aurochs -quiet -parse simple.slip SlipmmTokensPositions.peg (* Only Change to .peg file. *) token ::= sp start:position (keyword|punct|op|ident|integer) stop:position sp ; Cont ent s of the file simple.slip begin x := (3+4); print ((x-1)*(x+2)); end … many more tokens …. Parser Generators 34-39

Line Comments

How to add line-terminated comments introduced by #?

begin sum := 5+3; # Set sum to 8 prit(int(sum ); # Displ ay sum end

(* Only changes in .peg file *)

whitespace ::= [ \n\r\t] | BOF; (* BOF = beginning of file token *) linecomment ::= '#' [^\n]* '\n'; (* Comment goes from '#' to end of line *)

(* sp is at least one ignorable unit (space or comment) *) sp ::= (whitespace | linecomment )*;

Parser Generators 34-40

20 Nonnestable Block Comments begin sum := 5+3; # Set sum to 8 { Comment out several lines: x := sum * 2; z := x * x; } print(sum); # Display sum end

(* Only changes in .peg file *)

whitespace ::= [ \n\r\t] | BOF; (* BOF = beginning of file token *) linecomment ::= '# ' [^\n]* '\nn;'; (* Comment goes from ' #' to end of line * ) blockcomment ::= '{' ((~ '}') sigma)* '}'; (* Nonnestable block comment *)

(* sp is at least one ignorable unit (space or comment) *) sp ::= (whitespace | linecomment | blockcomment)*;

Parser Generators 34-41

Nestable Block Comments begin sum := 5+3; # Set sum to 8 { Comment out several lines: x := sum * 2; { Illustrate nested block comments: y = sum - 3:} z := x * x; } print(sum); # Display sum end

(* Only changes in .peg file *)

whitespace ::= [ \n\r\t] | BOF; (* BOF = beginning of file token *) linecomment ::= '#' [^\n]* '\n'; (* Comment goes from '#' to end of line *) blockcomment ::= '{' (blockcomment| ((~ '}') sigma))* '}'; (* Properly nesting block comment *)

(* sp is at least one ignorable unit (space or comment) *) sp ::= (whitespace | linecomment | blockcomment)*;

Parser Generators 34-42

21 Slip -- Programs

(* Assume tokens unchanged from before *)

start ::= program;

outfixing sp do appending sp do program ::= stm EOF;

stm ::= var:ident GETS exp | PRINT exp | BEGIN stmList END ;

stmList ::= (()stm SEMI)*;

exp ::= name:ident | value:integer | LPAREN exp binop:op exp RPAREN ; done; done;

Parser Generators 34-43

Aurouchs Errors

-Fatal error: exception Aurochs.Parse_error(1033)

This means that the .peg file has an error at the specified character number (1 033 in thsthis case).

Note: start position is 0-indexed; compared to 1-indexed emacs position in M-x goto-char.

-PARSE ERROR IN FILE nesting-block-comment-test.slip AT CHARACTER 12

This means that the parser encountered an error in parsing the input file at the specif ied character number.

Parser Generators 34-44