Abstract Syntax and Its Representation

Abstract Syntax and its Representation, Parsing, Unparsing

The first thing we must understand is BNF, Backus Naur Form, which is a metalanguage – a language that describes other languages. Here is an example we will develop throughout this paper.

1) := = ( + )

2) := =

3) := = Symbol | Number

This BNF grammar consists of two “productions” or rules that describe how to generate strings that are “valid”. The set of all possible valid strings is the language defined by the grammar. In BNF, expressions of the form < symbol > are called “non-terminal” symbols. Other symbols in the grammar are called “terminal” symbols. Starting with the symbol we can replace any nonterminal symbol with the expression on the right of “:= =”. Replacements continue until all nonterminal symbols have been removed. Note: Production 2 is equivalent to := = . Here’s an example. Each line represents the application of a single production. The production number is listed to the right.

(+ ) 1 (+ ( + )) 1 (+ ( + )) 2 (+ ( + )) 2 (+ ( + )) 2 (+ ( + x )) 3 (+ 9 ( + x )) 3 (+ 9 ( + 7 x )) 3

Since the last line contains only terminal symbols, the expression “(+ 9 (+ 7 x))” is a valid sentence in the language defined by the grammar. The lines above represent a “derivation” of the final string. In general, there are many possible derivations for a given string. Can you find derivations for each of the following strings?

(+ 3 2) (+ (+ 2 3)(+ a b)) (+ 9 (+ x y))

We can think of the BNF grammar as defining a concrete syntax or the external representation. The external representation is typically converted to an internal representation. The internal representation is one that is more convenient for software manipulation. Notice that is an recursive production since it describes an Aexpression in terms of itself. Scheme is particularly effective in representing “inductive” structures by using the define-datatype form. Here is the Scheme definition of the inductive structure for storing Aexpressions. (define symnum? (lambda (exp) (or (symbol? exp)(number? exp))))

(define-datatype aexpression aexpression? (add-exp (exp1 aexpression?) (exp2 aexpression?)) (t-exp (id symnum?)))

Since Texpressions are either Scheme symbols or numbers, we first define a predicate called “symnum?”. This predicate returns true if the passed parameter is a symbol or a number. Next we define the “aexpression” structure using define-datatype. There are two variants: 1) add-exp - this variant models production 1, and 2) t-exp – this variant models production 3. When defining the Aexpression structure we must define a variant for each non-terminal symbol in the list of productions. By defining the aexpression datatype we have created an internal representation for the “external” strings in the language. Lets look at some examples.

Example 1 Consider the Aexpression string (+ 3 4). Lets derive that string using the productions above.

(+ ) 1 (+ ) 2 (+ ) 2 (+ 3 ) 3 (+ 3 4 ) 3

We can derive an internal representation of the same string by mimicking the steps above with the Scheme structure.

(aexpression) (add-exp aexpression aepression) (add-exp (t-exp id) aexpression) (add-exp (t-exp id) (t-exp id)) (add-exp (t-exp 3) (t-exp id)) (add-exp (t-exp 3) (t-exp 4))

The process of converting the external representation of a string like (+ 3 4 ) to an internal representation like (add-exp (t-exp 3) (t-exp 4)) is called parsing. The internal representation is easier to process with software, and can dispense with some of the syntactic sugar that may appear in the external representation.

You can test the aexpression structure with the following code.

(define a (t-exp 1)) (define b (t-exp 2)) (define c (t-exp 3)) (define d (t-exp 4)) (define e (add-exp a b)) (define f (add-exp c d)) (define g (add-exp e f)) (aexpression? g)

The variable g references the expression below

1) (add-exp (add-exp (t-exp 1) (t-exp 2)) (add-exp (t-exp 3) (t-exp 4)))

(Notice that g refers to the internal representation of the external string below.

2) ( + (+ 1 2) (+ 3 4) ).

A parsing program could take 2) as input and produce 1). An unparsing program could take 1) as input and produce 2). We look at the unparser first, since it should be easier (with cases). Here is the code,

(define unparse-aexpression (lambda (ex) (cases aexpression ex (add-exp (exp1 exp2) (list '+ (unparse-aexpression exp1) (unparse-aexpression exp2))) (t-exp (id) id))))

The code above uses the cases form to process each variant record. When an expression like (add-exp 3 4) is encountered, the output is a list which starts with a + sign, followed by recursive calls for exp1 and exp2. When an expression like (t-exp 3) is encountered, the function returns 3.

The parser function is listed below. The form it takes is dependent on the external syntax. This is more properly the subject of chapter 3. Here is the code for the parser.

(define parse-aexpression (lambda (datum) (cond ((symnum? datum) (t-exp datum)) ((and (pair? datum)(eqv? (car datum) '+)) (add-exp (parse-aexpression (cadr datum)) (parse-aexpression (caddr datum)))) (else (eopl:error 'parse-expression "Invalid concrete syntax ~s" datum)))))

We can test the parsing and unparsing functions with the code below.

(define a (t-exp 1)) (define b (t-exp 2)) (define c (t-exp 3)) (define d (t-exp 4)) (define e (add-exp a b)) (define f (add-exp c d)) (define g (add-exp e f)) (unparse-aexpression g) (define h (quote (+ (+ 1 2)(+ 3 4)))) (unparse-aexpression (parse-aexpression h)) (parse-aexpression (unparse-aexpression g))