Course "Software Language Engineering"

Templates and friends

University of Koblenz-Landau Department of Computer Science Ralf Lämmel Software Languages Team

String template with one parameter import org.stringtemplate.v4.*; ... ST hello = new ST("Hello, "); hello.add("name", "World"); System.out.println(hello.render());

[http://www.antlr.org/wiki/display/ST4/Introduction, 10 December 2012]

using Antlr4.StringTemplate;

Template hello = new Template("Hello, "); hello.Add("name", "World"); Console.Out.WriteLine(hello.Render());

[http://www.antlr.org/wiki/display/ST4/Introduction, 10 December 2012]

import stringtemplate4 hello = stringtemplate4.ST("Hello, ") hello["name"] = "World" print str(hello.render())

[http://www.antlr.org/wiki/display/ST4/Introduction, 10 December 2012]

Exercised concepts A template is a declarative, Literal text parametrized, and executable Place holder for text description of a text generator. Other concepts Conditionals, loops over template “context” Expression holes for computing output ... Find a proper deﬁnition that ﬁts into the context of this lecture.

HelloWorld WebApp HelloHello World!World!

';'; ??>

HTML with embedded PHP Resulting HTML executed on the web server rendered in the web browser

101companies Ruby on Rails Web App

<%= notice %>

Name: <%= @company.name %>

Departments:

<% @company.departments.each do |department| %>

<%= link_to department.name, department_path(department) %>

<% end %>

<%= link_to 'Edit', edit_company_path(@company) %> | <%= link_to 'Back', companies_path %>

[https://github.com/101companies/101repo/blob/master/contributions/rubyonrails/companies/app/views/companies/show.html.erb]

...

Use of code delimiters (not shown here) and tag libraries (see preﬁx s). ... https://github.com/101companies/101repo/blob/master/contributions/strutsAnnotation/src/main/webapp/WEB-INF/content/company-detail.jsp

<%@ jet package="generator.website" class="Simple2HTML" imports ="java.util.Iterator org.eclipse.emf.common.util.EList datamodel.website.*;" %> <% Webpage website = (Webpage) argument; %> <%=website.getTitle()%> Similar to JSP. Works on top of EMF.

List of Articles Names

<%EList

list = website.getArticles(); for (Iterator

iterator = list.iterator(); iterator.hasNext();) { Article article = iterator.next();%> <%=article.getName()%>
Generative (MDE) approach such that <%}%> a designated class is generated from the template.

http://www.vogella.com/articles/EclipseJET/article.html

© 2012 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 12 “Model to Text” with JET import generator.website.Simple2HTML; Import generated class. import java.io.BufferedWriter; import java.io.FileWriter; import java.io.IOException; import java.util.Iterator; import datamodel.website.MyWeb; import datamodel.website.Webpage; Import EMF-based model classes. public class SimpleWebTest { private static String TARGETDIR = "./output/";

public static void main(String[] args) { EMFModelLoad loader = new EMFModelLoad(); MyWeb myWeb = loader.load(); createPages(myWeb); }

private static void createPages(MyWeb myWeb) { ... } http://www.vogella.com/articles/EclipseJET/article.html

... private static void createPages(MyWeb myWeb) { Simple2HTML htmlpage = new Simple2HTML(); FileWriter output; BufferedWriter writer; for (Iterator iterator = myWeb.getPages().iterator(); iterator .hasNext();) { Webpage page = iterator.next(); System.out.println("Creating " + page.getName()); try { output = new FileWriter(TARGETDIR + page.getName() + ".html"); writer = new BufferedWriter(output); writer.write(htmlpage.generate(page)); writer.close(); } catch (IOException e) { e.printStackTrace(); Actual template processing. } } } }

http://www.vogella.com/articles/EclipseJET/article.html

A template processor (also known as a template engine, template parser or template language) is software or a software component that is designed to combine one or more templates with a data model to produce one or more result documents.

[http://en.wikipedia.org/wiki/Template_processor, 10 December 2012]

A (web) template engine is software that is designed to process web templates and content information to produce output web documents. It runs in the context of a template system.

[http://en.wikipedia.org/wiki/Template_engine_(web), 10 December 2012]

• Genshi • Mako (template A (templating engine) • Apache Velocity language) • Microsoft • ASP.NET H ASP.NET Razor • Thymeleaf • TinyButStrong C • Haml view engine • Toupl • CakePHP • Handlebars • Mustache (template system) (template system) • Twig (template • CCAPS engine) J N • CheetahTemplate V • CodeCharge • JavaServer • Nevow • Text Template Studio Pages O Transformation • CTPP • Jinja (template • Open Power Toolkit D engine) Template • VlibTemplate • JSP Weaver P • Dylan Server W Pages K • PHPRunner • Web template E • Kid (templating S system language) • ERuby • Smarty • WebMacro F L T • FreeMarker • Lithium (PHP • Template Attribute G framework) Language M • Template Toolkit

[http://en.wikipedia.org/wiki/Category:Template_engines, 10 December 2012]

A web template system is composed of: • A template engine: the primary processing element of the system; • Content resource: any of various kinds of input data streams, such as from a relational database, XMLﬁles, LDAP directory, and other kinds of local or networked data; • Template resource: web templates speciﬁed according to a template language;

[http://en.wikipedia.org/wiki/Web_template_system, 10 December 2012]

Wikipedia does not deﬁne a general concept of template. It does deﬁne the notion of a web template:

“A web template is a tool used to separate content from presentation in web design, and for mass-production of web documents. It is a basic component of a web template system.Web templates can be used to set up any type of website. In its simplest sense, a web template operates similarly to a form letter for use in setting up a website.”

[http://en.wikipedia.org/wiki/Web_template, 10 December 2012]

Macros Compile-time meta-programming Run-time meta-programming Staged computation Multi-stage programming

A macro (from the Greek μακρό for "big" or "far") in computer science is a rule or patternthat specifies how a certain input sequence (often a sequence of characters) should be mapped to a replacement input sequence (also often a sequence of characters) according to a defined procedure. The mapping process that instantiates (transforms) a macro use into a specific sequence is known as macro expansion. A facility for writing macros may be provided as part of a software application or as a part of a programming language.

[...]

Macro systems that work at the level of abstract syntax trees are called syntactic macros and preserve the lexical structure of the original program.

[http://en.wikipedia.org/wiki/Macro_(computer_science), 10 December 2012]

© 2012 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 22 These deﬁnitions are not very general (abstract) and not helpful to demarcate templates and related concepts. Become a Wikipedia contributor!

Arguments (context) of templates. Text (hopefully syntax in the target language) Conditionals over context of template. Loops over context of templates. Declarations, statements, expressions in host language. ...

StringTemplate uses ANTLR. ANTLR uses StringTemplate.

[http://www.antlr.org/wiki/display/ST4/Introduction, 10 December 2012]

An extended “if” with keywords for “then” and “else”. ;;define my-if as a macro (define-syntax my-if (lambda (x) ;;establish that "then" and "else" are keywords (syntax-case x (then else) ( ;;pattern to match (my-if condition then yes-result else no-result) ;;transformer (syntax (if condition yes-result no-result)) ) )))

[http://www.ibm.com/developerworks/linux/library/l-metaprog2/index.html]

© 2012 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 27 A macro is pretty much like a template except that it is integrated into parsing/compile time of language processing and emits code (ASTs) for the same language.

Find a proper deﬁnition that ﬁts into the context of this lecture.

© 2012 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 28 often have complex enabling Boolean equations over these 2 Preprocessors aspects, which encode configuration dependencies, and Many programming language environments provide complex nesting to allow code sharing between “preprocessor” facilities, allowing considerable automated configurations. “editing” of a source file “before” it is seen by the language The term “preprocessor” comes from the fact that these processor. Preprocessors with conditionals are often used facilities have traditionally been grafted onto languages as as compile-time software configuration management tools, an afterthought, and consequently implemented as a with preprocessor “variables” naming configuration “preprocessing” pass to perform the “editing”, and aspects. These aspects can be Boolean (enabling or occurring before the compilation step (most Unix systems disabling a feature) or parametricConﬁguration (e.g. a integer specifying managementand many “C” compilers). For efficiency reasons, a buffersize), usually with a parameter value that by preprocessing is often now integrated into the compile step, convention disables the feature (e.g., buffersize == but the name remains. 0). with the C preprocessor The preprocessor provides, via special “preprocessor #ifndef Unix directives” easily found in the program text, several types #define Unix 1 of facilities that change the apparent source file: #endif • importing fixed blocks of text (C #include … char filename[132]; files) #if SDOS • inserting parameterized blocks of text (C macro short int result[256]; invocations, COBOL copybooks) int status; This code uses preprocessor #endif • selecting or deselecting existing blocks of text … (including other preprocessor directives) based on #if (SDOS|Unix)&!DEMOS somevariables configuration to dealvariable with or expression (e.g., C append_to(&filename,&extension); #if, #elsif, #else, #endif directives) #endif variability. Depending on the #if SDOS • defining blocks of text for later inclusion (C macro syscall(OS_openfile,&filename, #definevariables) set at compile OS_readonly,&status,&result,); • marking certain identifiers used as configuration if (status!=OS_no_error) (preprocessing)variables as defined time, (C different with empty syscall(OS_exit_with_error, #define &status,OS_dummy, source-codemacro) or undefined parts are selected. &status,&result); • testing the definedness status of such #elsif Unix configuration variables (C #ifdef, #ifndef, if (errno=fopen(&filename,readmode)) ) { exit(errno); } defined, undefined #else // DEMOS • combining two or more language lexemes to … create a new lexeme. This is almost always used #endif to construct a “gensym” identifier (C macro body [Ira D. Baxter, Michael Mehlich: “Preprocessor Conditional Removaloperation by Simple ## Partial). Evaluation”] Figure 1: Source code with preprocessor directives © 2012 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sleC, C++ and COBOL85 all provide a preprocessor29 Figure 1 shows typical configured source code. The facility defined by the language standard, based on the preprocessor variables Unix, DEMOS and SDOS are aspects language lexemes. Users of languages without native controlling which operating system calls are made. By preprocessing facilities often fall back on the C configuration, we mean all source code defining or preprocessor (if the lexeme structures are close enough, e.g. dependent on an aspect; for SDOS, this is the conditional FORTRAN), custom preprocessors (e.g., Generic declaration of a result buffer, the filename extension- PreProcessor [6]) or on ad hoc solutions modeled after appending code, and the “then” branch of the file opening those for C, often based on a string processing tool like logic. Note that configurations often overlap because of PERL. shared logic, such as appending the extension. Ada83 (and Ada95) were specifically designed without The configuration space managed can be enormous. N a preprocessor on the grounds that such editing can be done N Boolean configuration variables can provide for up to 2 by simply choosing, outside the scope of the language, possibly different instantiated configuration; the Linux which source components to use. However, this artifice operating system has roughly 1000 configuration variables forces large grain source components that replicate [7]. The preprocessor conditionals found in large systems considerable content, creating the very maintenance

!"#$%%&'()*+#,+-.%+/').-.+0#"1'()+2#(,%"%($%+3(+4%5%"*%+/()'(%%"'()+6024/789:+

8;<=>?;9@8@;AB8C+D9

Find a proper deﬁnition that ﬁts into the context of this lecture.

Expression holes VB9

Transformation language and data language are amalgamated on the XSLT grounds of both using XML.

[http://blogs.msdn.com/b/vbteam/archive/2008/02/21/vb-xml-cookbook-recipe-1-xml-transformations-using-xml-literals-replacing-xsl-for-each-doug-rothaus.aspx]

© 2012 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 31 We begin to touch upon the area of language embedding (to be studied more later). In the VB9 case, VB and XML are “syntactically amalgamated”.

How aware of the target language should templates be? What formal guarantees to provide? (Think of type safety.) How to serve the special case of host = target language? (“macros”) How does this discuss generalize to compile-time metaprogramming? ... to run-time metaprogramming?

StringTemplate, XPand, MOF M2T, and others are not language-aware. They basically produce any text.

© 2012 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 34 \v), which stand for arbitrary legal C source satisfying the have defined only a “preprocessor” language, treating the syntactic category lvalue. The left hand quoted string is regions between preprocessor directives as uninterpreted the rule LHS, and the right hand quoted string is the RHS. text, but it is more economical to amortize our The occurrence of the same syntax variable multiple times reengineering investments by operating on a full C in the LHS requires that the same exact sequence occur in grammar. Such a full grammar enables other reengineering both places; this is how the rule is constrained to match activities to be implemented, such as source reformatting, identical source and target. The occurrence of the syntax clone detection, refactoring, etc. Figure 12 shows a variable in the RHS requires that the changed program subgrammar for preprocessor expressions. include what was matched for that variable on the . LHS We wish to parse “C” source before preprocessor Finally, the if condition in this rule is a language-dependent expansion occurs. However, C preprocessor directives can decision procedure that makes sure that the program occur between any pair of tokens. One can code a grammar fragment matched by \v contains no side effects, which for that, but it is extremely unwieldy and therefore would make this transformation incorrect. impractical. Before rule use, a typical rewriting engine first parses Instead, we allow preprocessor directives in the the rule according to its rule language, and then parses the grammar only at places where they commonly occur in real quoted pattern in the language to be transformed (here programs. Figures 10 and 11 show subgrammars, which specified by the default domain phrase as the “C” allow directives only at file scope and in lists of statements, language), to construct pattern trees. At transformation respectively. Allowing directives in other places in the time, the rewrite engine matches the LHS pattern tree grammar is straightforward: replicate the preprocessor against portions of the program, and replaces matched trees conditional subgrammar and instantiate for that place type. by the corresponding RHS tree if the condition is satisfied. In the actual grammar we allowed them in locals lists, and The Figure 2 rule has the effect shown in Figure 3. inside expressions and several other places. before: (*Z)[a>>2]=(*Z)[a>>2]+1; 7 Parsing Unpreproccessed C after: (*Z)[a>>2]++; There are a number of other complications that occur trying to parse C this way. We have a special preprocessor Figure 3 that delays expanding directives if it can and collects information from include files without expanding them in Typically a transformation system will have a large place. “Different” values of a #defined variable may number of rules, and a large number of possible places in a reach the end of a preprocessor conditional. Macro calls program to apply them. It is beyond the scope of this paper often ambiguously look like function calls and have to describe how the transformation system chooses whichNon-syntactic possibly use to be of resolved C preprocessing later. Discussion of how all this is rules and where to apply them. The simple notion that all handled is beyond the scope of this paper. rules possessed are applied leaf-upwards to the entire parse tree for a file is adequate for this paper, and supported directly as one mode of operation of DMS. if (. . .) { . . . DMS captures comments and number literal formats (radix, leading zeros, etc) while lexing and attaches them to #if (. . .) the trees while parsing, and is later able to reproduce the } else { source program in its original- or transformed-form from the trees with all the essential formatting information. #endif Further discussion of the DMS prettyprint capability is . . . outside the scope of this paper. Comments attached to undisturbed trees, and to the roots of rewritten trees are }; preserved, and so comments are not generally lost or Figure 4: Badly nested preprocessor directives and changed by rewriting. (One can always build special rules [Ira D. Baxter, Michael Mehlich: “Preprocessor Conditional Removallanguage by Simple Partial conditionalsEvaluation”] that change this behavior). © 2012 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 35 With sundry additional help in place, DMS parses 6 A Simplified C grammar typically 85% of unpreprocessed C source files in large, In this section we briefly sketch a simplified version of real source systems directly, producing ASTs containing the “C” grammar we use, because the rewrite rules are preprocessor conditionals. The unparsable balance tends to defined in terms of grammar tokens. Arguably, we could use preprocessor directives in unstructured ways. Figure 4

!"#$%%&'()*+#,+-.%+/').-.+0#"1'()+2#(,%"%($%+3(+4%5%"*%+/()'(%%"'()+6024/789:+

8;<=>?;9@8@;AB8C+D9

A Formal Way from Text to Code Templates 111 Collection of templates Parameterized template deﬁnitions

tcoll = tmpl ∗ tmpl = define tn targ ∗ tstmt ∗ enddef ! " ! " targ = ttype tvn Template arguments are typed. tstmt = text texpr expand tn texpr ∗ | ! " | ! " if texpr tstmt ∗ else tstmt ∗ endif Template statements in this order: | ! " ! " ! " for tvn in texpr tstmt ∗ endfor actual text, expressions for walking | ! " ! " tn template names context, template expansion, text text phrases conditional and loop controlled by texpr expressions Arbitrary text can be context. ttype variable types generated tvn variable names at this point . Fig. 2. Syntactical domains of TL ⊥ [Guido Wachsmuth: “A Formal Way from Text to Code Templates”] the various syntactical© 2012 domainsRalf Lämmel, Software of aLanguage language. Engineering http://softlang.wikidot.com/course:sle A dynamic semantics is described 36 in an operational style by judgements for the execution of language instances. We follow some notational conventions in this paper: For semantical domains, we use standard domain constructors, namely products (“ ”), list types (“∗”), × and function domains (“ fin ”). We restrict ourselves to pointwise definition of functions, i.e. these functions→ are only defined for a finite subset of the domain X in f : X fin Y .Formostx X we have that f(x)= .Theentirelyundefined function→ is denoted by “ ”.∈ In inference rules, we reuse⊥ domain names as stems of meta-variables. Tuples⊥ and sequences are constructed via ,..., .Inorderto concatenate several lists to anewone,weusethenotation % ·: ...:· & .Updateof afunctionf for a point x to return y is denoted by f[x y].% Function· · & application is highlighted using the notation f @ x. '→

3ACoreTextTemplateLanguage

In this section, we provide formal semantics of the core concepts in template languages for text generation. For this purpose, we introduce TL ,afunctional language similar to StringTemplate, XPand, and even more to MOF⊥ M2T.

Syntax. Figure 2 shows a grammar of the template language TL .Webrieﬂy ⊥ read its rules: A template collection tcoll consists of a list tmpl∗ of templates. A template declares a list targ∗ of arguments and a list tstmt∗ of template statements. Template statements include simple text, expression evaluation, template expansion, conditional statement, and iteration. Template languages typically comprise an expression language for navigating the source model. This sublanguage highly depends on the technological space of the source model. In the grammarware space, we need to navigate syntax trees. In the modelware space, we need to navigate object-oriented models. For this reason, we do not specify an expression sublanguage in detail. Instead, we specify several requirements for its syntax and semantics. Syntactically, we assume domains for expressions (texpr), types (ttype), and variables (tvn). Static semantics of templates 112 G. Wachsmuth

Domains τ = ttype (Expression types) T = tvn ﬁn τ (Variable type table) → Θ = tn ﬁn τ ∗ (Template type table) → Principal judgements tcoll (Well-typedness of template collections) " Θ tmpl : Θ (Extraction of type context) " Θ tmpl (Well-typedness of templates) " Θ,T tstmt (Well-typedness of template statements) " T texpr : τ (Well-typedness of template expressions) " L τ : τ (Decomposition of list types) " B τ (Booleaness of types) "

Static semantics. The domains involved in the static semantics and the principal judgements are shown in Fig. 3. For the expression sublanguage, we require adomainoftypes(τ), a domain for variable environments (T )tostorevariable types, and a judgement of well-typedness of expressions under a given environment. Furthermore, we require types for list-like data structures. We assume the judgment τ : τ ! to hold for a list type τ iff the list elements are of type τ !. !L The judgement B τ is expected to hold iff τ is boolean-like, that is, elements of this type can be! mapped to boolean values. As for the core concepts of TL ,wedefineadomainfortypetables(Θ) to store the argument types of templat⊥ es. Judgements concern the extraction of a type table for a given template collection, the overall well-typedness of a template collection, the well-typedness of a template definition for a given type table, and the well-typedness of a statement for a given type table and variable environment. Figure 4 lists the inference rules for the judgements. Since most of them are straightforward, we do not discuss all these rules in detail. Let us focus on the well-typedness judgement for statements. Θ,T tstmt is meant to hold if the statement tstmt is well-typed in the context of Θ !and T .TheparameterΘ covers the template types whereas T corresponds to an environment mapping variables for template arguments or iterations to types. The axiom [text]encodesthewell-typednessofanytext.Therule[exp]defines well-typedness for expression evaluation: An expression evaluation is well-typed if its expression has a type in the context of the current variable environment. The premises of rule [expand]saythatatemplateexpansioniswell-typed if (1) a table look-up for tn delivers the types of the template arguments and (2) the actual parameter types are subsumed by the formal ones. The rule [for]defineswell-typednessofiterationasfollows:(1)Thetypeτ of the iteration expression is determined under the current variable environment T .Thistypeneedstobealisttypewithacorrespondingtypeτ ! for the list A Formal Way from Text to Code Templates 115

Well-typedness of template collections tcoll !

tmpl : Θ1 Θn 1 tmpl : Θn ⊥! 1 ∧ ··· ∧ − ! n Θn tmpl Θn tmpl ∧ ! 1 ∧ ··· ∧ ! n [collection] tmpl ,...,tmpl !$ 1 n %

Extraction of type context Θ tmpl : Θ !

targ = ttype tvn1 targ = ttype tvnn 1 1 ∧ ··· ∧ n n Θ @ tn = Θ" = Θ[tn ttype ,...,ttype ] ∧ ⊥∧ &→$ 1 n % [extract] Θ deﬁne tn targ ,...,targ ... enddef : Θ" !( $ 1 %) ( )

Well-typedness of templates Θ tmpl !

targ = ttype tvn1 targ = ttype tvnn 1 1 ∧ ··· ∧ n n T = [tvn1 ttype ,...,tvnn ttype ] ∧ ⊥ &→ 1 &→ n Θ,T tstmt 1 Θ,T tstmt m ∧ ! ∧ ··· ∧ ! [template] Θ deﬁne tn targ ,...,targ !( $ 1 n %) tstmt 1,...,tstmt m $ % enddef ( )

Θ,T text [text] ! T texpr : τ ! [expr] Θ,T texpr !( )

(1) Θ @ tn = τ1,...,τn $ % (2) T texpr : τ1 T texpr : τn ∧ ! 1 ∧ ··· ∧ ! n [expand] Θ,T expand tn texpr ,...,texpr !( $ 1 n %)

(1) T texpr : τ L τ : τ " ! ∧! (2) T " = T [tvn τ "] ∧ &→ (3) Θ,T" tstmt 1 Θ,T" tstmt n ∧ ! ∧ ··· ∧ ! [for] Θ,T for tvn in texpr tstmt 1,...,tstmt n endfor !( )$ %( )

(1) T texpr : τ B τ ! ∧! (2) Θ,T tstmt 1 Θ,T tstmt m ∧ ! ∧ ··· ∧ ! [if] Θ,T if texpr !( ) tstmt 1,...,tstmt n $ % else ( ) tstmt n+1,...,tstmt m $ % endif ( )

Fig. 6. Rules of TL dynamic semantics ⊥ A Formal Way from Text to Code Templates 115

Well-typedness of template collections tcoll !

tmpl : Θ1 Θn 1 tmpl : Θn ⊥! 1 ∧ ··· ∧ − ! n Θn tmpl Θn tmpl ∧ ! 1 ∧ ··· ∧ ! n [collection] tmpl ,...,tmpl !$ 1 n %

Extraction of type context Θ tmpl : Θ !

targ = ttype tvn1 targ = ttype tvnn 1 1 ∧ ··· ∧ n n Θ @ tn = Θ" = Θ[tn ttype ,...,ttype ] ∧ ⊥∧ &→$ 1 n % [extract] Θ deﬁne tn targ ,...,targ ... enddef : Θ" !( $ 1 %) ( )

Well-typedness of templates Θ tmpl !

Well-typedness of statements Θ,T tstmt ! Basically, check that all templates are Θ,T text [text] ! correctly invoked and that conditionals T texpr : τ ! and loops use a boolean condition and [expr] Θ,T texpr !( ) loops loop over a list.

(1) Θ @ tn = τ1,...,τn $ % (2) T texpr : τ1 T texpr : τn ∧ ! 1 ∧ ··· ∧ ! n [expand] Θ,T expand tn texpr ,...,texpr !( $ 1 n %)

(1) T texpr : τ L τ : τ " ! ∧! (2) T " = T [tvn τ "] ∧ &→ (3) Θ,T" tstmt 1 Θ,T" tstmt n ∧ ! ∧ ··· ∧ ! [for] Θ,T for tvn in texpr tstmt 1,...,tstmt n endfor !( )$ %( )

(1) T texpr : τ B τ ! ∧! (2) Θ,T tstmt 1 Θ,T tstmt m ∧ ! ∧ ··· ∧ ! [if] Θ,T if texpr !( ) tstmt 1,...,tstmt n $ % else ( ) tstmt n+1,...,tstmt m $ % endif ( ) [Guido Wachsmuth: “A Formal Way from Text to Code Templates”] © 2012 RalfFig. Lämmel, 6. SoftwareRules Language of TL Engineeringdynamic http://softlang.wikidot.com/course:sle semantics 39 ⊥ Dynamic semantics of templates 114 G. Wachsmuth

Domains ν (Expression values) β = true false (Boolean values) | ϕ = text (Text phrases) ψ = ϕ∗ T = tvn ﬁn ν (Variable environment) → Θ = tn ﬁn ( tvn ∗ tstmt ∗ ) (Template code table) → × Principal judgements tcoll Θ (Extraction of code table) # ⇒ Θ tmpl Θ Generate text # ⇒ T,Θ tstmt ψ (Execution of template statements) # ⇒ T texpr ν (Evaluation of template expressions) # ⇒ ν ϕ (Text conversion of values) # ⇒ L ν ν∗ (Decomposition of lists) # ⇒ B ν β (Boolean conversion of values) # ⇒

[GuidoFig. Wachsmuth: 5. Model “A Formal of TL Way fromdynamic Text to Code semantics Templates”] ⊥ © 2012 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 40 elements. (2) A new variable environment T ! is retrieved by updating the type of the iteration variable tvn to τ !.(3)Allstatementsintheiterationneedtobe well-typed under the new environment. The rule [if]forconditionalstatementsreadssimilarly:(1)Thetypeτ of the condition expression is determined. Thistypeneedstobeboolean-like.(2)All statements in a conditional statement need to be well-typed in the current context.

Dynamic semantics. While the static semantics is concerned with well-typedness judgements, the dynamic semantics provides judgements about text generation. Figure 5 shows the model of the dynamic semantics. We use a style that emphasises the similarities of the models of static and dynamic semantics. While Θ covers the template types in the static case, it models the template code in the dynamic case. Similar variations apply to the principal judgements. That is, code table extraction corresponds to type table extraction, text generation out of template statements to well-typedness of template statements, and text generation out of template expressions to type assignment for template expressions. For the expression sublanguage, we require a domain of values (ν), a domain for variable environments (T )tostorevariablevalues,andajudgementforeval- uating expressions under a given environment. The structure of this judgement implies side-condition free evaluation. This ensures model view separation [13]. Furthermore, we assume the judgment ν ν ,...,ν to hold for a list-like !L ⇒# 1 n $ data structure ν iﬀ ν1,...,νn are the elements of this structure. The judgement B ν β is expected to hold iﬀ ν can be converted into the boolean value β. Finally,! ⇒ we require a judgement ν ϕ for the conversion of values to text. Figure 6 lists the inference rules! for⇒ the judgements. Again, we do not discuss all these rules in detail. Let us focus on the text generation judgement for Code table extraction tcoll Θ ⇒

tmpl Θ1 Θn 1 tmpl Θn ⊥ 1 ⇒ ∧ ··· ∧ − n ⇒ [collection] tmpl ,...,tmpl Θ 1 n ⇒

Code table extraction Θ tmpl Θ ⇒

targ = ttype tvn1 targ = ttype tvnn 1 1 ∧ ··· ∧ n n Θ = Θ[tn tvn1,...,tvnn , tstmt 1,...,tstmt m ] ∧ → [extract] Θ deﬁne tn targ ,...,targ Θ 1 n ⇒ tstmt 1,...,tstmt m enddef

Statement evaluation T,Θ tstmt ψ ⇒

T,Θ text text [text] ⇒ T texpr ν ν ϕ ⇒ ∧⇒ [expr] T,Θ texpr ϕ ⇒

(1) Θ @ tn = tvn1,...,tvnn , tstmt 1,...,tstmt m [Guido Wachsmuth: “A Formal Way from Text to Code Templates”] (2) T texpr ν1 T texpr νn ∧ 1 ⇒ ∧ ··· ∧ n ⇒ (3) T = [tvn1© 2012ν 1Ralf,..., Lämmel,tvn Softwaren Languageνn] Engineering http://softlang.wikidot.com/course:sle 41 ∧ ⊥ → → (4) T , Θ tstmt 1 ψ1 T , Θ tstmt m ψm ∧ ⇒ ∧ ··· ∧ ⇒ (5) ψ = ψ1 : ...: ψm ∧ [expand] T,Θ expand tn texpr ,...,texpr ψ 1 n ⇒

(1) T texpr ν L ν ν1,...,νm ⇒ ∧ ⇒ (2) T1 = T [tvn ν1] Tm = T [tvn νm] ∧ → ∧ ··· ∧ → (3) T1, Θ tstmt 1 ψ1,1 T1, Θ tstmt n ψ1,n ∧ ⇒ ∧ ··· ∧ ⇒ ∧ ··· Tm, Θ tstmt 1 ψm,1 Tm, Θ tstmt n ψm,n ∧ ⇒ ∧ ··· ∧ ⇒ (4) ψ = ψ1,1 : ...: ψ1,n : ...: ψm,1 : ...: ψm,n ∧ [for] T,Θ for tvn in texpr tstmt 1,...,tstmt n endfor ψ ⇒

(1) T texpr ν B ν true ⇒ ∧ ⇒ (2) T,Θ tstmt 1 ψ1 T,Θ tstmt n ψn ∧ ⇒ ∧ ··· ∧ ⇒ (3) ψ = ψ1 : ...: ψn ∧ [then] T,Θ if texpr tstmt 1,...,tstmt n else ... endif ψ ⇒

(1) T texpr ν B ν false ⇒ ∧ ⇒ (2) T,Θ tstmt 1 ψ1 T,Θ tstmt n ψn ∧ ⇒ ∧ ··· ∧ ⇒ (3) ψ = ψ1 : ...: ψn ∧ [else] T,Θ if texpr ... else tstmt 1,...,tstmt n endif ψ ⇒

Fig. 6. Rules of TL dynamic semantics. ⊥ Code table extraction tcoll Θ ⇒

tmpl Θ1 Θn 1 tmpl Θn ⊥ 1 ⇒ ∧ ··· ∧ − n ⇒ [collection] tmpl ,...,tmpl Θ 1 n ⇒

Code table extraction Θ tmpl Θ ⇒

targ = ttype tvn1 targ = ttype tvnn 1 1 ∧ ··· ∧ n n Θ = Θ[tn tvn1,...,tvnn , tstmt 1,...,tstmt m ] ∧ → [extract] Θ deﬁne tn targ ,...,targ Θ 1 n ⇒ tstmt 1,...,tstmt m enddef

Statement evaluation T,Θ tstmt ψ ⇒

T,Θ text text [text] ⇒ T texpr ν ν ϕ ⇒ ∧⇒ [expr] T,Θ texpr ϕ ⇒

(1) Θ @ tn = tvn1,...,tvnn , tstmt 1,...,tstmt m (2) T texpr ν1 T texpr νn ∧ 1 ⇒ ∧ ··· ∧ n ⇒ (3) T = [tvn1 ν1,...,tvnn νn] ∧ ⊥ → → (4) T , Θ tstmt 1 ψ1 T , Θ tstmt m ψm ∧ ⇒ ∧ ··· ∧ ⇒ (5) ψ = ψ1 : ...: ψm ∧ [expand] T,Θ expand tn texpr ,...,texpr ψ 1 n ⇒

(1) T texpr ν B ν true ⇒ ∧ ⇒ (2) T,Θ tstmt 1 ψ1 T,Θ tstmt n ψn ∧ ⇒ ∧ ··· ∧ ⇒ (3) ψ = ψ1 : ...: ψn ∧ [then] T,Θ if texpr tstmt 1,...,tstmt n else ... endif ψ ⇒

(1) T texpr ν B ν false ⇒ ∧ ⇒ (2) T,Θ tstmt 1 ψ1 T,Θ tstmt n ψn ∧ ⇒ ∧ ··· ∧ ⇒ (3) ψ = ψ1 : ...: ψn ∧ [else] T,Θ if texpr ... else tstmt 1,...,tstmt n endif ψ ⇒

[GuidoFig. Wachsmuth: 6. Rules “A Formal of TL Waydynamic from Text to semantics.Code Templates”] ⊥ © 2012 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 42 Language-aware template languages

Make sure that only syntactically correct code is produced.

Thus, templates have result types: nonterminals of target language’s grammar.

Grammar transformations come to the rescue: amalgamate template and target syntax by enhancing target language’s grammar to include templates.

For each nonterminal of the target language’s grammar: Enable template expansion with such a result type. Enable conditionals with such a result type. Enable template deﬁnitions with such a result type. Enable nonterminal to generate text as statement. Enable the nonterminal as “type”.

Some special cases: Handle morphemes. Degenerate case of normal nonterminals. Handle */+ symbols. Extract (“fold”) new nonterminal for symbol. Enable loops with such a result type.

introduce texpr ttype tvn tn introduce targ = ttype tvn Introduce template grammar symbols foreach nt : N include nt = expand tn texpr ∗ include nt = if texpr ∗ nt else nt endif include tmpl = deﬁne nt tn targ ∗ nt enddef include tstmt = nt include dn = nt endfor Instrument nonterminals foreach nt : M fold ntM = nt include ntM = texpr Special cases for morphemes. include tstmt = ntM include dn = nt endfor foreach nt : L fold ntL = nt include ntL = for tvn in texpr nt endfor Another rule include tstmt = ntL endfor from the generic introduce tcoll = tmpl ∗ Instrument nonterminals

grammar [Guido Wachsmuth: “A Formal Way from Text to Code Templates”] Fig. 7. Generic adaptation script for the syntactical enhancement of a target language. © 2012 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 46

each nonterminal except morphem classes. We include rules for template expansion and for a conditional statement. Additionally, we include a rule for templates. Third, we enhance the definition of each morphem class. We fold each morphem class to a fresh nonterminal and include a rule for expression evaluation. Fourth, we enhance the definition of nonterminals occurring in a Kleene closure. We fold each of these domains to a fresh nonterminal and include a rule for iteration. Finally, we introduce template collections. The result of the adaptation is a grammar for a template language particu- larly concerned with the target language. In this template language, templates are associated with a particular syntactical domain of the target language. Fig- ure 8 gives an example. The upper part of the figure shows the grammar of a simple programming language PL: A program prog consists of a list of statements pstmt ∗. A statement is either a variable declaration, an assignment, a while loop, or a conditional statement. An expression pexpr is either an integer number, a character string, a variable, a sum, a difference, or a string concate- nation. The lower part of Fig. 8 shows the resulting grammar for the template language TLPL.

Static semantics. During the syntactical enhancement, two helper domains tstmt and dn are constructed. This allows us to use a generic model of the static semantics. The model diﬀers only slightly from the model for TL . Figure 9 high- lights the modiﬁcations. In Θ, we keep the syntactical domain of⊥ the template in An operator suite for grammar adaptation 110 G. Wachsmuth

[Guido Wachsmuth: “A Formal Way from Text to Code Templates”]

2Preliminaries

Grammar adaptation. In this paper, we employ formal grammar transformations for the stepwise adaptation of grammars [9]. This approach offers several benefits: First, the derivation of the template language is formally defined by the application of well-defined adaptation operators. Second, we prevent accidental modifications of the target language since preservation properties of adaptation operators are well understood [9]. Third, the adaptation is generic and can be applied to any target language. Finally, existing tool support automates the derivation process [11]. We rely on a very restricted operator suite for this paper. Figure 1 shows its concrete syntax: First, we can introduce anewruleforafreshnonterminal.We use the same operator to introduce nonterminals without a defining rule (e.g. for morphem classes). Second, we can include an additional rule for a nonterminal. If the nonterminal is yet undefined, include operates as introduce.Third,we can fold aphrase.Thisintroducesafreshnonterminaldefinedbythephrase.All occurrences of the phrase are replaced by the nonterminal. All these operators are purely constructive, i.e. they extend the language defined by the grammar [9]. Finally, we can execute operators foreach nonterminal in a set. There are three yielders of such sets: N yields all nonterminals except morphem classes, M yields all morphem classes, and L yields all nonterminals to which a Kleene closure is applied. Inside a foreach block, a variable provides access to the current nonterminal. yields a nonterminal’s name as a keyword. !·" Natural Semantics. In this paper, we use Natural Semantics [10] for the static and dynamic semantics of template languages. Typically, template languages are functional languages [6]. Describing functional languages by Natural Semantics is well understood. For example, Natural Semantics was used for both the static and dynamic semantics in the formal specification of Standard ML [12]. The general idea of a semantics definition in Natural Semantics is to provide axioms and inference rules for judgements over syntactical and semantical domains.A static semantics is typically described in terms of well-typedness judgements for Summary of the method

Deﬁnition of generic template language. Syntax, static and dynamic semantics Amalgamation with target language’s grammar. Use grammar transformation systematically. Additional amalgamation with semantics (omitted).

[Guido Wachsmuth: “A Formal Way from Text to Code Templates”]

Compute source code any time.

Reasons are purely economic: •Problem 1:Abstraction mechanisms (functions, objects, classes…) have runtime cost –Result 1: You don’t use them –Result 2: That costs you programmer productivity •Idea: Use generative programming (GPCE) •Problem 2:Writing good generative programs is hard