From Macros to Reusable Generative Programming Shriram Krishnamurthi Matthias Felleisen and Bruce F Duba Department of Computer Science Rice University Houston TX USA Department of Computer Science Seattle University Seattle WA USA Abstract Generative programming is widely used b oth to develop new programming languages and to extend existing ones with domainsp ecic sublanguages This pap er describ es McMicMac a framework for gener ative programming McMicMac uses treetransforming macros as lan guage sp ecications and enhances them with inherited and synthesized attributes The enhanced transformers can describ e general compila tion tasks Families of these sp ecications are group ed into mixinlike collections called vo cabularies Programmers can dene new languages by comp osing these vo cabularies Wehave implemented McMicMac for Scheme and used it to build several systems including the DrScheme programming environment The principles of McMicMac carry over to other languages and environments Intro duction Generative programming is an old and go o d ideawrite programs to write your programs for you This idea has b een applied widely from compilation to creating domainsp ecic languages to the masspro duction of comp onents It is gaining increasing prominence due to its p otential economic impact b ecause it can b oth reduce manual lab or and increase the eciency and correctness of programs In this pap er we describ e a general framework for generative programming in Scheme Concretelywe describ e McMicMac a framework develop ed at Rice University McMicMac supp orts the pro duction of extensible language com p onents A programmer can generate new languages by comp osing language comp onents Because the comp onents are parameterized over the base language McMicMac programmers can reuse the same languagedening comp onents to build many dierent languages The system thus greatly simplies the design and implementation of domainsp ecic languages as extensions of base languages Our system plays a crucial role in the construction of a large programming environment DrScheme Conversely the pro cess of building DrScheme has This work is partially supp orted by NSF grants CCR CDA and CCR and a Texas ATP grant help ed us debug and rene the design of our system McMicMac has also b een used to build to ols for languages other than Scheme Starting with a typ echecker and compiler for a parenthetical version of Java it has b een used to create lan guage extensions representing design patterns and other sp ecications The rest of this pap er is organized as follows Section providesanex tended example that illustrates some of the kinds of abstractions that genera tive programming makes p ossible Section describ es the structure of McMicMac through a series of examples and section renes this discussion Section sum marizes the deploymentofMcMicMac Section discusses related eorts and section oers concluding remarks and suggests directions for future work An Illustrative Example Supp ose a programmer wanted to distribute a library that creates and executes nitestate automata The library of typ es and pro cedures or classes with meth o ds should hide the actual representation used for the automata b ecause there areatleasttwo dierent automata representations data structures and exe cutable values Data structures are useful for manipulating automata as data eg for minimizing automata They can also b e run via an interpreter though this can b e inecient If the automata are b eing created solely for execution then it is typically much more ecient to translate them directly into co de such as functions or lab eled statements Hence the library should oer programmers a way to sp ecify inline automata in a representationindep endent yet intuitive manner This would allow clients to write expressions suchasthatshown in gure The example uses a new construct automaton to dene an automaton that starting in state checks whether a stream of s and s b egins with and then alternates strictly b etween the twovalues The sp ecication in gure might translate into the Scheme co de shown in gure It represents automata as pro cedures that consume an input stream The states are nested mutuallyrecursive pro cedures Each pro cedure represents one states transition relation All unexp ected inputs generate an exception unexpectedinputexn so the empty transition relation represents an error state If the automaton attempts to insp ect past the end of a nite input stream a dierent exception is raised which a client can handle to observe successful completion Because programming languages likeScheme optimize calls in tail p osition to jumps or gotos the state transitions in the example are quickand accumulate no evaluation context collo quiallystack space Unfortunately the library programmer cannot dene automaton b ecause it is not a pro cedure The subterms of automaton are not expressions in Scheme as pro cedure arguments must b e rather they are terms in a distinct domain sp ecic language Furthermore automaton is a binding construct as claried by gure which cannot b e dened pro cedurally Therefore automaton can not b e dened in most traditional programming languages automaton state state errorstate state state state errorstate errorstate Fig An Automaton Description lambda inputstream letrec state lambda case nexttoken inputstream errorstate state else raise makeob ject unexpectedinputexn state lambda case nexttoken inputstream state errorstate else raise makeob ject unexpectedinputexn errorstate lambda case nexttoken inputstream else raise makeob ject unexpectedinputexn state Fig Compiled Automaton Representation The automaton library mightcontain other op erations of this form For in stance the library mayprovide a means for interleaving the execution of two automata This would enable the client programmer to write runalternating M stream M stream which runs automaton M on stream stream and M on stream in strict alternation In this case even though b oth subterms of runalternating are legal Scheme expressions Schemes callbyvalue evaluation order would rst run M on stream until terminationwhichmaynever o ccurb efore it b egins to run M Thus runalternating also cannot b e a pro cedure in a callbyvalue language Both constructs illustrate useful and imp ortant abstractions that help pro grammers write software eectively These abstractions are not proceduralhow ever Instead they dene notations that are not part of the languages syntax In other cases they require b ehavior that is dierent from what the languages semantics sp ecies More generally automaton and runalternating are b oth linguistic abstractions ie they create a little language within a larger language denemacro automaton automaton startstate statename input newstate syntax lambda inputstream letrec statename lambda case removetoken inputstream inputnewstate else raise makeob ject unexpectedinputexn startstate Fig Automaton Macro to accomplish some sp ecialized task Generative programming frameworks must supp ort such language construction The McMicMac Framework McMicMac is a framework for creating languages such as that for automata We explain McMicMac through a series of examples reecting increasingly com plex proto cols The examples are intentionally simplistic in avor but they cor resp ond to some of the nontrivial uses wehave encountered while building DrScheme Macros Examples like automaton and runalternating are expressible as McMic Mac macros The macros of McMicMac are descendants of those in Lisp and Scheme They transform treeshap ed data rather than manipulating at data like the stringpro cessing macros in the C prepro cessor In short macros implement a simple form of extensible parsing A parser is conceptually a table of rules that map syntactic shap es to co de When the input matches a shap e called a trigger the parser lo oks up the trig gers transformation rule called the elaborator and uses it to pro duce abstract syntax The parsing table is traditionally xed thus limiting the input language a parser can recognize Macro denitions add rules to a parsers table Unlike traditional parse rules though macro rules do not directly generate abstract syntax Instead they generate terms in the source language The parser then reanalyzes the generated term and continues this pro cess until it obtains a canonical form In McMicMac the programmer denes triggers using a patternmatching no tation originally due to Kohlb ecker and Wand When an input term matches denemacro automaton automaton startstate statename input newstate syntax makeautomatonrep startstate list makestaterep statename maketransitionrep input newstate Fig Alternate Automaton Macro a trigger the matcher generates a pattern environment that maps pattern vari ables to the corresp onding source terms in the input It then invokes the elab o rator to generate a source term This term can b e parameterized over subterms in the input The elab orator extracts these input subterms from the pattern environment Figure presents a concrete example the macro for the automaton con struct of section The keyword denemacro is followed by a set of literals here automaton and that may app ear in the input The literals are fol lowed by the trigger All symb ols in the trigger that do not app ear in the literal set are pattern variables A pattern followed by ellipses matches zero or more instances of the pattern It binds each pattern variable to the sequence of subterms that corresp