A Practical Unication of Multi-stage Programming and Macros Nicolas Stucki Aggelos Biboudis Martin Odersky EPFL EPFL EPFL Switzerland Switzerland Switzerland [email protected] [email protected] [email protected] Abstract 1 Introduction Program generation is indispensable. We propose a novel Generative programming [9] is widely used in scenarios such unication of two existing metaprogramming techniques: as code conguration of libraries, code optimizations [43] multi-stage programming and hygienic generative macros. and DSL implementations [8, 41]. There are various kinds The former supports runtime code generation and execution of program generation systems ranging from completely in a type-safe manner while the latter oers compile-time syntax-based and unhygienic, to fully typed [35]. Modern code generation. macro systems, like Racket’s, can extend the syntax of the In this work we draw upon a long line of research on language [11]. On the ipside, other program generation metaprogramming, starting with Lisp, MetaML and MetaO- systems may provide a xed set of constructs oering staged Caml. We provide direct support for quotes, splices and top- evaluation [10, 16] like MetaML [38] and MetaOCaml [6, 20, level splices, all regulated uniformly by a level-counting 21, 23]. Phase Consistency Principle. Our design enables the construc- The latter techniques established a new programming tion and combination of code values for both expressions paradigm, called Multi-stage Programming (MSP) oering and types. Moreover, code generation can happen either at a principled, well-scoped and type-safe approach to code runtime à la MetaML or at compile time, in a macro fashion, generation [37]. Programmers make use of two constructs, à la MacroML. quote and splice, to delay and compose representations of We provide an implementation of our design in Scala and expressions. Conceptually, users are able to manually indi- we present two case studies. The rst implements the Hid- cate which parts of their program are dynamic and which den Markov Model, Shonan Challenge for HPC. The second static. Even though this technique is inspired by advance- implements the staged streaming library Strymonas. ments in partial evaluation [26] it proved useful to have it in a programming language with rst-class support. Part of the CCS Concepts • Software and its engineering → Lan- power of this programming model, comes from a regulation guage features; Macro languages; mechanism that attributes levels to terms [36]; these systems are type-safe in a modular way (type checking the genera- Keywords Macros, Multi-stage programming, Scala tor ensures the validity of the generated code). Nowadays, ACM Reference Format: gaining inspiration from MetaML and MetaOCaml, many Nicolas Stucki, Aggelos Biboudis, and Martin Odersky. 2018. A programming languages provide support for similar mecha- Practical Unication of Multi-stage Programming and Macros. In nisms such as F#, Haskell (Template Haskell [33] and later Proceedings of the 17th ACM SIGPLAN International Conference on Typed Template Haskell [15]), Converge [42] and others. Generative Programming: Concepts and Experiences (GPCE ’18), No- While MSP is primarily a metaprogramming technique for vember 5–6, 2018, Boston, MA, USA. ACM, New York, NY, USA, runtime code generation it has been shown that its semantics 14 pages. hps://doi.org/10.1145/3278122.3278139 can specify compile-time metaprogramming as well. MacroML [12] showed that the treatment of staged evalu- ation can form the basis for generative macros (i.e. macros Permission to make digital or hard copies of all or part of this work for that cannot inspect code) or more precisely, function inlin- personal or classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear ing. Theoretically it has been proven that MacroML’s inter- this notice and the full citation on the rst page. Copyrights for components pretation is a denotational semantics where MetaML is the of this work owned by others than ACM must be honored. Abstracting with internal language of the model. Monnier et al.[25] rst ex- credit is permitted. To copy otherwise, or republish, to post on servers or to pressed inlining as staged computation but MacroML oered redistribute to lists, requires prior specic permission and/or a fee. Request a user-level perspective by reusing the same mechanisms of permissions from [email protected]. quotes and splices; where splices can appear at the top-level GPCE ’18, November 5–6, 2018, Boston, MA, USA © 2018 Association for Computing Machinery. (not nested in a quote). Modular Macros [44] prototyped a ACM ISBN 978-1-4503-6045-6/18/11...$15.00 compile-time variant of MetaOCaml which also comprises hps://doi.org/10.1145/3278122.3278139 part of our inspiration. GPCE ’18, November 5–6, 2018, Boston, MA, USA N. Stucki, A. Biboudis, M. Odersky def power_s(x: Expr[Double], n: Int): Expr[Double] = Scala is a multi-paradigm programming language for the if (n == 0) '(1.0) JVM oering a metaprogramming API called scala.reect [5]. else if (n % 2 == 1) '(~x * ~power_s(x, n - 1)) scala.reect supports type-aware, runtime and compile-time else '{ val y = ~x * ~x; ~power_s('(y), n / 2) } code generation providing an expressive and powerful sys- tem to the user (both generative and analytical). Besides the inline def power(x: Double, inline n: Int): Double = ~power_s('(x), n) success of scala.reect, the API exposed compiler internals and gave rise to portability problems between compiler ver- valx=2 sions [24]. We implemented our system for the Dotty [39] // 1) staged, runtime generation compiler for Scala and we believe that the design is portable val power5 = ('{ (x: Double) => ~power_s('(x), 5)}).run in other languages as well. power5(x) // 2) macro, compile-time generation Organization power(x, 5) First, in Section2, we introduce a motivat- // Both generate:{ valy=x*x; val y2=y*y;x* y2} ing example to explain the high-level semantics of quotes and splices. In Section3 we present PCP and the details of multi-staging and macros. In Section4 we discuss how to Figure 1. Power function, staged or inlined implement cross-stage persistence (CSP) in this system. In Section5 we show how to simplify the handling of type splices in quoted code. In Section6 we discuss lifted lambdas While the same line of work inspired many metaprogram- and β-reduction optimizations. Section7 describes the im- ming libraries and language features, to our knowledge built- plementation in Dotty. Section8 presents two case studies 1: in support for both run-time MSP and generative macros (i) we give a sample solution to the Hidden Markov Model has not been implemented previously in a unifying man- challenge as specied in Shonan Challenge for Generative ner. We advocate that such a unication has a two-fold ben- Programming [1] and (ii) we port Strymonas [22], a staged et: 1) users rely on a single abstraction to express code library for streams. In Section9 we discuss the related work generation and 2) having a single subsystem in the com- and conclude in Section 10. piler favors maintainability. Our view regarding top-level splices is on par with the benets of MSP on domain-specic 2 Overview of Quotes and Splices optimizations[7, 21, 22]: in modern programming languages, inlining (à la C++) with a suciently smart partial evaluator Our metaprogramming system is built on two well-known 2 is not necessarily equivalent with domain-specic optimiza- fundamental operations: quotation and splicing. A quota- tions that can be done at compile-time. tion is expressed as '(...) or '{...} for expressions (both In our work a staged library can be used, unaltered, either forms are equivalent) and as '[...] for types. Splicing is as a macro or a run-time code generator. We illustrate stag- expressed with the ~ prex operator. ing and macros via the folklore example of a simple power If e is an expression, then '(e) or '{e} represent the function, which has been used for demonstrating partial opaque typed abstract syntax tree representing e. If T is a evaluation techniques. The power_s, staged function is de- type, then '[T] represents the opaque type structure repre- ned recursively using the basic method of exponentiation senting T. The precise denitions of typed abstract syntax by squaring. The inline function power becomes a macro by tree or type structure do not matter for now, the expressions expanding power_s. In Figure1 we see two dierent ways are used only to give some intuition that they represent code to use it: 1) staged; generation happens at runtime and 2) as a value. Conversely, ~e evaluates the expression e, which inlined generation happens at compile-time. must yield a typed abstract syntax tree or type structure, and embeds the result as an expression (respectively, type) in Contributions In this paper, inspired from MetaML and the enclosing program. Informally, by quoting we delay the MacroML we present a practical implementation of homoge- evaluation of an expression—or we stage, in MSP terms—and neous generative metaprogramming (HGMP) for Scala: by splicing, we evaluate an expression before embedding the • We present a design with quotes, splices, and top-level result in the surrounding quote. splices to support both MSP and macros simultane- Quotes and splices are duals of each other. For arbitrary ously. expressions e: T and types T we have ~'(e) = e and ~'[T] • We extend the operation of splicing to handle terms = T; for arbitrary AST-typed expressions e2: Expr[T] and and types uniformly. t: Type[T] we have '(~e) = e and '(~t) = t. • We present how our system operates under a MetaML- inspired check, Phase Consistency Principle (PCP), that 1The code of the case studies, along with unit tests and benchmarks are at regulates free variable accesses in quoted and spliced hps://github.com/nicolasstucki/doy-staging-gpce-2018 expressions and types uniformly, for both MSP and 2Or more accurately quasiquotation, which represents quotes with unquoted macros.