Scala Macros: Let Our Powers Combine!

Total Page:16

File Type:pdf, Size:1020Kb

Scala Macros: Let Our Powers Combine! Scala Macros: Let Our Powers Combine! On How Rich Syntax and Static Types Work with Metaprogramming Eugene Burmako EPFL, Switzerland eugene.burmako@epfl.ch ABSTRACT Compile-time metaprogramming has been proven immensely Compile-time metaprogramming can be thought of as the useful enabling programming techniques such as language algorithmic construction of programs at compile-time. It’s virtualization, embedding of external domain-specific lan- often used with the intent of allowing programmers to gen- guages, self-optimization, and boilerplate generation among erate parts of their programs rather than having to write many others. these program portions themselves. Thus, metaprograms In the recent production release of Scala 2.10 we have in- are programs who have a knowledge of other programs, and troduced macros, an experimental facility which gives its which can manipulate them. users compile-time metaprogramming powers. Alongside of Across languages and paradigms, this sort of metapro- the mainline release of Scala Macros, we have also intro- gramming has been proven immensely useful, acting as an duced other macro flavors, which provide their users with enabling force behind a number of programming techniques, different interfaces and capabilities for interacting with the such as: language virtualization (overloading/overriding se- Scala compiler. mantics of the original programming language) [12], em- In this paper, we show how the rich syntax and static bedding of external domain-specific languages (tight inte- types of Scala synergize with macros, through a number gration of external DSLs into the host language) [40, 48], of real case studies using our macros (some of which are self-optimization (self-application of optimizations based on production systems) such as language virtualization, type analysis of the program’s own code) [35], and boilerplate providers, materialization of type class instances, type-level generation (automatizing repetitive patterns which cannot programming, and embedding of external DSLs. We explore be readily abstracted away by underlying language) [36, 30]. how macros enable new and unique ways to use pre-existing In the recent production release of Scala 2.10 we have in- language features such as implicits, dynamics, annotations, troduced Scala Macros [6] as a new experimental language string interpolation and others, showing along the way how feature– Scala’s realization of compile-time metaprogram- these synergies open up new ways of dealing with software ming. This new feature enables the compiler to recognize development challenges. certain methods in Scala programs as metaprograms, or macros, which are then themselves invoked at certain points of compilation. When invoked, macros are provided with a Categories and Subject Descriptors compiler context, which exposes the compiler’s representa- D.3.3 [Programming Languages]: Language Constructs tion of the program being compiled along with an API pro- and Features viding certain compiler functionality such as parsing, type- checking and error reporting. Using the API available in the General Terms context, macros can influence compilation by, for example, changing the code being compiled or affecting type inference Languages performed by the typechecker. The most basic form of compile-time metaprogramming in Keywords our system is achieved by def macros, plain methods whose Compile-Time Metaprogramming, Type Classes, Domain- invocations are expanded during compilation. In addition to Specific Languages, Scala these def macros, we have identified, implemented, and ex- perimented with different macro flavors: dynamic macros, string interpolation macros, implicit macros, type macros, 1. INTRODUCTION and macro annotations. Each of these flavors encompasses some different way in which macros are presented to, and can be used by users. We will go on to explore a number Permission to make digital or hard copies of all or part of this work for of applications, which have proven markedly difficult or im- personal or classroom use is granted without fee provided that copies are possible to achieve via other means, each of which exercise not made or distributed for profit or commercial advantage and that copies one of these macro flavors. bear this notice and the full citation on the first page. To copy otherwise, to Our contributions are as follows: republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. • Scala ’13, Montpellier, France We describe a number of macro flavors, which are in- Copyright 2013 ACM 978-1-4503-2064-1 ...$15.00. tegrated in a principled way alongside of Scala’s rich syntax and strong static type system. Even in this simple form, the macro is arguably more use- ful than a corresponding function in an eager language such • We provide a comprehensive validation of the utility of as Scala, because it does not calculate the message unless the these macro flavors through a number of real case stud- asserted condition is violated. The necessity to shield the ies. We show that macros (a) enable language virtu- evaluation of the message for performance reasons usually alization, (b) can implement a form of type providers, produces noticeable amounts of boilerplate, which cannot (c) can be used to automatically generate type class be easily abstracted away. Scala does support lazy eval- instances, (d) simplify type-level programming, and uation with by-name parameters, but the inner workings (e) enable embedding of external domain-specific lan- of their internal representation might also degrade perfor- guages. We additionally go on to show that macros mance. Macros are able to address the performance problem can re-implement non-trivial language features such as without downsides. code lifting and materialization of type class instances. In addition to def macros, introduced in Scala 2.10, we The rest of the paper is organized as follows. Section 2 have created a fork of Scala [5], where we have conceived, im- provides a basic introduction to Scala macros. Section 3 in- plemented, and experimented with a number of other macro troduces the macro flavors we have experimented with, set- flavors– macros which provide different interfaces and capa- ting the stage for Section 4, which outlines some of the use bilities for interacting with the Scala compiler. cases that these flavors enable and discusses alternative ways of achieving similar functionality. Throughout the paper we 3. HAMMERS: THE MACRO FLAVORS deliberately avoid going into the details of our macro sys- Macros realize the notion of textual abstraction [17], which tem (expansion semantics, integration with the typechecker, consists of recognizing pieces of text that match a specifica- handling of hygiene, interplay between macros, etc) in or- tion and replacing them according to a procedure. In Lisp, der to focus specifically on how macros work together with the origin of macros, programs are represented in a homo- Scala’s rich syntax and static type system. geneous way with S-expressions. Therefore recognition and replacement of program fragments can be done uniformly, 2. INTUITION regardless of whether a transformed fragment represents e.g. To get acquainted with metaprogramming in Scala, let an arithmetic expression or a function definition. us explore the simplest flavor of Scala macros, def macros, In Scala, a language with rich syntax and static types, which were inspired by macros in Lisp [17, 13] and Nemerle compile-time transformations of code naturally distinguish [36]. terms and types, expressions and definitions, following the Def macros are methods, whose calls are expanded at com- architecture of scalac, the Scala compiler. Therefore it pile time. Here, expansion means transformation into a code makes sense to recognize the following three realizations of snippet derived from the method being called and its argu- textual abstraction in Scala: term macros, which expand ments. When such macros are expanded, they operate with terms, type macros, which expand types, and macro anno- a context, which exposes the code to be expanded and rou- tations, which expand definitions. In this section we will tines to manipulate code snippets. highlight these three kinds of macros along with their fla- The def macro context provides the opaque type Code vors, which appear on the intersection with other language representing untyped code snippets, exposes the macroAp- features. plication method, which returns the call being expanded, and defines the q string interpolator, which makes it possible 3.1 Def macros to create and pattern match snippets using the convenient The most natural flavor of term macros are def macros, string literal syntax. For example, q"$x + $y" creates a briefly covered in the Section 2. To the programmer, def snippet which represents addition of two arguments speci- macros look like regular Scala methods with an unusual fied by snippets x and y, and val q"$x + $y" = z pattern property– when a method call in a Scala program is resolved matches z as addition and binds x and y to summands. to represent an application of a def macro, that macro defini- Asserts are the canonical example of familiar function- tion is expanded by invoking a corresponding metaprogram, ality, which can be enhanced with def macros. The as- called macro implementation. As a convenience, the macro sert function evaluates the provided boolean expression and engine automatically destructures the method call being ex- raises an error if the result of evaluation is false.
Recommended publications
  • Reflection, Zips, and Generalised Casts Abstract
    Scrap More Boilerplate: Reflection, Zips, and Generalised Casts Ralf L a¨mmel Vrije Universiteit & CWI, Amsterdam Abstract Writing boilerplate code is a royal pain. Generic programming promises to alleviate this pain b y allowing the programmer to write a generic #recipe$ for boilerplate code, and use that recipe in many places. In earlier work we introduced the #Scrap y our boilerplate$ approach to generic p rogramming, which exploits Haskell%s exist- ing type-class mechanism to support generic transformations and queries. This paper completes the picture. We add a few extra #introspec- tive$ or #reflective$ facilities, that together support a r ich variety of serialisation and de-serialisation. We also show how to perform generic #zips$, which at first appear to be somewhat tricky in our framework. Lastly, we generalise the ability to over-ride a generic function with a type-specific one. All of this can be supported in Haskell with independently-useful extensions: higher-rank types and type-safe cast. The GHC imple- mentation of Haskell readily derives the required type classes for user-defined data types. Categories and Subject Descriptors D.2. 13 [Software Engineering]: Reusable Software; D.1.1 [Programming Techniques]: Functional Programming; D.3. 1 [Programming Languages]: Formal Definitions and Theory General Terms Design, Languages Keywords Generic p rogramming, reflection, zippers, type cast 1 Introduction It is common to find that large slabs of a program consist of #boil- erplate$ code, which conceals b y its b ulk a smaller amount of #in- teresting$ code. So-called generic p rogramming techniques allow Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or c ommercial advantage and that copies b ear this notice and the full citation on the first page.
    [Show full text]
  • Template Meta-Programming for Haskell
    Template Meta-programming for Haskell Tim Sheard Simon Peyton Jones OGI School of Science & Engineering Microsoft Research Ltd Oregon Health & Science University [email protected] [email protected] Abstract C++, albeit imperfectly... [It is] obvious to functional programmers what the committee did not realize until We propose a new extension to the purely functional programming later: [C++] templates are a functional language evalu- language Haskell that supports compile-time meta-programming. ated at compile time...” [12]. The purpose of the system is to support the algorithmic construction of programs at compile-time. Robinson’s provocative paper identifies C++ templates as a ma- The ability to generate code at compile time allows the program- jor, albeit accidental, success of the C++ language design. De- mer to implement such features as polytypic programs, macro-like spite the extremely baroque nature of template meta-programming, expansion, user directed optimization (such as inlining), and the templates are used in fascinating ways that extend beyond the generation of supporting data structures and functions from exist- wildest dreams of the language designers [1]. Perhaps surprisingly, ing data structures and functions. in view of the fact that templates are functional programs, func- tional programmers have been slow to capitalize on C++’s success; Our design is being implemented in the Glasgow Haskell Compiler, while there has been a recent flurry of work on run-time meta- ghc. programming, much less has been done on compile-time meta- This version is very slightly modified from the Haskell Workshop programming. The Scheme community is a notable exception, as 2002 publication; a couple of typographical errors are fixed in Fig- we discuss in Section 10.
    [Show full text]
  • Metaprogramming Concepts of Programming Languages
    Metaprogramming Concepts of Programming Languages Alexander Schramm Institut für Softwaretechnik und Programmiersprachen 2. November 2015 A. Schramm 2. November 2015 1/39 Table of Contents Introduction Runtime Reflection in Java Runtime Metaprogramming in Ruby C++ Templates Haskell Templates Lisp Macros Conclusion A. Schramm 2. November 2015 2/39 Outline Introduction Runtime Reflection in Java Runtime Metaprogramming in Ruby C++ Templates Haskell Templates Lisp Macros Conclusion A. Schramm 2. November 2015 3/39 Motivation Which issues do we want to tackle? I Avoid writing boilerplate code I Write code that shows our intention I Expand the syntax of languages I Write type independent code in strongly typed languages A. Schramm 2. November 2015 4/39 What is Metaprogramming Definition: Metaprogramming Metaprograming describes different ways to generate and manipulate code Differentations: I Compile time vs. runtime metaprogramming I Domain language vs. host language A. Schramm 2. November 2015 5/39 Differentations of Metaprograms Compile time Metaprogramming Ways to manipulate or generate code during compilation, e.g: Macros, Templates Runtime Metaprogramming Ways to manipulate or generate code, while it is executed, e.g: dynamic methods, Reflections A. Schramm 2. November 2015 6/39 Differentations of Metaprograms Domain Language The Programming Language, in which the metaprogram is written Host Language The Programming Language of the generated code I Can be different (YACC, Compilers) I Domain language can be a subset of the host language (C++ Templates) I Domain language can be an integral part of the host language (Ruby) A. Schramm 2. November 2015 7/39 Outline Introduction Runtime Reflection in Java Runtime Metaprogramming in Ruby C++ Templates Haskell Templates Lisp Macros Conclusion A.
    [Show full text]
  • Expressive and Strongly Type-Safe Code Generation
    Expressive and Strongly Type-Safe Code Generation Thomas Winant, Jesper Cockx, and Dominique Devriese imec-DistriNet, KU Leuven [email protected] Meta-programs are useful to avoid writing boilerplate code, for polytypic programming, etc. However, when a meta-program passes the type checker, it does not necessarily mean that the programs it generates will be free of type errors, only that generating object programs will proceed without type errors. For instance, this well-typed Template Haskell [5] meta-program generates the ill-typed object program not 'X'. notX :: Q Exp notX = [j not 'X' j] Fortunately, Template Haskell will type-check the generated program after generation, and detect the type error. We call such meta-programming systems weakly type-safe. Even though weakly type-safe meta-programming suffices for guaranteeing that the resulting program is type-safe, it has important downsides. Type errors in the generated code are presented to the application developer, who may simply be a user of the meta-program. If the meta-program is part of a library, the developer has little recourse other than contacting the meta-program authors about the bug in their code. Moreover, composing weakly type-safe meta-programs can be brittle because the type of generated programs is not specified in a machine-checked way. We are interested in what we call strongly type-safe meta-programming, which offers a stronger guarantee: when a meta-program is strongly type-safe, all generated programs are guaranteed to be type-safe too. As such, bugs in meta-programs are detected as early as possible.
    [Show full text]
  • Sample Webpage Using Html
    Sample Webpage Using Html If necrophiliac or protective Paul usually incenses his gibbons jettison shoreward or hedge quirkily and nutritionally, how schizogonous is Skelly? Nealon never calumniates any Beelzebub amounts hydraulically, is Zacherie heterotypic and other enough? Symbolical and rhinocerotic Duffie never slags languishingly when Wendell dunt his etherealisation. Dup is using html documents have no one troubling me how about html tags you can be Example HTML Document. HTML Hyperlink Codes. Trydo Creative Agency and Portfolio Bootstrap Template. How to use a webpage using a particular group multiple selectors. Corporation for html commands and percentage value pairs within the webpage document to push it is to the internet, like the only. Just add a webpage using two things, java servlets and expand your webpages are a blog and its incredible portfolio. Create various simple contact form in HTML and CSS by some our HTML contact form code tutorial However remind you margin to build your contact-us page create a jiffy. W3CSS Templates. 13 cool tricks for HTML Freelancercom. Appco has a good understanding how to amend the first step is all about getting familiar with pictures and implement content or anything else that focuses your studies! Believe it or addict that is all i need to create a basic web page letter of hero it's foreign to. HTML Tags Chart source wwwweb-sourcenet Tag Name Code Example Browser View. Another advantage exactly as a little more time and creativity makes a no one above, sierra provides all the information is it is it. You wonder look its the underlying HTML of any document by clicking on View and then select Source in Netscape You first use thrift to learn a new tags Using.
    [Show full text]
  • Interface Evolution Via Virtual Extension Methods Brian Goetz Fourth Draft, June 2011
    Interface evolution via virtual extension methods Brian Goetz Fourth draft, June 2011 1. Problem statement Once published, it is impossible to add methods to an interface without breaking existing implementations. (Specifically, adding a method to an interface is not a source- compatible change.) The longer the time since a library has been published, the more likely it is that this restriction will cause grief for its maintainers. The addition of closures to the Java language in JDK 7 place additional stress on the aging Collection interfaces; one of the most significant benefits of closures is that it enables the development of more powerful libraries. It would be disappointing to add a language feature that enables better libraries while at the same time not extending the core libraries to take advantage of that feature1. V1 of the Lambda Strawman proposed C#-style static extension methods as a means of creating the illusion of adding methods to existing classes and interfaces, but they have significant limitations – for example, they cannot be overridden by classes that implement the interface being extended, so implementations are stuck with the “one size fits all” implementation provided as an extension2, and they are not reflectively discoverable. 2. Virtual extension methods3 In this document, we outline a mechanism for adding new methods to existing interfaces, which we call virtual extension methods. Existing interfaces can be augmented without compromising backward compatibility4 by adding extension methods to the interface, whose declaration would contain instructions for finding the default implementation in the event that implementers do not provide a method body.
    [Show full text]
  • Identitymodel Documentation
    IdentityModel Documentation Dominick Baier and Brock Allen May 11, 2021 IdentityModel 1 IdentityModel 3 2 IdentityModel.AspNetCore 5 3 IdentityModel.AspNetCore.OAuth2Introspection7 4 IdentityModel.OidcClient 9 5 oidc-client.js 11 5.1 Overview................................................. 11 5.2 Discovery Endpoint........................................... 12 5.3 Token Endpoint.............................................. 14 5.4 Token Introspection Endpoint...................................... 17 5.5 Token Revocation Endpoint....................................... 17 5.6 UserInfo Endpoint............................................ 18 5.7 Dynamic Client Registration....................................... 18 5.8 Device Authorization Endpoint..................................... 19 5.9 Protocol and Claim Type Constants................................... 19 5.10 Creating Request URLs (e.g. for Authorize and EndSession endpoints)................ 20 5.11 Fluent API for the X.509 Certificate Store................................ 22 5.12 Base64 URL Encoding.......................................... 22 5.13 Epoch Time Conversion......................................... 22 5.14 Time-Constant String Comparison.................................... 22 5.15 Overview................................................. 23 5.16 Worker Applications........................................... 23 5.17 Web Applications............................................ 25 5.18 Extensibility............................................... 27 5.19 Overview................................................
    [Show full text]
  • Advanced-Java.Pdf
    Advanced java i Advanced java Advanced java ii Contents 1 How to create and destroy objects 1 1.1 Introduction......................................................1 1.2 Instance Construction.................................................1 1.2.1 Implicit (Generated) Constructor.......................................1 1.2.2 Constructors without Arguments.......................................1 1.2.3 Constructors with Arguments........................................2 1.2.4 Initialization Blocks.............................................2 1.2.5 Construction guarantee............................................3 1.2.6 Visibility...................................................4 1.2.7 Garbage collection..............................................4 1.2.8 Finalizers...................................................5 1.3 Static initialization..................................................5 1.4 Construction Patterns.................................................5 1.4.1 Singleton...................................................6 1.4.2 Utility/Helper Class.............................................7 1.4.3 Factory....................................................7 1.4.4 Dependency Injection............................................8 1.5 Download the Source Code..............................................9 1.6 What’s next......................................................9 2 Using methods common to all objects 10 2.1 Introduction...................................................... 10 2.2 Methods equals and hashCode...........................................
    [Show full text]
  • Appendix a and Appendix B
    This PDF contains 2 Appendices: Appendix A and Appendix B. Appendix A Answers to the Test Your Knowledge Questions This appendix has the answers to the questions in the Test Your Knowledge section at the end of each chapter. Chapter 1 – Hello, C#! Welcome, .NET! 1. Why can a programmer use different languages, for example, C# and F#, to write applications that run on .NET? Answer: Multiple languages are supported on .NET because each one has a compiler that translates the source code into intermediate language (IL) code. This IL code is then compiled to native CPU instructions at runtime by the CLR. 2. What do you type at the prompt to create a console app? Answer: You enter dotnet new console. 3. What do you type at the prompt to build and execute C# source code? Answer: In a folder with a ProjectName.csproj file, you enter dotnet run. 4. What is the Visual Studio Code keyboard shortcut to view Terminal? Answer: Ctrl + ` (back tick). Answers to the Test Your Knowledge Questions 5. Is Visual Studio 2019 better than Visual Studio Code? Answer: No. Each is optimized for different tasks. Visual Studio 2019 is large, heavy- weight, and can create applications with graphical user interfaces, for example, Windows Forms, WPF, UWP, and Xamarin.Forms mobile apps, but it is only available on Windows. Visual Studio Code is smaller, lighter-weight, code-focused, supports many more languages, and is available cross-platform. In 2021, with the release of .NET 6 and .NET Multi-platform App User Interface (MAUI), Visual Studio Code will get an extension that enables building user interfaces for desktop and mobile apps.
    [Show full text]
  • Model Oriented Programming: Bridging the Code-Model Divide
    Model Oriented Programming: Bridging the Code-Model Divide Omar Badreddin Timothy C. Lethbridge School of Electrical Engineering and Computer Science School of Electrical Engineering and Computer Science University of Ottawa University of Ottawa Ottawa, Canada Ottawa, Canada [email protected] [email protected] Abstract—Model Driven Engineering proposes the use of models as the main development artifacts. This methodology II. STATE OF THE ART MODELING PRACTICES involves generating code from models and then perhaps adding Modeling enables developers to work at a higher level of some details to the generated code. Frequently, it is required to abstraction. This view seems to be well accepted in both also reverse-engineer code to generate models. Despite the wide academia and industry. However, the overwhelming majority acceptance of modeling benefits, the use of Model Driven Engineering practices remains limited. We present model of developers still treat source code as their primary work oriented programming as a new paradigm to reduce the ever- medium [2]. Software engineers may draw diagrams when present tension between model-centric and code-centric explaining concepts in design meetings, and include some of development styles. The core of this approach is to add UML them in design documents, but relatively few use models to abstractions such as associations and state machines directly into actually drive development effort. Some communities, like the a high-level programming language code. In this approach, model open source developers, remain entirely code-centric. diagrams become just another abstract view of the code. The need for reverse engineering is eliminated, since everything in the A.
    [Show full text]
  • Nested Class Modularity in Squeak/Smalltalk
    Springer, Nested Class Modularity in Squeak/Smalltalk Nested Class Modularity in Squeak/Smalltalk Modularität mit geschachtelten Klassen in Squeak/Smalltalk by Matthias Springer A thesis submitted to the Hasso Plattner Institute at the University of Potsdam, Germany in partial fulfillment of the requirements for the degree of Master of Science in ITSystems Engineering Supervisor Prof. Dr. Robert Hirschfeld Software Architecture Group Hasso Plattner Institute University of Potsdam, Germany August 17, 2015 Abstract We present the concept, the implementation, and an evaluation of Matriona, a module system for and written in Squeak/Smalltalk. Matriona is inspired by Newspeak and based on class nesting: classes are members of other classes, similarly to class instance variables. Top-level classes (modules) are globals and nested classes can be accessed using message sends to the corresponding enclosing class. Class nesting effec- tively establishes a global and hierarchical namespace, and allows for modular decomposition, resulting in better understandability, if applied properly. Classes can be parameterized, allowing for external configuration of classes, a form of dependency management. Furthermore, parameterized classes go hand in hand with mixin modularity. Mixins are a form of inter-class code reuse and based on single inheritance. We show how Matriona can be used to solve the problem of duplicate classes in different modules, to provide a versioning and dependency management mech- anism, and to improve understandability through hierarchical decomposition. v Zusammenfassung Diese Arbeit beschreibt das Konzept, die Implementierung und die Evaluierung von Matriona, einem Modulsystem für und entwickelt in Squeak/Smalltalk. Ma- triona ist an Newspeak angelehnt und basiert auf geschachtelten Klassen: Klassen, die, wie zum Beispiel auch klassenseitige Instanzvariablen, zu anderen Klassen gehören.
    [Show full text]
  • Minimal Perl for UNIX and Linux People
    Minimal Perl For UNIX and Linux People BY TIM MAHER MANNING Greenwich (74° w. long.) For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact: Special Sales Department Manning Publications Co. Cherokee Station PO Box 20386 Fax: (609) 877-8256 New York, NY 10021 email: [email protected] ©2007 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Manning Publications Co. Copyeditor: Tiffany Taylor 209 Bruce Park Avenue Typesetters: Denis Dalinnik, Dottie Marsico Greenwich, CT 06830 Cover designer: Leslie Haimes ISBN 1-932394-50-8 Printed in the United States of America 12345678910–VHG–1009080706 To Yeshe Dolma Sherpa, whose fortitude, endurance, and many sacrifices made this book possible. To my parents, Gloria Grady Washington and William N. Maher, who indulged my early interests in literature. To my limbic system, with gratitude for all the good times we’ve had together.
    [Show full text]