Unification of Compile-Time and Runtime Metaprogramming in Scala
Total Page:16
File Type:pdf, Size:1020Kb
Unification of Compile-Time and Runtime Metaprogramming in Scala THÈSE NO 7159 (2017) PRÉSENTÉE LE 6 MARS 2017 À LA FACULTÉ INFORMATIQUE ET COMMUNICATIONS LABORATOIRE DE MÉTHODES DE PROGRAMMATION 1 PROGRAMME DOCTORAL EN INFORMATIQUE ET COMMUNICATIONS ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE POUR L'OBTENTION DU GRADE DE DOCTEUR ÈS SCIENCES PAR Eugene BURMAKO acceptée sur proposition du jury: Prof. J. R. Larus, président du jury Prof. M. Odersky, directeur de thèse Dr D. Syme, rapporteur Prof. S. Tobin-Hochstadt, rapporteur Prof. V. Kuncak, rapporteur Suisse 2017 To future travels. Acknowledgements I would like to heartily thank my advisor Martin Odersky for believing in my vision and trusting me with the autonomy to explore its avenues. It was astonishing to see my work shipped as part of a programming language used by hundreds of thousands of developers, and it would not have been possible without Martin’s encouragement and support. It is my pleasure to thank the members of my thesis jury, Viktor Kunčak, James Larus, Don Syme and Sam Tobin-Hochstadt for their time and insightful comments. I am grateful to my colleagues at EPFL and students whom I supervised during my PhD studies. We hacked many fun things and shared many good times together. My special thanks goes to Denys Shabalin whose relentless passion for excellence shaped our numerous joint projects. A lot of the ideas described in this dissertation were born from discussions and collaborations with Denys. I am continuously humbled by the privilege of being part of the Scala community. During my time at EPFL, I had the chance to work together with Scala enthusiasts all over the world, and that was an amazing experience. Were I to undertake my projects alone, some of the most interesting discoveries would most likely not have happened at all. I did my best to give credit for the most important contributions in the individual chapters of the dissertation. Finally, I would like to thank my family - Nikolai, Elena, Julia, Andrey and cute little Anna. Whatever was going on at work, however much I was hacking during our rare get-togethers, I always knew that I had their love and support. 27 October 2016 Eugene Burmako i Abstract Metaprogramming is a technique that consists in writing programs that treat other programs as data. This paradigm of software development contributes to a multitude of approaches that improve programmer productivity, including code generation, program analysis and domain-specific languages. Many programming languages and runtime systems provide support for metaprogramming. Programming platforms often distinguish the notions of compile-time and runtime metaprogramming, depending on the phase of the program lifecycle when metaprograms execute. It is common for different lifecycle phases to be hosted in different environ- ments, so it is also common for different kinds of metaprogramming to provide different capabilities to metaprogrammers. In this dissertation, we present an exploration of the idea of unifying compile-time and runtime metaprogramming in Scala. We focus on the practical aspect of the exploration; most of the described designs are available as popular software products, and some of them have become part of the standard distribution of Scala. First, guided by the motivation to consolidate disparate metaprogramming techniques available in earlier versions of Scala, we introduce scala.reflect, a unified metaprogram- ming framework that uses a language model derived from the Scala compiler to run metaprograms both at compile time and at runtime. Secondly, armed by the newfound metaprogramming powers, we describe Scala macros, a language-integrated compile-time metaprogramming facility based on scala.reflect. Thanks to the comprehensive nature of scala.reflect, macros are able to work with both syntactic and semantic information about Scala programs, enabling a wide range of previously impractical or impossible use cases. Finally, based on our experience and user feedback, we identify key strengths and weaknesses of scala.reflect and macros. We propose scala.meta, a new unified metapro- gramming framework, and inline/meta, a new macro system based on scala.meta, that take the best from their predecessors and address the most important problems. Keywords: Programming Languages, Compilers, Metaprogramming, Reflection, Macros. iii Astratto Metaprogrammazione è la tecnica che consiste nel creare programmi che trattano gli altri programmi come dati. Questo paradigma di sviluppo software contribuisce alla multitudine di approcci che migliorano la produttività di programmatore, compresi la generazione di codici, i procedimenti automatizzati di analisi del software, e i linguaggi specifici di dominio. Molti linguaggi di programmazione e sistemi a tempo di esecuzione consentono la metaprogrammazione. Le piattaforme software spesso fanno distinzione tra le nozioni di metaprogrammazione a tempo di compilazione e a tempo di esecuzione, a seconda della fase del ciclo di vita del software quando sono eseguiti i metaprogrammi. È normale che cicli di vita diversi siano presentati in ambienti diversi, ed è altresì normale che diversi tipi di metaprogrammazione forniscano capacitá diverse ai metaprogrammatori. In questa dissertazione presentiamo la ricerca dell’idea di unire la metaprogrammazione a tempo di compilazione e a tempo di esecuzione in Scala. Particolare risalto viene dato all’aspetto pratico della ricerca: la maggior parte dei descritti design sono disponibili come popolari prodotti software, ed alcuni di loro sono parte integrante della distribuzione standard di Scala. Innanzitutto, guidati dalla motivazione di consolidare le tecniche disparate di metapro- grammazione disponibili in versioni precedenti di Scala, introduciamo scala.reflect, un framework unificato di metaprogrammazione che usa il modello di linguaggio derivato dal compilatore Scala per eseguire i metaprogrammi sia a tempo di compilazione, sia a tempo di esecuzione. Secondariamente, muniti delle nuove capacità fornite dalla metaprogrammazione, de- scriviamo le macroistruzioni, un’infrastruttura di metaprogrammazione a tempo di compilazione integrata al linguaggio, basata su scala.reflect. Grazie alla natura esau- stiva di scala.reflect, le macroistruzioni possono adoperare sia informazioni sintattiche, sia semantiche sui programmi Scala, consentendo una vasta gamma di casi d’uso che precedentemente erano impraticabili o impossibili. Infine, basati sulla nostra esperienza e commenti degli utenti, abbiamo identificato i punti di forza e deboli di scala.reflect e macros. Proponiamo scala.meta, un nuovo framework v Astratto unificato di metaprogrammazione, e inline/meta, un nuovo sistema di macroistruzioni basato su scala.meta, che unisce gli elementi migliori dei suoi predecessori, risolvendone allo stesso tempo le pecche più evidenti. Parole chiave: linguaggi di programmazione, compilatori, metaprogrammazione, rifles- sione, macroistruzione vi Contents Acknowledgementsi Abstract (English/Italiano) iii Table of contentsx 1 Introduction1 1.1 Background...................................1 1.2 Target audience.................................2 1.3 Preliminaries..................................2 1.4 Terminology...................................3 1.5 Thesis statement................................4 1.6 Contributions..................................5 I Unification7 2 Ad hoc metaprogramming9 2.1 JVM reflection.................................9 2.2 Scala signatures................................. 12 2.3 On-demand reification............................. 14 2.4 Compiler plugins................................ 16 2.5 Conclusion................................... 18 3 Scala.reflect 19 3.1 Intuition..................................... 20 3.1.1 High-level architecture......................... 20 3.1.2 Example: the isImmutable metaprogram............... 26 3.1.3 Compile-time execution........................ 29 3.1.4 Runtime execution........................... 31 3.1.5 Sharing metaprograms......................... 34 3.2 Language model................................ 37 3.2.1 Overview................................ 38 3.2.2 Trees................................... 41 vii Contents 3.2.3 Symbols................................. 43 3.2.4 Types.................................. 48 3.3 Notations.................................... 50 3.3.1 Reify................................... 51 3.3.2 Quasiquotes............................... 55 3.3.3 TypeTags................................ 58 3.3.4 Hygiene................................. 60 3.4 Environments.................................. 62 3.4.1 Scala compiler............................. 63 3.4.2 Dotty.................................. 64 3.4.3 IDEs................................... 64 3.4.4 JVM................................... 65 3.4.5 JavaScript................................ 66 3.4.6 Native.................................. 67 3.5 Operations................................... 67 3.6 Conclusion................................... 72 II Scala macros 75 4 Def macros 77 4.1 Motivation................................... 78 4.2 Intuition..................................... 79 4.3 Macro defs and macro impls.......................... 84 4.3.1 Term parameters............................ 85 4.3.2 Type parameters............................ 88 4.3.3 Separate compilation.......................... 89 4.3.4 Split or merge?............................. 90 4.4 Macro expansion................................ 92 4.4.1 Expansion pipeline........................... 93 4.4.2 Effects on