1 A Programmable Programming Language 2 3 4 MATTHIAS FELLEISEN, ROBERT BRUCE FINDLER, MATTHEW FLATT, SHRIRAM KRISHNAMURTHI, 5 ELI BARZILAY, JAY MCCARTHY, SAM TOBIN-HOCHSTADT 6 7 In addition to libraries and packages, programmers develop and use em- eDSLs. Finally, most available IDEs do not even understand eDSLs bedded problem-specic languages as building blocks of soware systems. 8 or perceive the presence of code wrien in eDSLs. ese languages help developers state solutions for the various aspects of the 9 e goal of the Racket project is to explore this emerging idea of problem in appropriate terms. Complete soware systems compose these 10 partial solutions into a coherent whole. Sadly, this form of work is conducted language-oriented programming (LOP) 11 without real support from the underlying programming language. at two dierent levels. At the practical level, our goal is to build 12 What this emerging development method calls out for, then, are program- a programming language that enables language-oriented soware 13 ming languages that make the creation and use of embedded languages as design. is language must facilitate the easy creation of eDSLs, 14 convenient as the creation of libraries. Implementing this kind of program- the immediate development of components in these newly created ming language has been the goal of our 20-year-old Racket project. is 15 languages, and the integration of components in distinct eDSLs. paper presents the state of the project and sketches how it will proceed. 16 At the conceptual level, the case for LOP is analogous to the 17 CCS Concepts: •So ware and its engineering ! Language types; Con- ones for object-oriented programming or for concurrency-oriented 18 text specic languages; Development frameworks and environments; programming [3]. e former arose from making the creation and 19 ACM Reference format: manipulation of objects syntactically simple and dynamically cheap, 20 Mahias Felleisen, Robert Bruce Findler, Mahew Fla, Shriram Krishna- the laer from Erlang’s inexpensive process creation and message 21 murthi, Eli Barzilay, Jay McCarthy, Sam Tobin-Hochstadt. 2017. A Pro- passing. Both of these innovations enabled new ways of developing 22 grammable Programming Language. 0, 0, Article 0 ( 2017), 7 pages. soware and triggered research projects. e question is how we 23 DOI: 10.1145/nnnnnnn.nnnnnnn will realize LOP and how this will aect the world of soware. 24 e decision to develop a new language, Racket, is partly a his- 25 1 PROBLEMS VS PROGRAMMING LANGUAGES torical artifact and partly due to our desire to free ourselves from 26 any unnecessary constraints of industrial mainstream languages 27 In the ideal world, soware developers ought to analyze each prob- lem in the language of its domain and then articulate solutions while we investigate LOP. e next section spells out how Racket 28 got started, how we honed in on LOP, and what this idea implies. 29 in matching terms. ey could thus easily communicate with do- main experts and separate problem-specic ideas from the details of 30 2 THE PRINCIPLES OF RACKET 31 general-purpose languages and specic program design decisions. 32 In reality, however, programmers use a mainstream programming e Racket project dates back to January 1995, when we started 33 language that someone else picked for them. To compensate for this it as a language for experimenting with pedagogic programming 34 conict, they resort to—and on occasion build their own—domain- languages [15]. Working on these languages quickly taught us that 35 specic languages embedded within the chosen language (eDSLs). a language itself is a problem-solving tool. We then soon found 36 For example, JavaScript programmers employ jery for interacting ourselves developing dierent languages for dierent parts of the 37 with the DOM and React for dealing with events and concurrency. project: one (meta-) language for expressing many pedagogic lan- 38 As developers solve their problems in appropriate eDSLs, they com- guages; another one for specializing the DrRacket IDE [15]; and a 39 pose these solutions into one system. In short, developers eectively third for managing congurations. In the end, our soware was a 1 40 write multi-lingual soware in a common host language. multi-lingual system—just as described in the introduction. 41 Sadly, multi-lingual eDSL programming currently rests on an ad Racket’s guiding principle reects the insight we gained: 42 hoc basis and is rather cumbersome. To create and deploy a lan- Empower programmers to create new programming guage, programmers must usually step outside the chosen language 43 Draftlanguages easily and to add them with a friction-free 44 to set up conguration les, run compilation tools, and link in the process to a code base. resulting object-code les. Worse, the host languages fail to sup- 45 With “language” we mean a new syntax, a static semantics, and a port the proper and sound integration of components in dierent 46 dynamic semantics, which usually maps the new syntax to elements 47 1e numerous language-like libraries in scripting languages (JavaScript, Python, of the host language and possibly external languages via an FFI. 48 and Ruby), books such as Fowler and Parson’s [20], and web sites such as Tomass- For a concrete example, take a look at gure 1. It displays a dia- 49 ei’s [tomassetti.me/resources-create-programming-languages/, last visited May 21, 2017] are evidence for the desire of programmers to use and develop eDSLs. gram of the architecture of a recently developed pair of scripting 50 2 Permission to make digital or hard copies of all or part of this work for personal or languages for video editing [2]. eir purpose is to assist peo- 51 classroom use is granted without fee provided that copies are not made or distributed ple who turn recordings of conference presentations into YouTube 52 for prot or commercial advantage and that copies bear this notice and the full citation videos and channels. Most of their work is repetitive—adding pre- 53 on the rst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permied. To copy otherwise, or republish, ludes and postludes, concatenating playlists, and superimposing 54 to post on servers or to redistribute to lists, requires prior specic permission and/or a 55 fee. Request permissions from [email protected]. 2e video language, including an overview of the implementation, is available as a 56 © 2017 ACM. XXXX-XX/2017/0-ART0 $15.00 DOI: 10.1145/nnnnnnn.nnnnnnn use-case artifact at ccs.neu.edu/racket/pubs/#icfp17-acf. 57 58 , Vol. 0, No. 0, Article 0. Publication date: 2017. 0:2 • Felleisen et. al. 1 extends syntax, accommodating even conventional variants such as Algol video typed/video 2 60.4 Like many languages, Racket comes with “baeries included.” 3 builds on builds on Most distinctively, Racket eliminates the hard boundary between 4 library and language, overcoming a seemingly intractable conict. video/ffi turnstile 5 In practice, this means new linguistic constructs are as seamlessly 6 builds on builds on imported as functions and classes from libraries and packages. For 7 for builds on syntax-parse builds on example, Racket’s class system and loops are imports from plain 8 libraries, yet most programmers use these constructs without ever 9 builds on noticing their nature as user-dened concepts. 10 racket Racket’s key innovation is a modular syntax system [17, 26], an 11 improvement of Scheme’s macro system [11, 24, 25], which in turn 12 improves on Lisp’s tree-transformation system. A Racket module Fig. 1. LOP, a small example 13 provides services such as functions, classes, and linguistic con- 14 structs. To implement these services, a module may require the 15 services of other modules. In this world of modules, creating a new 16 audio—and only a small number of steps demand manual interven- language simply means creating a module that provides the services 17 tion. e task calls for a domain-specic scripting language, and for a language. Such a module may subtract linguistic constructs 18 video is a declarative eDSL that meets this need. from a base language, re-interpret some others, and add a few new 19 e typed/video language adds a type system to video. Clearly, ones. A language is rarely built from scratch. 20 the domain of type systems comes with its own language of exper- Like Unix shell scripts, which specify their dialect on the rst 21 tise; typed/video’s implementation therefore uses turnstile [6], line, every Racket module species its language on the rst line, too. 22 an eDSL created for expressing type systems. Similarly, the im- is language specication refers to a le that contains a language- 23 plementation of video’s rendering facility calls for bindings to a dening module. Creating this le is all it takes to install a language. 24 multimedia framework. Ours separates the binding denitions and Practically speaking, a programmer may develop a language in one 25 the repetitive details of FFI calls into two parts: an eDSL for multime- tab of the IDE, while another tab may be a module wrien in the 26 dia FFIs, dubbed video/ffi, and a single program in it. Finally, in language of the rst. Without ever leaving the IDE to run compilers, 27 support of creating all these eDSLs, Racket comes with the syntax- linkers, or any other tools, the developer can modify the language 28 parse eDSL [7], which targets the domain of eDSL creation. implementation in the rst tab and immediately experience this 29 e LOP principle implies two subsidiary guidelines: modication in the second.