1 A Programmable Programming Language 2 3 4 MATTHIAS FELLEISEN, ROBERT BRUCE FINDLER, MATTHEW FLATT, SHRIRAM KRISHNAMURTHI, 5 ELI BARZILAY, JAY MCCARTHY, SAM TOBIN-HOCHSTADT 6 7 In addition to libraries and packages, programmers develop and use em- eDSLs. Finally, most available IDEs do not even understand eDSLs bedded problem-specic languages as building blocks of soware systems. 8 or perceive the presence of code wrien in eDSLs. ese languages help developers state solutions for the various aspects of the 9 e goal of the Racket project is to explore this emerging idea of problem in appropriate terms. Complete soware systems compose these 10 partial solutions into a coherent whole. Sadly, this form of work is conducted language-oriented programming (LOP) 11 without real support from the underlying programming language. at two dierent levels. At the practical level, our goal is to build 12 What this emerging development method calls out for, then, are program- a programming language that enables language-oriented soware 13 ming languages that make the creation and use of embedded languages as design. is language must facilitate the easy creation of eDSLs, 14 convenient as the creation of libraries. Implementing this kind of program- the immediate development of components in these newly created ming language has been the goal of our 20-year-old Racket project. is 15 languages, and the integration of components in distinct eDSLs. paper presents the state of the project and sketches how it will proceed. 16 At the conceptual level, the case for LOP is analogous to the 17 CCS Concepts: •So ware and its engineering → Language types; Con- ones for object-oriented programming or for concurrency-oriented 18 text specic languages; Development frameworks and environments; programming [3]. e former arose from making the creation and 19 ACM Reference format: manipulation of objects syntactically simple and dynamically cheap, 20 Mahias Felleisen, Robert Bruce Findler, Mahew Fla, Shriram Krishna- the laer from Erlang’s inexpensive process creation and message 21 murthi, Eli Barzilay, Jay McCarthy, Sam Tobin-Hochstadt. 2017. A Pro- passing. Both of these innovations enabled new ways of developing 22 grammable Programming Language. 0, 0, Article 0 ( 2017), 7 pages. soware and triggered research projects. e question is how we 23 DOI: 10.1145/nnnnnnn.nnnnnnn will realize LOP and how this will aect the world of soware. 24 e decision to develop a new language, Racket, is partly a his- 25 1 PROBLEMS VS PROGRAMMING LANGUAGES torical artifact and partly due to our desire to free ourselves from 26 any unnecessary constraints of industrial mainstream languages 27 In the ideal world, soware developers ought to analyze each prob- lem in the language of its domain and then articulate solutions while we investigate LOP. e next section spells out how Racket 28 got started, how we honed in on LOP, and what this idea implies. 29 in matching terms. ey could thus easily communicate with do- main experts and separate problem-specic ideas from the details of 30 2 THE PRINCIPLES OF RACKET 31 general-purpose languages and specic program design decisions. 32 In reality, however, programmers use a mainstream programming e Racket project dates back to January 1995, when we started 33 language that someone else picked for them. To compensate for this it as a language for experimenting with pedagogic programming 34 conict, they resort to—and on occasion build their own—domain- languages [15]. Working on these languages quickly taught us that 35 specic languages embedded within the chosen language (eDSLs). a language itself is a problem-solving tool. We then soon found 36 For example, JavaScript programmers employ jery for interacting ourselves developing dierent languages for dierent parts of the 37 with the DOM and React for dealing with events and concurrency. project: one (meta-) language for expressing many pedagogic lan- 38 As developers solve their problems in appropriate eDSLs, they com- guages; another one for specializing the DrRacket IDE [15]; and a 39 pose these solutions into one system. In short, developers eectively third for managing congurations. In the end, our soware was a 1 40 write multi-lingual soware in a common host language. multi-lingual system—just as described in the introduction. 41 Sadly, multi-lingual eDSL programming currently rests on an ad Racket’s guiding principle reects the insight we gained: 42 hoc basis and is rather cumbersome. To create and deploy a lan- Empower programmers to create new programming guage, programmers must usually step outside the chosen language 43 Draftlanguages easily and to add them with a friction-free 44 to set up conguration les, run compilation tools, and link in the process to a code base. resulting object-code les. Worse, the host languages fail to sup- 45 With “language” we mean a new syntax, a static semantics, and a port the proper and sound integration of components in dierent 46 dynamic semantics, which usually maps the new syntax to elements 47 1e numerous language-like libraries in scripting languages (JavaScript, Python, of the host language and possibly external languages via an FFI. 48 and Ruby), books such as Fowler and Parson’s [20], and web sites such as Tomass- For a concrete example, take a look at gure 1. It displays a dia- 49 ei’s [tomassetti.me/resources-create-programming-languages/, last visited May 21, 2017] are evidence for the desire of programmers to use and develop eDSLs. gram of the architecture of a recently developed pair of scripting 50 2 Permission to make digital or hard copies of all or part of this work for personal or languages for video editing [2]. eir purpose is to assist peo- 51 classroom use is granted without fee provided that copies are not made or distributed ple who turn recordings of conference presentations into YouTube 52 for prot or commercial advantage and that copies bear this notice and the full citation videos and channels. Most of their work is repetitive—adding pre- 53 on the rst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permied. To copy otherwise, or republish, ludes and postludes, concatenating playlists, and superimposing 54 to post on servers or to redistribute to lists, requires prior specic permission and/or a 55 fee. Request permissions from [email protected]. 2e video language, including an overview of the implementation, is available as a 56 © 2017 ACM. XXXX-XX/2017/0-ART0 $15.00 DOI: 10.1145/nnnnnnn.nnnnnnn use-case artifact at ccs.neu.edu/racket/pubs/#icfp17-acf. 57 58 , Vol. 0, No. 0, Article 0. Publication date: 2017. 0:2 • Felleisen et. al.
1 extends syntax, accommodating even conventional variants such as Algol video typed/video 2 60.4 Like many languages, Racket comes with “baeries included.” 3 builds on builds on Most distinctively, Racket eliminates the hard boundary between 4 library and language, overcoming a seemingly intractable conict. video/ffi turnstile 5 In practice, this means new linguistic constructs are as seamlessly 6 builds on builds on imported as functions and classes from libraries and packages. For 7 for builds on syntax-parse builds on example, Racket’s class system and loops are imports from plain 8 libraries, yet most programmers use these constructs without ever 9 builds on noticing their nature as user-dened concepts. 10 racket Racket’s key innovation is a modular syntax system [17, 26], an 11 improvement of Scheme’s macro system [11, 24, 25], which in turn 12 improves on Lisp’s tree-transformation system. A Racket module Fig. 1. LOP, a small example 13 provides services such as functions, classes, and linguistic con- 14 structs. To implement these services, a module may require the 15 services of other modules. In this world of modules, creating a new 16 audio—and only a small number of steps demand manual interven- language simply means creating a module that provides the services 17 tion. e task calls for a domain-specic scripting language, and for a language. Such a module may subtract linguistic constructs 18 video is a declarative eDSL that meets this need. from a base language, re-interpret some others, and add a few new 19 e typed/video language adds a type system to video. Clearly, ones. A language is rarely built from scratch. 20 the domain of type systems comes with its own language of exper- Like Unix shell scripts, which specify their dialect on the rst 21 tise; typed/video’s implementation therefore uses turnstile [6], line, every Racket module species its language on the rst line, too. 22 an eDSL created for expressing type systems. Similarly, the im- is language specication refers to a le that contains a language- 23 plementation of video’s rendering facility calls for bindings to a dening module. Creating this le is all it takes to install a language. 24 multimedia framework. Ours separates the binding denitions and Practically speaking, a programmer may develop a language in one 25 the repetitive details of FFI calls into two parts: an eDSL for multime- tab of the IDE, while another tab may be a module wrien in the 26 dia FFIs, dubbed video/ffi, and a single program in it. Finally, in language of the rst. Without ever leaving the IDE to run compilers, 27 support of creating all these eDSLs, Racket comes with the syntax- linkers, or any other tools, the developer can modify the language 28 parse eDSL [7], which targets the domain of eDSL creation. implementation in the rst tab and immediately experience this 29 e LOP principle implies two subsidiary guidelines: modication in the second. In short, language development is a 30 friction-free process in Racket. 31 (1) Enable creators of a language to enforce its invariants. In the world of shell scripts, the rst-line convention eventually 32 A programming language is an abstraction, and abstrac- opened the door to a slew of alternatives to shells: Perl, Python, 33 tions are about integrity. Java, for example, comes with Ruby, etc. e Racket world is experiencing a similar phenome- 34 memory safety and type soundness. When a program con- non, with language libraries proliferating within its eco-system: 35 sists of pieces in dierent languages, values ow from one racket/base, the Racket core language; racket, the “baeries in- 36 context into another and need protection from operations cluded” variant; and typed/racket, a typed variant. Some lesser- 37 that might violate their integrity. See section 4. known examples are datalog and a web-server language [27, 30]. 38 (2) Turn extra-linguistic mechanisms into linguistic constructs. When precision is needed, we use the lower-case name of the lan- 39 A LOP programmer who resorts to extra-linguistic mecha- guage in typewriter font; otherwise we use just “Racket.” 40 nisms eectively acknowledges that the chosen language 3 41 lacks expressive power [13]. e numerous external lan- demo 42 guages required to deal with Java projects—a congura- #lang racket/base (provide 43 tion language, a project description language,Draft a makele ;; type MaxPath = [Listof Edge] 44 language—represent symptoms of this problem. We treat such gaps as challenges. See section 5. ;; Natural -> MaxPath 45 walk-simplex) 46 ese principles have been developed in a feedback loop that in- 47 cludes DrRacket [15] plus typed [36], lazy [4], and pedagogical (require "constraints" graph) 48 languages [15]. 49 ;; Natural -> MaxPath 50 3 LIBRARIES AND LANGUAGES RECONCILED (define (walk-simplex timing) ··· (maximizer #:x 2) ··· ) 51 Racket is an heir to Lisp and Scheme. Unlike its ancestors, Racket 52 emphasizes functional over imperative programming without en- Fig. 2. A plain Racket module 53 forcing an ideology. Racket is agnostic when it comes to surface 54 55 4See docs.racket-lang.org/algol60/ [last visited May 24, 2017] as well as 3Like many Programming Language researchers, we subscribe to a weak form of the hashcollision.org/brainfudge 56 [last visited Feb 6, 2017], which shows how Racket Sapir-Whorf hypothesis. copes with obscure syntax. 57 58 , Vol. 0, No. 0, Article 0. Publication date: 2017. A Programmable Programming Language • 0:3
1 Figure 2 is an illustrative module. Its rst line—pronounced “hash 01 #lang racket new-lam 2 lang racket base”—says that it is wrien in racket/base. e mod- 02 3 ule provides a single function, walk-simplex. e accompanying 03 (provide (rename-out [new-lambda lambda])) 4 line comments—introduced via semicolons—informally state a type 04 5 denition and a function signature in terms of this type denition; 05 (require (for-syntax syntax/parse)) 6 gure 5 shows how developers can use typed/racket to replace 06 ... 7 such comments with statically checked types. To implement this 07 ;; Syntax -> Syntax 08 (define-syntax (new-lambda stx) 8 function, the module imports functionality from the "constraints" 09 (syntax-parse stx 9 module in gure 3. e last three lines of gure 2 sketch the deni- 10 [(new-lambda (x:id (˜literal ::) predicate:id) body:expr) 10 tion of the walk-simplex function, which refers to the maximizer 11 (syntax 11 function imported from "constraints". 12 (lambda (x) 12 e "constraints" module in gure 3 expresses the implemen- 13 (unless (predicate x) 13 tation of its only service in a domain-specic language because 14 (define name (object-name predicate)) 14 it deals with simplexes, which are naturally expressed with a sys- 15 (error 'lambda "˜a expected, given: ˜e" name x)) 15 tem of inequalities. e module’s simplex language inherits the 16 body))])) 16 line-comment syntax from racket/base but uses inx syntax other- 17 ... 17 wise. As the comments state, the module exports a single function, 18 maximizer, which consumes two optional keyword parameters. Fig. 4. lambda, re-defined 19 When called as (maximizer #:x n)—as in gure 2—it produces 20 the maximal y value of the system of constraints. As the lower half 21 of gure 3 shows, these constraints are specied with conventional line 09 Like many functional languages, Racket comes with paern- 22 syntax. matching constructs. is one uses syntax-parse from 23 the above-mentioned library. Its rst piece species the 24 #lang simplex constraints to-be-matched tree (stx); the remainder species a series 25 of paern-responses clauses. 26 ;; implicitly provides synthesized function maximizer: line 10 is paern matches any syntax tree whose rst token is 27 ;; #:x Real -> Real new-lambda followed by a parameter specication and a 28 ;; #:y Real -> Real body. e annotation :id demands that the paern vari- 29 ables x and predicate match only identiers in the respec- 30 #:variables x y tive positions. Similarly, :expr allows only expressions to 31 match the body paern variable. 3 * x + 5 * y <= 10 32 line 11 A compile-time function synthesizes new trees with syntax. 3 * x - 5 * y <= 20 33 line 12 e generated syntax tree is a lambda expression. Speci- 34 Fig. 3. A module for describing a simplex shape cally, the function generates an expression that uses lambda. 35 e underline marks its origin as the ambient language: 36 here, racket. 37 In support of this kind of programming, Racket’s modular syn- others Wherever the syntax system encounters the paern vari- 38 tax system benets from several key innovations. A particularly ables x, predicate, and body, it inserts the respective sub- 39 illustrative one is the ability to incrementally re-dene the meaning trees that match x, predicate, and body. 40 of existing language constructs via the module system. is ability When another module uses "new-lam" as its language, the com- 41 allows eDSL creators to ease their users into a new language by piler elaborates the surface syntax into the core language like this: 42 reusing familiar syntax, re-interpreted. (lambda (x :: integer?) (+ x 1)) 43 Take lambda expressions, for example. SupposeDraft a developer –elaborates to−→ (lambda (x :: integer?) (+ x 1)) 44 video wishes to equip a scripting language such as with functions –elaborates to−→ (new-lambda (x :: integer?) (+ x 1)) 45 that check whether their arguments satisfy specied predicates. –elaborates to−→ (lambda (x) 46 Figure 4 shows the basic idea: (unless (integer? x) 47 line 01 e module uses the racket language.
1 optional. Naturally, the compile-time functions could then be modi- import functions and data structures, and wrap these imports in 2 ed to check some or all of these annotations statically, eventually Racket values. Figure 6 illustrates the idea with the sketch of a mod- 3 resulting in a language that resembles typed/racket. ule; video’s initial implementation consisted of just such a set of 4 racket/base bindings to a video-rendering framework. 5 4 SOUND COOPERATION BETWEEN LANGUAGES with ffi/unsafe Of course, when a racket/base module im- 6 A LOP-based soware system consists of multiple, cooperating com- ������ ports the ffi/unsafe library, the language 7 ponents, each wrien in domain-specic languages. Cooperation of the module is unsound. racket/base 8 means that the components exchange values, while “multiple lan- A language developer that starts with the 9 guages” implies that these values are created in distinct languages. ������ unsound eDSL is likely to make it sound as 10 In this seing things can easily go wrong, as gure 5 demonstrates racket/base the second step. To this end, the language 11 with a toy scenario. On the le, a module wrien in typed/racket with contracts is equipped with run-time checks similar to 12 exports a numeric dierentiation function. On the right, a mod- ������ those found in dynamically typed scripting 13 ule wrien in racket imports this function and applies it in three typed/racket languages to prevent the ow of bad values 14 dierent ways, all illegal. If such illegal uses of the function were to unsound primitives. Unfortunately, this 15 to go undiscovered, developers would not be able to rely on type form of protection is ad hoc, and unless de- 16 information for designing functions or for debugging, nor could velopers are hyper-sensitive, the error messages may originate from 17 compilers rely on them for optimizations. In general, cooperating inside the library, blaming some racket/base primitive operation. 18 multi-lingual components must respect the invariants that each To address this form of problem, Racket comes with higher-order 19 participating language establishes. contracts [16]. With those, language developers may uniformly 20 protect the API of a library from bad values. For example, the 21 #lang typed/racket TR #lang racket RR video/ffi language provides language constructs for making the 22 bindings to the video-rendering framework safe. In addition to plain 23 (provide diff) (require "TR.rkt") logical assertions, Racket’s developers are also experimenting with 24 contracts for checking protocols, especially temporal ones [9]. e 25 (: diff ;; scenario 1 built-in blame mechanism of the contract library ensures sound 26 ((Real -> Real) (diff 1) blame assignment [10]. 27 -> ;; scenario 2 Finally, a language developer may wish to check some logical 28 (Real -> Real))) (define (f-bool x) invariants before the programs run. Checking simple types is one (define (diff f) #true) 29 particular example, but checking other forms of types is also possible. (lambda (x) (diff f-bool) 30 e typed/video language illustrates this point with a type system (define lo (f (- x eps))) ;; scenario 3 31 (define hi (f (+ x eps))) (define (f-char x) that checks the input and output types of functions and also numeric 32 (/ (- hi lo) (string x x)) constraints on the integer arguments; as a result, no script can 33 (* 2 eps)))) (diff f-str) possibly render a video of negative length. Similarly, typed/racket 34 is a typed variant of (most of) racket. Fig. 5. Protecting invariants 35 #lang racket/base ffi 36 37 In the real world, programming languages satisfy a spectrum (provide 38 of guarantees about invariants. For example, C++ is unsound.A ;; [Vectorof [Vectorof Real]] -> [Vectorof Real] 39 running C++ program may apply any operation to any bit paern simplex) 40 and, as long as the hardware does not object, the program execution 41 continues. e program may even terminate “normally,” printing (require ffi/unsafe) 42 all kinds of output aer the misinterpretation of the bits. By con- 43 Draft(define (simplex M) trast, Java does not allow the misrepresentation of bits; but it is ··· (ffi-simplex-set ··· ) ··· ) 44 only somewhat more sound than C++ [1]. ML improves on Java 45 again and is completely sound; no value is ever manipulated by any (define lib-simplex (ffi-lib "./coin-Clp/lib/libClp")) 46 inappropriate operation. 47 Racket aims to mirror this spectrum of soundness at two lev- (define ffi-simplex-set 48 els: the level of language implementation itself and the level of (get-ffi-obj "simplex" lib-simplex (_fun _bytes -> _void))) 49 cooperation between two components wrien in dierent embed- Fig. 6. A Racket module using the foreign-function interface 50 ded languages. Consider the soundness of languages rst. As the 51 literature on domain-specic languages suggests [20], these lan- 52 Now consider the soundness of cooperating languages. As be- guages normally evolve in a particular manner, and this is true fore, it is up to the language developer to anticipate how programs 53 for the Racket world, too. A rst implementation is oen a thin 54 in this language interact with others. For example, the creator of veneer over an ecient C-level API. Racket developers create this typed/video 55 provides no protection for its programs. By contrast, kind of veneer with a foreign interface that allows parenthesized the creators of typed/racket intended the language to be used 56 C-level programming [5]. Programmers can refer to a C library, 57 58 , Vol. 0, No. 0, Article 0. Publication date: 2017. A Programmable Programming Language • 0:5
1 in a multi-lingual context. erefore typed/racket compiles the student programs that cannot successfully aack the server, poke 2 types of exported functions into the above-mentioned higher-order around in the le system for solutions, access external connections 3 contracts. When, for example, an exported function must always to steal other students’ solutions, and so on. Naturally, shill’s 4 be applied to integer values, the generated contract inserts a check implementation makes extensive use of Racket’s means for running 5 that ensures the “integerness” of the argument at every application code in sandboxes and harvesting resources via custodians. 6 site for this function; there is no need to insert such a check for 7 the function’s return points, because the function is statically type 6 THE STATE OF AFFAIRS 8 checked. For a function that consumes an integer-valued function, e preceding sections explain how Racket enables programmers 9 the contract must ensure that the function argument always re- 10 turns an integer. In general, a contract wraps exported values with (1) to create languages via linguistic reuse for specic tasks 11 a proxy [31] that controls access to the value. e idea is due to and aspects of a problem; 12 Mahews and Findler [29], while Tobin-Hochstadt and Felleisen’s (2) to equip a language with almost any conventional level 13 Blame eorem [35] shows that if something goes wrong with such of soundness (as found in ordinary language implementa- 14 a mixed system, the run-time exception points to two faulty compo- tions); and 15 nents and their boundary as the provable source of the problem [10]. (3) to exploit a variety of internalized operating-system ser- 16 In general, Racket supplies a range of protection mechanisms, and vices for the construction of run-time libraries for these 17 a language creator can use these mechanisms to implement a range embedded languages. 18 of soundness guarantees for cooperating eDSLs. What makes this form of language-oriented programming work is 19 incrementality. If conventional syntax is not a concern, developers 20 5 UNIVERSALITY VS EXPRESSIVENESS can create new languages from old ones, one construct at a time. 21 Just because a general purpose language can compute all partial- Similarly, they do not have to deliver a sound and secure product 22 recursive functions, programmers cannot necessarily express all all at once. ey can create a new language as a wrapper around an 23 their ideas about programs in this language [13]. is point is best existing C-level library, gradually tease out more of the language 24 illustrated with an example. Imagine the problem of building an IDE from the interface, and make the language as sound or secure as 25 for a new programming language in the very same language. Like time permits or a growing user based demands. 26 any modern IDE, it is supposed to enable users to compile and run Furthermore, the entire process takes place within the Racket 27 their code. If the code goes into an innite loop, the user must be eco-system. A developer creates a language as a Racket module 28 able to terminate it with a mouse click. To implement this capability and installs it by “importing” it into another module. is tight 29 in a natural5 manner, the language must internalize the idea of a coupling has two implications. First, the development tools of the 30 controllable process, a thread. If it does not, the implementor of the eco-system can be used for the creation of language modules and 31 IDE must step outside the language and somehow re-use processes their clients. Second, the language becomes available for creating 32 from the underlying operating system. more languages. Large projects oen employ a tower involving a 33 For a programming language researcher, “stepping outside the few dozen languages—all of which helps manage the huge degree 34 language” signals a failure. Or, as Dan Ingalls [21] phrases it, “[an] of complexity in modern soware systems. 35 operating system is a collection of things that don’t t into a lan- 36 guage[; there] shouldn’t be one.” Hence, we have worked to identify 37 services that Racket borrows from the surrounding operating sys- 38 tem and to assimilate such extra-linguistic mechanisms into the 39 language itself [19]. Here are three sample constructs for which ...... 40 programmers used to step outside of Racket but no longer need to:
41 music-score transitions sandboxes: which restrict access to resources;
42 scene-description inspectors: which control reective capabilities; 43 Draft custodians: which manage resources (e.g., threads, sockets). 44 ������������ ������������ ������������ 45 To understand how the inclusion of such services helps language racket/ffi 46 designers, let us turn to a recent example, the shill language [32]. shill 47 Roughly speaking, is a secure scripting language in Racket’s �������� 48 eco-system. With shill, a developer articulates ne-grained secu- 49 rity and resource policies—say, which les a function may access C 50 or which binaries the script may run—and the language ensures Fig. 7. A sketch of an industrial example 51 that these constraints are satised. To make this concrete, consider a homework server to which students can submit programs. e 52 Sony’s Naughty Dog game studio has created such a large project— 53 instructor might wish to run an auto-grade process for all the sub- missions. Using a shill script, the homework server can execute actually, a framework for creating projects. Roughly speaking, their 54 Racket-based architecture provides languages for describing scenes, 55 5An alternative is to rewrite the entire program before handing it to the given compiler, transitions between scenes, the scores for scenes, and so on. Domain 56 which is exactly what distinguishes “expressiveness” from “universality.” specialists use these languages to describe pieces of a game. e 57 58 , Vol. 0, No. 0, Article 0. Publication date: 2017. 0:6 • Felleisen et. al.
1 Racket implementation composes these domain-specic programs inside the Racket eco-system, their work exposed serious gaps be- 2 and then compiles them into dynamically linked libraries for a C- tween Racket’s principle of language-oriented programming and its 3 based game engine. Figure 7 sketches the arrangement graphically. approach to enforcing security policies. e team had to alter many 4 Racket’s approach to language-oriented programming is by no of Racket’s security mechanisms and invent new ones. Clearly, 5 means perfect. To start with, recognizing when a library should Racket must make this step much easier, meaning more research is 6 become a language requires a tasteful judgment call. e next steps needed to turn security into an integral part of language creation. 7 require good choices in terms of linguistic constructs, syntax, and Finally, LOP poses brand new challenges for tool builders. An 8 run-time primitives. IDE typically provides tools for a single programming language 9 As for concrete syntax, Racket currently has strong support for or a family of related languages, among them debuggers, tracers, 10 the typical, incremental Lisp-style syntax development. It comes and prolers. Good tools communicate with developers on the 11 with merely traditional support for conventional syntax, i.e., gener- terms of the source language. By its very nature, LOP calls for 12 ating lexers and parsers. While traditional parsing introduces the customization of such tools to many languages, their abstractions, 13 above-mentioned natural separation between surface syntax and and their invariants. We have partially succeeded in building a tool 14 meaning, it also means that the development process is no longer for debugging programs in the syntax language [8]; we have the 15 incremental. e proper solution would be to inject Racket ideas foundations of a debugging framework[28]; and we have started to 16 into a context where conventional syntax is the default.6 explore how to infer scoping rule and high-level semantics for newly 17 As for static checking, Racket currently forces language designers introduced, language-level abstractions [33, 34]. Customizing such 18 to develop such checkers wholesale, not on an incremental basis. e tools automatically to newly created (combinations of) languages 19 type checker for typed/racket, for example, looks like the type remains a wide-open challenge. 20 checker for any conventionally typed language; it is a complete In sum, programming language research has stopped short of 21 recursive-descent algorithm that traverses the module’s representa- the ultimate goal, namely, to provide soware developers with 22 tion and algebraically checks types. What Racket developers really tools to formulate solutions in the languages of problem domains. 23 want is a way to aach type checking rules to linguistic constructs, Racket is one aempt to continue the search for proper linguistic 24 so that such algorithms can be synthesized as needed. abstractions. While the project has achieved remarkable success in 25 Chang et al.’srecent work is probably a rst step toward a solution this direction, it also shows that programming language research 26 of this problem [6]. us far, they have demonstrated how their has many problems to solve before the vision of language-oriented 27 approach can equip a DSL with any structural type system in an programming becomes reality. 28 incremental and modular manner. A fully general solution will also 29 have to cope with sub-structural type systems, such as Rust’s, and Acknowledgments. We thank Claire Alvis, Robert Cartwright, 30 static program analyses, such as those found in most compilers. Ryan Culpepper, John Clements, Stephen Chang, Richard Cobbe, 31 As for dynamic checking, Racket suers from two problems. On Greg Cooper, Christos Dimoulas, Bruce Duba, Carl Eastlund, Burke 32 one hand, it provides the building blocks for making language coop- Fetscher, Cormac Flanagan, Kathi Fisler, Dan Friedman, Tony Gar- 33 eration sound, but developers must create the necessary soundness nock Jones, Paul Graunke, Dan Grossman, Kathy Gray, Casey Klein, 34 harnesses on an ad hoc basis. To facilitate the composition of compo- Eugene Kohlbecker, Guillaume Marceau, Jacob Mahews, Sco 35 nents in dierent languages, we need both a theoretical framework Owens, Greg Peyjohn, Jon Raind, Vincent St-Amour, Paul Steck- 36 and abstractions for the partial automation of this task. ler, Stevie Strickland, James Swaine, Asumu Takikawa, Kevin Tew, 37 On the other hand, the available spectrum of soundness mecha- Neil Toronto, and Adam Wick for their contributions. 38 nisms lacks power at both ends, and how to integrate these pow- A preliminary version of this paper appeared in the proceedings of 39 ers seamlessly is unclear. To achieve full control over its context, SNAPL 2015 [14]. In addition to the SNAPL reviewers, Sam Caldwell, 40 Racket probably needs access to assembly languages on all possible Eduardo Cavazos, John Clements, Byron Davies, Ben Greenman, 41 platforms—from hardware to browsers. To realize the full power Greg Hendersho, Manos Renieris, Marc Smith, Vincent St-Amour, 42 of types, typed/racket will have to be equipped with dependent and Asumu Takikawa made several suggestions to improve the 43 types. For example, when a Racket program usesDraft vectors, its corre- presentation of this material. e anonymous CACM reviewers 44 sponding typed variant type-checks what goes into these vectors challenged several pieces of our original submission, which greatly 45 and what comes out, but like ML or Haskell, indexing is le to a improved the exposition. 46 (contractual) check in the run-time system. Tobin-Hochstadt and Over 20 years, this work has been generously supported by our 47 his Typed Racket group are currently working on rst steps in this host institutions (Rice University, University of Utah, Brown Univer- 48 direction, focusing on numeric constraints [23], similar to Xi and sity, University of Chicago, Northeastern University, Northwestern 49 Pfenning’s work [37]. University, Brigham Young University, University of Massachuses 50 As for security, the Racket project calls for a signicant break- Lowell, Indiana University) as well as several funding agencies, foun- 51 through. While the shill team was able to construct the language dations, and companies: AFOSR, CISCO, CORD, Darpa, the Depart- 52 6 ment of Education’s FIPSE program, the ExxonMobil Foundation, 53 Language workbenches, such as Spoofax [22], deal with conventional syntax for DSLs but do not support the incremental modication of existing languages. A recent Microso, the Mozilla Foundation, NSF, and the Texas Advanced 54 report [12] suggests, however, that these tool chains are also converging toward the Technology Program. 55 idea of language creation as language modication. We conjecture that, given sucient time, the development of Racket and language workbenches will arrive at similar 56 designs and the laer will then support DSL integration, too. Soware. Racket is available at racket-lang.org. 57 58 , Vol. 0, No. 0, Article 0. Publication date: 2017. A Programmable Programming Language • 0:7
1 REFERENCES [28] Guillaume Marceau, Gregory H. Cooper, Jonathan P. Spiro, Shriram Krishna- 2 [1] Nada Amin and Ross Tate. Java and Scala’s type systems are unsound: the exis- murthi, and Steven P. Reiss. e design and implementation of a dataow language for scriptable debugging. pages 59–86, September 2007. 3 tential crisis of null pointers. In Object-Oriented Programming Systems, Languages & Applications, pages 838–848, 2016. [29] Jacob Mahews and Robert Bruce Findler. Operational semantics for multi- 4 [2] Leif Andersen, Stephen Chang, and Mahias Felleisen. Super 8 languages for language programs. ACM Transactions on Programming Languages and Systems, 5 making movies. In International Conference on Functional Programming, 2017, to 31(3):1–44, 2009. [30] Jay McCarthy. e two-state solution. In Object-Oriented Programming Systems, 6 appear. [3] Joe Armstrong. Concurrency oriented programming. In Fruhjahrsfachgespr¨ ach¨ der Languages & Applications, pages 567–582, 2010. 7 German Unix User Group, guug.de/veranstaltungen/ffg2003/papers/, last [31] Mark Samuel Miller. Robust Composition: Towards a Unied Approach to Access Control and Concurrency Control 8 visited Feb 6, 2017, 2003. . PhD thesis, Johns Hopkins University, Baltimore, [4] Eli Barzilay and John Clements. Laziness without all the hard work. In Functional Maryland, USA, May 2006. 9 and Declarative Programming in Education, pages 9–13, 2005. [32] Sco Moore, Christos Dimoulas, Dan King, and Stephen Chong. Shill: A secure 10 [5] Eli Barzilay and Dmitry Orlovsky. Foreign interface for PLT Scheme. In Scheme shell scripting language. In Operating Systems Design and Implementation, pages 183–199, 2014. 11 and Functional Programming, pages 63–74, 2004. [6] Stephen Chang, Alex Knauth, and Ben Greenman. Type systems as macros. In [33] Justin Pombrio and Shriram Krishnamurthi. Resugaring: liing evaluation se- 12 Principles of Programming Languages, pages 694–705, 2017. quences through syntactic sugar. In Programming Language Design and Implemen- 13 [7] Ryan Culpepper. Fortifying macros. J. Functional Programming, 22(4–5):439–476, tation, pages 361–371, 2014. [34] Justin Pombrio, Shriram Krishnamurthi, and Mitchell Wand. Inferring scope 14 2012. [8] Ryan Culpepper and Mahias Felleisen. Debugging macros. Science of Computer through syntactic sugar. In International Conference on Functional Programming, 15 Programming, 75(7):496–515, July 2010. 2017, to appear. 16 [9] Christos Dimoulas, Max New, Robert Findler, and Mahias Felleisen. Oh Lord, [35] Sam Tobin-Hochstadt and Mahias Felleisen. Interlanguage migration: from please don’t let contracts be misunderstood. In International Conference on Func- scripts to programs. In Dynamic Language Symposium, pages 964–974, 2006. 17 tional Programming, pages 117–131, 2016. [36] Sam Tobin-Hochstadt and Mahias Felleisen. e design and implementation of 18 [10] Christos Dimoulas, Sam Tobin-Hochstadt, and Mahias Felleisen. Complete mon- Typed Scheme. In Principles of Programming Languages, pages 395–406, 2008. [37] Hongwei Xi and Frank Pfenning. Eliminating array bound checking through 19 itors for behavioral contracts. In European Symposium on Programming, pages 214–233, 2012. dependent types. In Programming Language Design and Implementation, pages 20 [11] R. Kent Dybvig, Robert Hieb, and Carl Bruggeman. Syntactic abstraction in Scheme. 249–257, 1998. 21 Lisp and Symbolic Computation, 5(4):295–326, December 1993. 22 [12] Sebastian Erdweg, Tijs van der Storm, Markus Vlter, Laurence Tra, Remi Bosman, William R. Cook, Albert Gerritsen, Angelo Hulshout, Steven Kelly, Alex Loh, 23 Gabril Konat, Pedro J. Molina, Martin Palatnik, Risto Pohjonen, Eugen Schindler, 24 Klemens Schindler, Riccardo Solmi, Vlad Vergu, Eelco Visser, Kevin van der Vlist, Guido Wachsmuth, and Jimi van der Woning. Evaluating and comparing language 25 workbenches: Existing results and benchmarks for the future. Computer Languages, 26 Systems and Structures, 44, Part A:24 – 47, 2015. 27 [13] Mahias Felleisen. On the expressive power of programming languages. Science of Computer Programming, 17:35–75, 1991. 28 [14] Mahias Felleisen, Robert Bruce Findler, Mahew Fla, Shriram Krishnamurthi, 29 Eli Barzilay, Jay McCarthy, and Sam Tobin-Hochstadt. e Racket Manifesto. 30 In omas Ball, Rastislav Bodik, Shriram Krishnamurthi, Benjamin S. Lerner, and Greg Morrise, editors, First Summit on Advances in Programming Languages 31 (SNAPL 2015), volume 32 of Leibniz International Proceedings in Informatics (LIPIcs), 32 pages 113–128, 2015. [15] Robert Bruce Findler, John Clements, Cormac Flanagan, Mahew Fla, Shriram 33 Krishnamurthi, Paul Steckler, and Mahias Felleisen. DrScheme: A programming 34 environment for Scheme. J. Functional Programming, 12(2):159–182, 2002. 35 [16] Robert Bruce Findler and Mahias Felleisen. Contracts for higher-order functions. In International Conference on Functional Programming, pages 48–59, 2002. 36 [17] Mahew Fla. Composable and compilable macros: You want it when? In Interna- 37 tional Conference on Functional Programming, pages 72–83, 2002. 38 [18] Mahew Fla. Bindings as sets of scopes. In Principles of Programming Languages, pages 705–717, 2016. 39 [19] Mahew Fla, Robert Bruce Findler, Shriram Krishnamurthi, and Mahias 40 Felleisen. Programming languages as operating systems (or revenge of the son of the Lisp machine). In International Conference on Functional Programming, pages 41 138–147, 1999. 42 [20] Martin Fowler and Rebecca Parsons. Domain-specic Languages. Addison-Wesley, 43 2010. Draft [21] Dan H. Ingalls. Design principles behind smalltalk. Byte Magazine, 6:286–298, 44 1981. 45 [22] Lennart C.L. Kats and Eelco Visser. e Spoofax language workbench. In Object- 46 Oriented Programming Systems, Languages & Applications, pages 444–463, 2010. [23] Andrew M. Kent, David Kempe, and Sam Tobin-Hochstadt. Occurrence typing 47 modulo theories. In Programming Language Design and Implementation, pages 48 296–309, 2016. [24] Eugene E. Kohlbecker, Daniel P. Friedman, Mahias Felleisen, and Bruce F. Duba. 49 Hygienic macro expansion. In Lisp and Functional Programming, pages 151–161, 50 1986. 51 [25] Eugene E. Kohlbecker and Mitchell Wand. Macros-by-example: Deriving syntactic transformations from their specications. In Principles of Programming Languages, 52 pages 77–84, 1987. 53 [26] Shriram Krishnamurthi. Linguistic Reuse. PhD thesis, Rice University, 2001. 54 [27] Shriram Krishnamurthi, Peter Walton Hopkins, Jay McCarthy, Paul T. Graunke, Greg Peyjohn, and Mahias Felleisen. Implementation and use of the PLT Scheme 55 web server. Higher-Order and Symbolic Computation, 20(4):431–460, 2007. 56 57 58 , Vol. 0, No. 0, Article 0. Publication date: 2017.