The Compiler Forest omitted for submission Abstract gram transformation rules, and runtime operations invoked by the Compilers targeting complex execution environments, such as compiler at the cluster, machine, and core levels are very different. computer clusters composed of machines with multi-core CPUs However, the DryadLINQ compiler has a monolithic implementa- and GPUs, are difficult to write. To address this problem, we intro- tion, including several other modules besides the three mentioned duce partial compilers, which can pass subtasks to child compilers above. Maintaining and updating the DryadLINQ codebase is quite and combine the various plans they create, as a generalization of onerous; its monolithic nature makes it brittle to changes and hard traditional compilers. We define a set of high-level polymorphic to understand. Our goal is to refactor the compiler into a hierarchy operations that manipulate partial compilers as first-class values. of completely independent compilers that cooperate to implement These mechanisms provide a software architecture for modular a single translation. compiler construction, which allows the building of a forest of Our main contribution is a novel software architecture con- compilers. We explore the mathematical structure of partial com- structed using a standard type-theoretical interface for building co- pilers and its ties to theorem proving and category theory. Moti- operating compilers. We present in Section2 the notion of a partial vated by the problem of distributed computation, we demonstrate compiler; this is a compiler that needs “help” from other compilers the software engineering advantages of our approach by building to produce a complete result. The resulting composite compilers some complex compilers. We describe the complete implementa- form our compiler forests. Formally, one uses polymorphic compo- tion of a large-scale query-processing system written as a modular sition operations on compilers and partial compilers. combination of many simple partial compilers. Mathematically, partial compilers and their composition, as well as our other polymorphic operations, can be placed on a solid foun- dation using tools from category theory, particularly categorical 1 Introduction logic, described in Section6. The categorical formalism suggests Today’s computers are routinely composed of multiple compu- a set of powerful polymorphic operators that can be used to manip- tational units: multi-core processors, hyperthreaded processors, ulate partial compilers as first-class objects, particularly the com- graphics processors, and multi-processors; we will use the term position and tensor operations given in Section2, as well as oth- “execution engine” for these computational resources. Writing ers, given in Section3. These operations can be seen as a form of software that effectively exploits a complex hierarchy of execu- “structured programming” manipulating compilers as values. tion engines is a daunting task. Based on this small set of well-defined operations, we obtain The work presented in this paper was motivated by the Dryad- a theory of correctness of composite compilers. This is described LINQ compiler [33] and its underlying distributed runtime system in Section4; it is a Hoare-type logic for reasoning about partially- Dryad [16]. DryadLINQ translates programs written in the LINQ correct partial compilers and their compositions. programming language (Language INtegrated Query) [20] into The theoretical foundations we establish have immediate appli- large-scale distributed computations that run on shared-nothing cations in practice. We have used our approach successfully for computer clusters, using multiple cores for each machine. writing compilers that target complex execution engines (such as The basic functionality provided by DryadLINQ is to compile computer clusters). A compiler built as a tree of partial compilers programs for execution on a computer cluster. A cluster can be seen has a modular architecture (to some degree induced by the struc- as a big execution engine composed of many computers that do ture of the execution engine) and is easily tested and extended. the actual work. At a finer grain each computer may have multiple To demonstrate this, we revisit the original problem of compil- CPU cores. Logically, the core DryadLINQ compilation process is ing LINQ for computer clusters in Section5. In order to expose correspondingly structured as a three-stage process: (1) translating the fundamental ideas without undue detail, we consider only a re- a cluster-level computation into a set of interacting machine-level duced and stylized version of LINQ called µLINQ. This language computations, (2) translating each machine-level computation into is rich enough to express many interesting computations, includ- a set of CPU core-level computations, and (3) implementing each ing the popular MapReduce [8] computation model. In the text we core-level computation in terms of machine code (via .Net). In the build a fully functional compiler for µLINQ that executes programs case of DryadLINQ, the source language for all these layers is es- on a computer cluster with multi-core machines. We use our formal sentially the same, viz LINQ, but the optimization strategies, pro- machinery to argue about the correctness of the µLINQ compiler. We have validated this architecture by two preliminary compiler implementations: a (simplified) implementation of DryadLINQ, and a large-scale matrix computations compiler, both described briefly in Section 5.4 (a detailed description is reserved for a sepa- rate publication). We conjecture that partial compilers will prove to be a useful concept even for structuring traditional compilers, since the formalism of partial compilers does not depend on the existence of a complex composite execution engine. [Copyright notice will appear here once ’preprint’ option is removed.] 1 2011/11/6 2.1 Definition We now formalize all these intuitions with a typed lambda calculus. query plan We do not spell the calculus out in detail, but we make use of product and function types, labeled sum types, and list types, as Partial well as base types. Operationally, we generally assume there are G compiler no side effects, that types denote sets, and that functions are the query plan R generate reduce usual set-theoretic ones. If we discuss side-effects then we assume a call-by-value monadic semantics along the lines of that of Moggi’s sub-query sub-plan computational lambda calculus [2, 23]. Compiler Compilers C transform queries into plans so they are typed as: Child C : query −! plan Compiler as pictured on the left of Figure1. All computations involved act on data, which we take to be values of a given type “data”. In order to execute plans on specified input data we assume available a suitable “run” operation: Figure 1. Left: A compiler translates queries to plans. Right:A partial compiler is unable to completely compile the input query; it run : plan × data −! data reduces the query to a sub-query to be handled by a child compiler. Given a plan for the sub-query, the partial compiler then generates In order to avoid formalizing details of the runtime used for execut- a plan for the entire query. ing plans, we often model plans as computations: plan = data ! data and then run is just function application: run(p; d) = p(d). We do Our type-theoretic approach was originally inspired by Milner’s not specify the relationship between query and plan; in particular, tactics, which are used in theorem proving and proof assistants the plan type of some compiler may be the same or a subset of the (Section 7.1). We were then influenced by the categorical ideas query type of some other compiler. that inspired the precise typing of our partial compilers, as well, as Our formalism models partial compilers as two separate pieces noted above, as the operations we use over them. It is fascinating that perform, respectively, query reduction and plan generation to note that, in their turn, the categorical ideas derive from Godel’s¨ (see Figure1, right). We begin by considering unary partial com- 1958 Dialectica interpretation of intuitionistic logic [1]. Since that pilers, which need just one child compiler. Given an input query, interpretation can be viewed as a form of intuitionistic logical con- such a partial compiler returns a query: it “reduces” the original sequence that may not be ultimately surprising. Many prior prac- query to a “simpler” (sub-)query, to be fed to its child. This is done tical systems have been organized by using cooperating compilers by a query reduction function (Section 7.2). Our formalism provides us with much-needed tools R:query−!query0 to allow rigorous reasoning about such systems. Our theory and practice evolved concurrently: the practice de- Note that the sub-query may be written in a different language than fined and constrained the problems to be solved while the theory the original query. This extra generality is natural, since the sub- enabled us to refactor our implementations to correspond to the query may be targeted to a different engine than the query. emerging formalism more directly and to generalize the scope of The partial compiler also contains a plan generation function our abstractions. This iterative process is not yet complete. This pa- G :query × plan0 −!plan per is only a first step: much theoretical and practical work remains to be done (Section8). that, given the original query together with a (sub-)plan for the sub- query, generates a plan for the original query. Much as before, the sub-plan may be of a different type than the original plan. A partial 2 Compilers and partial compilers compiler PC is therefore typed as: Some of our terminology is inherited from LINQ, which in turn PC : (query!query0) × (query × plan0 !plan) inherited terminology from databases. We call the programs fed as inputs to a compiler “queries” and the outputs generated by the To make formulas more readable we employ syntactic sugar for compiler “plans” (as shown on the left of Figure1).