Portland State University PDXScholar Computer Science Faculty Publications and Presentations Computer Science 2015 The Expression Problem, Gracefully Andrew P. Black Portland State University, [email protected] Follow this and additional works at: https://pdxscholar.library.pdx.edu/compsci_fac Part of the Programming Languages and Compilers Commons Let us know how access to this document benefits ou.y Citation Details Black, Andrew P., "The Expression Problem, Gracefully" (2015). Computer Science Faculty Publications and Presentations. 139. https://pdxscholar.library.pdx.edu/compsci_fac/139 This Post-Print is brought to you for free and open access. It has been accepted for inclusion in Computer Science Faculty Publications and Presentations by an authorized administrator of PDXScholar. Please contact us if we can make this document more accessible: [email protected]. Andrew P. Black — The Expression Problem, Gracefully The Expression Problem, Gracefully Andrew P. Black Portland State University [email protected] Abstract cult to solve: the problem is over-constrained, in the sense The “Expression Problem” was brought to prominence that the problem statement itself rules out any possibility by Wadler in 1998. It is widely regarded as illustrating of a perfect solution. The various published solutions fill a that the two mainstream approaches to data abstraction — three-dimensional space. Two dimensions are frequently dis- procedural abstraction and type abstraction — are comple- cussed and compared. The first is the set of features offered mentary, with the strengths of one being the weaknesses of by the implementation language (classes, class extensions, the other. Despite an extensive literature, the origin of the generic types, algebraic data, etc.); the second is the pattern problem remains ill-understood. I show that the core prob- employed by the programmer (abstract factory, visitor, ob- lem is in fact the use of global constants, and demonstrate ject algebra, etc.). The third dimension, in contrast, is hardly that an important aspect of the problem goes away when mentioned: which of the constraints the authors choose to Java is replaced by a language like Grace, which eliminates loosen to obtain a solution. To understand what these con- them. straints are, we need to examine the problem in more detail. Keywords data abstraction, algebraic data types, rows and 2. The Problem columns, procedural abstraction, objects, expression prob- As described by Krishnamurthi, Felleisen, and Friedman [5], lem, extensibility. the expression problem considers a representation of some 1. Introduction composite heterogeneous recursive structure, such as the ex- pression tree of a programming language. There are opera- The Expression problem was given that name by Wadler tions (also called “tools”) on those structural elements, for in 1998 [9], and demonstrates the importance of a catchy example, code generation, or pretty-printing. We want to be name in securing the immortality of publication. Wadler did able to extend this structure in two orthogonal dimensions: not claim to have invented the problem; on the contrary, – the data dimension, in which we might add new variants he pointed out that it had already been widely discussed, to the composite data, for example, representations of new notably by Krishnamurthi, Felleisen, and Friedman in an programming language statements, and ECOOP paper published earlier that year [5]. Oliveira and – the operation dimension, in which we might add new op- Cook’s later ECOOP paper [6] opens by declaring that erations, such as interpretation. The “expression problem” (EP) [4, 7, 9] is now a Krishnamurthi, Felleisen, and Friedman’s original constraint classical problem in programming languages. It refers was that “ideally, these extensions should not require any to the difficulty of writing data abstractions that can changes to existing code”. Wadler added the constraint be easily extended with both new operations and new of static type safety (e.g., no casts), and strengthened no data variants. changes to existing code to no recompilation of existing It does not take much study of the voluminous literature on code. Torgersen, in his comprehensive survey of the problem the expression problem to realize why it has proven so diffi- and its solution using generics [8], observes that “existing code” includes not only the implementation of the expres- sion tree, but also the creation code — the code responsible for creating instances of the structure — and the client code, which sends messages to those instances. 3. The Solutions In a paper of this length, we cannot even survey the extensive literature on solving the expression problem. Instead, we [Copyright notice will appear here once ’preprint’ option is removed.] will examine four interesting points in the solution space. To be presented at MASPEGHI 2015 1 2015/6/8 3.1 Smalltalk To understand the problem, imagine that the original tree Smalltalk systems solve the expression problem very neatly, contained (among others) a class Sum, representing a sum by allowing modules to both add new classes, and to add expression, with instance variables left and right, and that features to existing classes. Ruby has the same property. The the only operation initially implemented by Sum and its sib- Smalltalk language itself does not have a concept of module, lings is pretty. The types assigned to these instance variables but every Smalltalk system does, and the ANSI Standard [1] would naturally include only the pretty operation. To add an specifies an “Interchange Format” intended for communicat- eval operation, we subclass Sum to create EvaluableSum, ing modules between Smalltalk implementations. which adds an eval method, the body of which contains Modules can contain class definitions, but can also con- { return left.eval() + right.eval() } tain method definitions and class re-definitions, which can Unfortunately, this won’t type-check, because the static be used to add methods and instance variables to existing types of left and right are too restrictive. If we assume that the classes. Thus, it is simple to extend a representation in the creation code uniformly instantiates the evaluable subtypes, data dimension by writing a module that defines a new class, then left and right will indeed contain evaluable objects, but and equally simple to extend a representation in the oper- Java’s type system loses this information. ation dimension by writing a module that adds a method to The basic idea behind the fix is to make the type of each of the classes representing the variants of the composite the instance variables a parameter. Then, without chang- structure. ing the source code, their declared type can be changed. This solution meets two out of the three constraints. First, Torgersen [8] works through the details using Java Gener- no existing code need be changed — with the obvious excep- ics. Unfortunately, the intricacies of Java’s type system — tion that there must be some changes in the creation code to F-bounding the parametric types, and then creating fixpoint allow the new structural variants to be created, and in the subclasses to instantiate them — forces him to observe “that client code so that the new tools can be employed. Second, the initial simplicity of the . approach has disappeared”. independent compilation is preserved: Smalltalk compiles In spite of all of this, the Java “solutions” fail for a second, each method separately, so adding individual methods poses completely different reason: they require us to change the no difficulty. Of course, it fails to meet Wadler’s constraint original code. Although the machinery of type parameters of static typing: Smalltalk does not have a static type system. may spare us from having to change the code in the base It is instructive to ask why this solution works: what prop- classes, no type machinery can spare us from changing the erty of Smalltalk makes it possible? It does not depend on creation code. Why is this? In a normal Java program the Smalltalk running in an image or on meta-programming, al- creation code will create instances of the various classes of though these techniques are certainly used in the implemen- expression by applying the new operator to the global name tation. The critical properties are that modules and classes of the class. The structure of the Java language says that we are orthogonal, and that classes are named by global vari- can’t rebind that name to something else, and the restrictions ables, not global constants. Thus, the loading of a new mod- of the expression problem say that we can’t change the code ule can change the meaning of a class name. in the expression classes. Notice that there is nothing fundamental that stops this It is the thesis of this paper that this is a major problem. process from working with static typing — if we assume The creation code is not something insignificant: in a com- that all changes are extensions that create subtypes. (Clearly, piler, for example, the “creation code” is the whole of the deleting methods from existing objects would not be type- parser. A “solution” to the expression problem that requires safe, but the expression problem is concerned with exten- that we edit every AST-node creation statement in the parser sion, not deletion.) just because we have added a new optimization pass is no solution. In practice, this problem can be mitigated by the 3.2 The Java Family of Languages clever use of Java’s import statement, which can be used to introduce short names to stand in for the global names of In Java as originally defined, the expression problem was the base classes, but redefining these short names still re- unsolvable. Extension in the data dimension was easily quires recompilation of the creation code, thus violating one achieved by a package that added a new class implementing of Wadler’s requirements for a solution.