Generalizers: New Metaobjects for Generalized Dispatch Christophe Rhodes Jan Moringen David Lichteblau Department of Computing Universität Bielefeld ZenRobotics Ltd Goldsmiths, University of Technische Fakultät Vilhonkatu 5 A London 33594 Bielefeld FI-00100 Helsinki London SE14 6NW [email protected] [email protected] [email protected] bielefeld.de ABSTRACT first two chapters covered programmer interface concepts This paper introduces a new metaobject, the general- and the functions in the programmer interface [17, Chapter izer, which complements the existing specializer metaobject. 28] and were largely incorporated into the final standard; With the help of examples, we show that this metaobject al- the third chapter, covering a Metaobject Protocol (MOP) lows for the efficient implementation of complex non-class- for CLOS, was not. based dispatch within the framework of existing metaob- Nevertheless, the CLOS MOP continued to be developed, ject protocols. We present our modifications to the generic and the version documented in [8] has proven to be a rea- function invocation protocol from the Art of the Metaob- sonably robust design. While many implementations have ject Protocol; in combination with previous work, this pro- derived their implementations of CLOS from either the Clos- duces a fully-functional extension of the existing mechanism ette illustrative implementation in [8], or the Portable Com- for method selection and combination, including support for mon Loops implementation of CLOS from Xerox Parc, there method combination completely independent from method have been largely from-scratch reimplementations of CLOS selection. We discuss our implementation, within the SBCL (in CLISP1 and CCL2, at least) incorporating substantial implementation of Common Lisp, and in that context com- fractions of the Metaobject Protocol as described. pare the performance of the new protocol with the standard one, demonstrating that the new protocol can be tolerably efficient. AMOP space Report-No.: http://eprints.gold.ac.uk/id/eprint/9924 sparse slots Categories and Subject Descriptors D.1 [Software]: Programming Techniques|Object-oriented Programming; D.3.3 [Programming Languages]: Lan- guage Constructs and Features • General Terms • Languages, Design Keywords generic functions, specialization-oriented programming, method CLOS selection, method combination message-not-understood• arXiv:1403.2765v2 [cs.PL] 22 Apr 2014 1. INTRODUCTION The revisions to the original Common Lisp language [16] in- cluded the detailed specification of an object system, known as the Common Lisp Object System (CLOS), which was eventually standardized as part of the ANSI Common Lisp standard [12]. The object system as presented to the stan- Figure 1: MOP Design Space dardization committee was formed of three chapters. The Although it has stood the test of time, the CLOS MOP is neither without issues (e.g. semantic problems with make-method-lambda [1]; useful functions such as compute- effective-slot-definition-initargs being missing from the standard) nor is it a complete framework for the metaprogrammer to implement all conceivable variations of object-oriented behaviour. While metaprogramming offers 1GNU CLISP, at http://www.clisp.org/ 2Clozure Common Lisp, at http://ccl.clozure.com/ some possibilities for customization of the object system be- specializer haviour, those possibilities cannot extend arbitrarily in all specializer-accepts-generalizer-p directions (conceptually, if a given object system is a point in design space, then a MOP for that object system allows s-a-g-p gen.-spec. generalizer exploration of a region of design space around that point; see figure 1). In the case of the CLOS MOP, there is still generalizer-of an expectation that functionality is implemented with meth- class-of ods on generic functions, acting on objects with slots; it is class argument not possible, for example, to transparently implement sup- subtypep port for \message not understood" as in the message-passing eql paradigm, because the analogue of messages (generic func- eql-specializer tions) need to be defined before they are used. eql-specializer-object Nevertheless, the MOP is flexible, and is used for a number of things, including: documentation generation (where in- Figure 2: Dispatch Comparison trospection in the MOP is used to extract information from a running system3); object-relational mapping4 and other approaches to object persistence [11]; alternative backing describing each protocol function in detail and, where ap- stores for slots (hash-tables [7] or symbols [3]); and pro- plicable, relating it to existing protocol functions within the grammatic construction of metaobjects, for example for in- CLOS MOP. We survey related work in more detail in sec- teroperability with other language runtimes' object systems. tion 4, touching on work on customized dispatch schemes in other environments. Finally, we draw our conclusions from One area of functionality where there is scope for customiza- this work, and indicate directions for further development, tion by the metaprogrammer is in the mechanics and seman- in section 5; reading that section before the others indicates tics of method applicability and dispatch. While in princi- substantial trust in the authors' work. ple AMOP allows customization of dispatch in various dif- ferent ways (the metaprogrammer can define methods on 2. EXAMPLES protocol functions such as compute-applicable-methods, In this section, we present a number of examples of dis- compute-applicable-methods-using-classes), for exam- patch implemented using our protocol, which we describe ple, in practice implementation support for this was weak in section 3. For reasons of space, the metaprogram code until relatively recently5. examples in this section do not include some of the neces- Another potential mechanism for customizing dispatch is im- sary support code to run; complete implementations of each plicit in the class structure defined by AMOP: standard spe- of these cases, along with the integration of this protocol cializer objects (instances of class and eql-specializer) into the SBCL implementation [13] of Common Lisp, are are generalized instances of the specializer protocol class, included in the authors' repository6. and in principle there are no restrictions on the metapro- A note on terminology: we will attempt to distinguish be- grammer constructing additional subclasses. Previous work tween the user of an individual case of generalized dispatch [10] has explored the potential for customizing generic func- (the \programmer"), the implementor of a particular case tion dispatch using extended specializers, but there the of generalized dispatch (the \metaprogrammer"), and the metaprogrammer must override the entirety of the generic authors as the designers and implementors of our general- function invocation protocol (from compute-discriminating- ized dispatch protocol (the\metametaprogrammer", or more function on down), leading to toy implementations and du- likely \we"). plicated effort. This paper introduces a protocol for efficient and controlled 2.1 CONS specializers handling of new subclasses of specializer. In particular, One motivation for the use of generalized dispatch is in an it introduces the generalizer protocol class, which gener- extensible code walker: a new special form can be han- alizes the return value of class-of in method applicability dled simply by writing an additional method on the walking computation, and allows the metaprogrammer to hook into generic function, seamlessly interoperating with all existing cacheing schemes to avoid needless recomputation of effec- methods. In this use-case, dispatch is performed on the tive methods for sufficiently similar generic function argu- first element of lists. Semantically, we allow the program- ments (See Figure 2). mer to specialize any argument of methods with a new kind The remaining sections in this paper can be read in any of specializer, cons-specializer, which is applicable if and order. We give some motivating examples in section 2, in- only if the corresponding object is a cons whose car is eql cluding reimplementations of examples from previous work, to the symbol associated with the cons-specializer; these as well as examples which are poorly supported by previ- specializers are more specific than the cons class, but less ous protocols. We describe the protocol itself in section 3, specific than an eql-specializer on any given cons. The programmer code using these specializers is unchanged 3 as in many of the systems surveyed at from [10]; the benefits of the protocol described here are: https://sites.google.com/site/sabraonthehill/ that the separation of concerns is complete { method se- lisp-document-generation-apps lection is independent of method combination { and that 4e.g. CLSQL, at http://clsql.b9.com/ 5the Closer to MOP project, at http://common-lisp.net/ 6the tag els2014-submission in http://christophe. project/closer/, attempts to harmonize the different im- rhodes.io/git/specializable.git corresponds to the plementations of the metaobject protocol in Common Lisp. code repository at the point of submitting this paper. the protocol allows for efficient implementation where pos- (walk form env (cons form call-stack)))))) sible, even when method selection is customized. In an ap- (defmethod walk plication such as walking source code, we would expect to ((expr (cons let)) env call-stack) encounter special forms (distinguished by particular atoms (flet ((let-binding